Midscene

The interesting unit isn't any single tool. It's "vision LLM + remote browser" as a primitive. Once you have it, you stop writing selectors. Full code, .env setup, the one footgun, end-to-end: @midscene/i-let-an-open-source-vision-model-drive-a-cloud-browser-no-dom-no-selectors-just-screenshots-6eae8cbf34cc" target="_blank" rel="nofollow noopener">medium.com/@midscene/i-le…

What this unlocks: • Smoke tests for any app, regardless of how the DOM rebuilds • Scraping with prose instead of XPath • Visual assertions that understand semantics • Same script runs Web / Android / iOS / Desktop You stop writing selectors. You start writing intent.

I let an open-source vision model drive a cloud browser. ~80 lines of TypeScript. No DOM. No selectors. Just screenshots. It worked on the first try. Full write-up + code below 👇

Using Midscene at your company? Reply or DM — we'll add you 🙌 👉 midscenejs.com

🏢 Inside ByteDance: Douyin, TikTok, Lark, Doubao, Volcengine, Soda Music, Dongchedi… 🌍 Outside ByteDance: Alibaba, Xiaomi, AVATR, iQiyi, Bilibili, Autel… ⚠️ Not every company using Midscene made this list — still collecting.

A counterintuitive number: Midscene's external community group (3,400+ members) is now bigger than the internal ByteDance user group (1,800+). A question we keep getting: "Does anyone at ByteDance use Midscene? Anyone outside?" Time to share the receipts 👇

The trick is invalidation. UI changed → falls back to model decision automatically. Never get stuck on a stale cache entry. Three months into writing an E2E suite, you'll appreciate this. → midscenejs.com/caching

Real numbers from our docs: an e-commerce search script runs 51s without cache, 28s with cache hit. Your speedup depends on script, model, and page complexity.

The real cost of AI-driven E2E tests isn't a single inference. It's regression — same script, 50 runs a day, every run hitting the model.

Electron, Qt, native GTK — they all work in CI now, because we go pure vision and don't need accessibility APIs. Attached: latest green run #25790418497 of that workflow. → midscenejs.com/computer-getti…

Our fix is one apt-get install + one API flag. Midscene wraps xvfb-run for you, no manual Xvfb startup. Full workflow Midscene Studio runs in CI — copy-paste ready in the repo: github.com/web-infra-dev/…

Underrated capability: Midscene.js runs desktop automation headless on Linux CI. Most teams assume desktop UI automation needs a real machine or VM.

Want to see one for real? Public report, opens in any browser: lf3-static.bytednsdoc.com/obj/eden-cn/nu… (A run where Midscene auto-likes a tweet on @midscene_ai.)

Three completely different fixes for: prompt was wrong / screenshot was off / model didn't get it. Drop the report into a GitHub issue or Slack — anyone can reproduce your case without setting up the environment.

The piece of Midscene.js I'm most proud of isn't the AI part. It's the HTML replay report.