web-infra-dev/midscene is an E2E (End-to-End) testing framework that reads app screens with AI vision, operates them on your behalf, and checks the results. AI vision means judging a screen by looking at it as an image. PlaywrightAgent interprets screenshots to perform clicks, input and scrolling on web pages, and visually verifies whether what appears on screen is correct. The repository description also presents E2E testing that drives interfaces in natural language. The latest release is v1.16.0, and the given sources do not state what changed in this version.
The README describes Midscene.js as a testing framework that runs E2E tests against web, mobile and desktop apps in natural language using AI vision and verifies the visible results. It can be integrated into existing tests through Midscene Test's YAML or the Agent API for Playwright and Puppeteer, and a single API can be used as-is across multiple platforms and custom interfaces. Run results appear together in an HTML report — screenshots, actions and assertion outcomes — so failure points can be matched back against the screen. The implementation language is TypeScript, the license is MIT, and the repository has 15341 GitHub stars. It was created on July 23, 2024.
There are, however, large parts that remain unverified. The given sources contain no accuracy or failure-rate figures for recognizing and operating screens, and no comparative numbers against other automation testing tools. The natural-language execution and verification capabilities described in the README are documentation claims, not independently verified results. For scenarios centered on checking what appears on screen, it is an option worth trying. More of your own testing is needed before you can rely on it.