Midscene is an AI-driven UI automation framework that enables developers to control user interfaces across web browsers, mobile devices (Android, iOS, HarmonyOS), and desktop applications using natural language descriptions. The framework employs a vision-first approach for element localization and task planning, reducing the reliance on DOM inspection or platform-specific selectors.
This page provides a high-level introduction to Midscene's architecture, core concepts, and design philosophy.
Midscene's architecture is built on three foundational principles:
ModelConfigManager packages/core/package.json22-26Sources: packages/web-integration/package.json2-10 packages/core/package.json3-4 README.md46-55
The framework is organized as a monorepo using pnpm workspaces and nx for task orchestration package.json45-59 Foundation packages provide platform-agnostic functionality, while platform SDKs implement device-specific automation.
Title: Midscene Monorepo Organization
Sources: package.json1-65 packages/web-integration/package.json110-122 packages/core/package.json86-100 packages/shared/package.json92-103 packages/android/package.json44-54 packages/cli/package.json21-33
The execution flow begins with natural language input or YAML scripts, which are processed by the Agent class packages/core/package.json42-46 The Agent coordinates with internal services for planning and execution. All platform-specific operations flow through an interface contract defined by AbstractInterface packages/core/package.json32-36
Title: Core Automation Execution Flow
Sources: packages/core/package.json42-46 packages/core/package.json22-26 packages/core/package.json32-36 packages/web-integration/package.json78-82
Midscene uses a pure vision approach for element localization, meaning that UI actions rely primarily on screenshot analysis. This design enables:
Qwen3.x, Doubao-Seed-2.1, GLM-4.6V, gemini-3.5-flash, and UI-TARS README.md67-71@ui-tars/action-parser to translate visual grounding results into executable actions packages/core/package.json88Sources: packages/web-integration/package.json4-10 packages/core/package.json88 README.md67-71
Midscene routes different task types to specialized models through intent-based configuration managed by ModelConfigManager packages/core/package.json22-26 This allows users to optimize for cost or performance by selecting different providers for planning vs. simple element localization.
Title: Intent-Based Model Routing
Sources: packages/core/package.json22-26 packages/shared/package.json40-44 README.md67-71
Midscene supports multiple integration levels, all converging on the same automation core:
midscene-web and midscene-android for CI/CD and mobile execution packages/web-integration/package.json105-107 packages/android/package.json20-22 packages/cli/package.json9-11Sources: packages/web-integration/package.json17-92 packages/cli/package.json1-12 apps/chrome-extension/static/manifest.json1-45 README.md32-34
Midscene implements caching to improve performance and reduce AI costs. Replaying scripts with cache enabled allows for faster results by skipping redundant AI calls. Caching can be controlled via environment variables like MIDSCENE_CACHE packages/web-integration/package.json98-103
Sources: packages/web-integration/package.json98-103
Every automation run generates visualized reports. These include:
midscene-skills to control any platform with OpenClaw README.md32-34Sources: packages/visualizer/package.json8-21 packages/core/package.json37-41 README.md32-34