Agentic QA · Android today
Describe the test. It drives the device.
Vision Agent gives a live vision model one screenshot and a compacted UI hierarchy per turn, lets it tap, type and scroll through a real device, and returns pass, fail or inconclusive against a goal you wrote in plain English.
adb screencap · 8–10 fps
The loop
One turn: look, decide, act. Then again.
01 — Look
Screenshot and hierarchy
Each turn the model receives a fresh frame of the device plus a compacted UI tree — enough structure to name elements, small enough to stay in context.
02 — Act
A closed set of tools
Tap, type, scroll, press a navigation key, launch, stop or restart an app. Nothing else is reachable — there is no shell tool.
03 — Judge
Three honest verdicts
Pass, fail, or inconclusive. A model that could not reach the screen says so instead of guessing, and the run is kept either way.
Artifacts
Every run leaves a trace you can read.
A machine-readable run.json with every tool call, result and hierarchy digest — and a self-contained report.html screenshot timeline a human can review without installing anything.
{
"run_id": "run_8f21ac",
"goal": "sign in and check the profile screen",
"device": "emulator-5554",
"turns": [
{ "n": 1, "tool": "launch_app",
"args": { "package": "com.example.app" },
"result": "ok", "hierarchy_digest": "a3f1…" },
{ "n": 2, "tool": "tap",
"args": { "x": 540, "y": 1712 },
"result": "ok", "clamped": false }
],
"verdict": "pass",
"duration_ms": 42310
}
Goals
Free-form, not a script
The goal is a sentence. No selectors to maintain, no page objects to refactor when the layout moves.
Devices
One driver interface
Android runs through adb today. iOS slots in behind the same DeviceDriver without touching the loop.
Review
A report anyone can open
A self-contained screenshot timeline per run. No dashboard login to hand a designer or a PM.
Scope
Package allowlist
The agent can only launch, stop or restart packages you named. Everything else is refused before it reaches adb.