How it works

The agent loop, with nothing hidden in it.

The device layer sits behind a DeviceDriver interface. Android is implemented today; iOS can be added without touching the agent loop.

Input per turn

What the model sees

One screenshot of the current screen and a compacted UI hierarchy digest. Nothing is remembered for it — the trace is the memory.

Output per turn

Exactly one tool call

Coordinates are clamped to the real screen bounds before anything is dispatched, so a hallucinated tap lands inside the device or not at all.

Termination

A verdict, or the budget

The run ends when the model reports a verdict for the goal, or when the turn budget runs out — in which case the verdict is inconclusive, not a guess.

Tool vocabulary

tap(x, y)Coordinates clamped to the real screen bounds before dispatch.

type(text)Types into the focused field. No key-event escape hatch.

scroll(direction, distance)Bounded swipe in one of four directions.

press(key)Navigation keys only, from a closed enum — back, home, recents.

launch_app(package)Regex- and allowlist-checked package name.

stop_app(package) · restart_app(package)Same validation as launch. Nothing else may be stopped.

report(verdict, evidence)Ends the run with pass, fail or inconclusive and the frames that justify it.

Live control surface

Watch it work, or read it later.

Start a run from the browser, watch the device mirror, and watch tool calls, model commentary and findings stream in over a per-run WebSocket. Finished runs land in the sidebar next to the same report the CLI writes.

The mirror polls adb screencap at 8–10 fps rather than streaming scrcpy, which would need a v4l2loopback module and an ffmpeg transcode per run. Smooth enough for UI, honest about what it is.

Verdict semantics

pass The goal was reached and the agent can point at the frame where it happened.

fail The app did something the goal forbids, or the path is genuinely broken. This is the only verdict that should block a merge.

inconclusive A login wall, a missing fixture, a flaky network. Treated as a signal to fix the harness, not the app.