# Phone Harness Documentation
> Technical docs for phone-harness: the CLI, pre-imported helpers, dual input backends, connection gates, OCR, and diagnostics that drive a real iPhone through macOS iPhone Mirroring.
## Context Links
- [Agent index](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms.txt)
- [Human interactive docs](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e)
- [GitHub repository](https://github.com/ShawnPana/phone-harness)
## Repository Metadata
- Repository: ShawnPana/phone-harness
- Generated: 2026-08-10T20:26:54.459Z
- Updated: 2026-08-10T21:10:47.530Z
- Runtime: Grok CLI
- Format: Documentation
- Pages: 17
## Page Index
- 01. [Overview](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/01-overview.md) - What phone-harness exposes, who it is for, runtime assumptions (macOS Sequoia+, iPhone Mirroring, real phone), and the first docs routes to follow.
- 02. [Installation](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/02-installation.md) - Prerequisites, clone to ~/.phone-harness, pyobjc and editable pip install, PATH command, skill registration, and verification with phone-harness --doctor.
- 03. [Quickstart](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/03-quickstart.md) - First successful script: check connection state, print screen_info, run a short stdin Python block with pre-imported helpers, and read the success signal.
- 04. [Connection and session states](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/04-connection-and-session-states.md) - connection_state values ready, blocked, no-window, not-running; ensure_mirroring gates; blocked interstitial markers; and why reconnect is a user physical action.
- 05. [Coordinates, capture, and OCR](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/05-coordinates-capture-and-ocr.md) - Global screen-point space, window bounds vs image pixels, capture return shape, Vision OCR boxes with confidence and tap-ready centers, and why coordinates must not be cached.
- 06. [Input backends](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/06-input-backends.md) - Default background backend (SkyLight event records, no focus steal) versus classic mirror backend (CGEvent HID + activate); selection via PHONE_HARNESS_BACKGROUND and automatic fallback.
- 07. [Consent and limits](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/07-consent-and-limits.md) - When not to use the phone, outward-facing and irreversible actions that require user consent, and hard product limits (one session, no multi-touch, DRM black frames, keycode typing).
- 08. [See, act, and verify](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/08-see-act-and-verify.md) - OCR-first reading, tap and tap_text, wait_stable, screenshot for unlabeled icons, and the capture-as-ground-truth verification loop after every action.
- 09. [Navigate apps and type text](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/09-navigate-apps-and-type-text.md) - home, app_switcher, open_app via Spotlight, press key combos, type_text US keycode constraints, and Home Screen tap_icon versus in-app tap_text.
- 10. [Scroll and collect lists](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/10-scroll-and-collect-lists.md) - scroll_screen movement detection, scroll_until predicates, scroll_collect de-dup extraction, wheel versus flick behavior, settle windows, and stop reasons reached-end and max-scrolls.
- 11. [Extend agent helpers](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/11-extend-agent-helpers.md) - Edit agent-workspace/agent_helpers.py for task-specific primitives, PH_AGENT_WORKSPACE override, auto-load into the script namespace, and the tap_icon Home Screen pattern.
- 12. [Register as an agent skill](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/12-register-as-an-agent-skill.md) - phone-harness skill output, Claude Code and Codex skill install paths, trigger text, and re-sync after pulls so the agent auto-reaches for phone control.
- 13. [CLI reference](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/13-cli-reference.md) - phone-harness entry points: stdin Python exec with helpers in scope, --doctor and doctor, skill, help flags, usage errors when args or TTY lack a script, and the ./phone-harness checkout launcher.
- 14. [Helpers API](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/14-helpers-api.md) - Pre-imported public helpers: session, capture, ocr, gestures, scroll family, navigation, timing, and re-exported backend primitives with signatures, defaults, return shapes, and raised errors.
- 15. [Environment variables](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/15-environment-variables.md) - PHONE_HARNESS_BACKGROUND default and falsey values, PH_AGENT_WORKSPACE path override, temp capture directory under phone-harness, and how backend selection falls back when SkyLight load fails.
- 16. [Doctor diagnostics](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/16-doctor-diagnostics.md) - Ordered --doctor ladder: pyobjc, Accessibility, Screen Recording, app installed, running, window, capture size, Vision OCR; exit codes; and the note that fresh machines may need extra prompts.
- 17. [Troubleshooting](https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/17-troubleshooting.md) - Failure modes and recovery: blank capture after Screen Recording grant, window not found, silent taps without Accessibility or focus, blocked iPhone in Use, Home Screen label misses, and type_text field focus.
## Source File Index
- `agent-workspace/agent_helpers.py`
- `install.md`
- `phone-harness`
- `pyproject.toml`
- `README.md`
- `SKILL.md`
- `src/phone_harness/admin.py`
- `src/phone_harness/background.py`
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/ocr.py`
- `src/phone_harness/run.py`
---
## 01. Overview
> What phone-harness exposes, who it is for, runtime assumptions (macOS Sequoia+, iPhone Mirroring, real phone), and the first docs routes to follow.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/01-overview.md
- Generated: 2026-08-10T20:15:30.661Z
### Source Files
- `README.md`
- `pyproject.toml`
- `src/phone_harness/run.py`
- `src/phone_harness/helpers.py`
- `SKILL.md`
- `phone-harness`
---
title: "Overview"
description: "What phone-harness exposes, who it is for, runtime assumptions (macOS Sequoia+, iPhone Mirroring, real phone), and the first docs routes to follow."
---
`phone-harness` is a local CLI that drives a **real iPhone** through the macOS **iPhone Mirroring** window: stdin Python is `exec`'d with capture, Vision OCR, gesture, navigation, and scroll helpers already in scope. There is no daemon, no WebDriverAgent, and no jailbreak — each invocation re-queries the mirror window, captures pixels, and posts HID-level (or SkyLight background) events against global screen points.
## What it exposes
| Surface | Role |
|--------|------|
| `phone-harness` CLI | Primary entry: run a Python script from stdin with helpers pre-imported |
| `phone-harness --doctor` / `doctor` | Permission and session ladder (pyobjc → Accessibility → Screen Recording → app → window → capture → OCR) |
| `phone-harness skill` | Prints `SKILL.md` for agent skill registration |
| `./phone-harness` | Checkout launcher: `PYTHONPATH=src` + `python3 -m phone_harness.run` without a global install |
| `phone_harness.helpers` | Protected core primitives used by every script |
| `agent-workspace/agent_helpers.py` | Agent-editable extensions auto-loaded into the same script namespace |
| Env: `PHONE_HARNESS_BACKGROUND` | Default `1`: background (SkyLight) backend; `0`/`false`/`no` forces classic mirror + activate |
| Env: `PH_AGENT_WORKSPACE` | Override path for agent helpers (default: repo `agent-workspace/`) |
Console entry point (from `pyproject.toml`): `phone-harness = phone_harness.run:main`. Package requires Python `>=3.10` and PyObjC frameworks: Quartz, Vision, AppKit, ApplicationServices. Install guidance also targets Python 3.12+ with explicit `pip install` of those frameworks.
### Typical invocation
```bash
phone-harness <<'PY'
print(connection_state())
print(screen_info())
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PY
```
Helpers such as `ocr`, `tap`, `tap_text`, `scroll_collect`, `home`, `open_app`, and `wait_stable` are injected from `helpers` (plus non-underscore names from `agent_helpers.py`). Raw Quartz remains available via `import Quartz` inside the script for escape hatches the helpers do not cover.
## Who it is for
| Audience | Use |
|----------|-----|
| Coding agents (Claude Code, Codex, or any agent with a skill file) | Automate iOS-only work: open apps, tap, type, swipe, read the mirrored screen |
| Developers integrating local phone control | Thin Python harness over capture + HID events, editable workspace helpers |
| Humans supervising an agent on a real device | Pair once, grant terminal permissions, connect/lock the phone when mirroring pauses |
Use phone-harness only when the task **needs the phone** (iOS-only apps, number/2FA-bound flows, on-device UI). Prefer Mac or web when those suffice. Outward-facing or hard-to-reverse actions (messages, posts, purchases, deletes, settings changes) require explicit user consent; do not linger in personal content beyond the task.
## Runtime assumptions
| Requirement | Detail |
|-------------|--------|
| Host OS | macOS **Sequoia+** with system app **iPhone Mirroring** (`com.apple.ScreenContinuity`, `/System/Applications/iPhone Mirroring.app`) |
| Device | A **paired, real iPhone** — not a simulator. Pairing prompts need the physical phone once |
| Transport | The mirroring **window** is the whole stack: eyes = window capture + Vision OCR; hands = CGEvent HID (classic) or SkyLight event records (background default) |
| Terminal permissions | **Accessibility** (taps/keystrokes; immediate) and **Screen Recording** (capture; often needs terminal restart) |
| Session ownership | Connecting or resuming mirroring is a **physical user action**. The harness never taps Connect/Continue or poll-waits to reconnect |
| Process model | **Stateless** — no long-running service; window bounds and captures are re-queried per call |
```text
agent / shell
│ stdin Python
▼
phone-harness (run.main)
│ exec(helpers + agent_helpers)
▼
helpers ──► background.py (default) or mirror.py
│ │
│ ├─ eyes: CGWindowListCreateImage / screencapture
│ └─ hands: SLPSPostEventRecordTo / CGEvent HID
▼
ocr.py (Vision) → [{text, confidence, x, y, w, h}] global screen points
▼
iPhone Mirroring window → real iPhone
```
Unlocking the physical phone pauses mirroring ("iPhone in Use"). Agents must stop, relay the error from `ensure_mirroring()` / `connection_state()`, and wait for the user to lock or reconnect — never tap through interstitials.
## Control model
The supported agent loop is **see → act → verify**:
1. **See** — Prefer `ocr()` (text + tap-ready centers). Use `screenshot()` + a vision-capable model for unlabeled icons.
2. **Act** — `tap` / `tap_text` / gestures / `open_app` / `type_text` / scroll family. Coordinates are **global screen points**; never cache them across calls (the window moves).
3. **Verify** — `wait_stable()` then `ocr()` or `screenshot()`. There is no DOM; the capture is ground truth.
Connection gate before work:
| `connection_state()` | Meaning |
|----------------------|---------|
| `ready` | Connected; safe to proceed (`ensure_mirroring()` activates and returns the window) |
| `blocked` | Connect / "iPhone in Use" / paused interstitial (OCR markers such as `iphone in use`, `lock your iphone`, `mirroring ended`, `to connect`) |
| `no-window` | App open, no phone window |
| `not-running` | iPhone Mirroring not running |
## Input backends
| Backend | Module | Behavior |
|---------|--------|----------|
| Background (default) | `background.py` | Capture without focus steal; mouse via SkyLight private APIs. Keyboard still briefly activates (record path not fully implemented). Loads unless `PHONE_HARNESS_BACKGROUND` is falsey; falls back to classic if SkyLight load fails |
| Classic mirror | `mirror.py` | `screencapture` + `CGEventPost`; window must be frontmost or events are swallowed |
Both backends share the same coordinate space and helper API (`tap`, `drag`, `type_text`, etc. re-exported from the selected module).
## Hard product limits
- One phone, one session.
- No multi-touch (no pinch / two-finger gestures).
- No camera or Face ID flows; DRM video often captures black.
- Typing uses **HID keycodes** (US layout constraints on `type_text`); unicode key payloads are not the mirror path.
- Home Screen: label text is not the tappable icon — use agent helper `tap_icon(label)` (~35 pt above label); in-app UI uses `tap_text` normally.
- Accessibility tree inside the mirror window is empty — AppleScript `click at` fails silently; only HID/SkyLight input works.
## Repository layout
:::files
phone-harness/
├── phone-harness # Dev launcher (sh → python -m phone_harness.run)
├── pyproject.toml # package phone-harness, console script, pyobjc deps
├── SKILL.md # Agent skill body (phone-harness skill)
├── install.md # Permissions, clone path, skill install
├── README.md # Setup prompt + architecture summary
├── agent-workspace/
│ └── agent_helpers.py # Agent-edited primitives (e.g. tap_icon)
└── src/phone_harness/
├── run.py # CLI: stdin exec, --doctor, skill, help
├── helpers.py # Pre-imported public API + backend select
├── mirror.py # Classic capture + CGEvent transport
├── background.py # Default no-focus backend
├── ocr.py # Vision → screen-point boxes
└── admin.py # --doctor ladder
:::
Canonical install home is `~/.phone-harness` (editable `pip install -e .`) so agents always know where source, `SKILL.md`, and `agent-workspace/` live.
## Verification signal
After install and pairing:
```bash
phone-harness --doctor
phone-harness <<'PY'
print(screen_info())
PY
```
Success: doctor reports clear (or only non-fatal notes), and `screen_info()` returns window bounds in screen points plus `img_px`. Blank capture after granting Screen Recording almost always means the terminal was not restarted.
## Next
Clone to ~/.phone-harness, pyobjc + editable install, PATH, skill registration, and --doctor.
First stdin script: connection_state, screen_info, and a short helper block.
ready / blocked / no-window / not-running, ensure_mirroring, and physical reconnect.
Full pre-imported helper surface: session, OCR, gestures, scroll, navigation, timing.
phone-harness skill output, Claude Code / Codex install paths, and re-sync after pulls.
When not to use the phone, consent gates, and hard product limits.
---
## 02. Installation
> Prerequisites, clone to ~/.phone-harness, pyobjc and editable pip install, PATH command, skill registration, and verification with phone-harness --doctor.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/02-installation.md
- Generated: 2026-08-10T20:14:59.002Z
### Source Files
- `install.md`
- `pyproject.toml`
- `phone-harness`
- `src/phone_harness/run.py`
- `src/phone_harness/admin.py`
- `README.md`
---
title: "Installation"
description: "Prerequisites, clone to ~/.phone-harness, pyobjc and editable pip install, PATH command, skill registration, and verification with phone-harness --doctor."
---
Installation installs the `phone-harness` console script (`phone_harness.run:main`) from an editable checkout at `~/.phone-harness`, registers the agent skill body printed by `phone-harness skill`, and verifies the permission/session ladder with `phone-harness --doctor`.
## Prerequisites
| Requirement | Detail |
|---|---|
| OS | macOS Sequoia or later (ships **iPhone Mirroring** at `/System/Applications/iPhone Mirroring.app`) |
| Phone | Real iPhone paired to iPhone Mirroring (open the app once manually; pairing needs the physical device) |
| Python | Package metadata: `requires-python >= 3.10`. `install.md` documents **Python 3.12+** for the install path |
| pyobjc | `pyobjc-framework-Quartz`, `pyobjc-framework-Vision`, `pyobjc-framework-AppKit` (declared deps also include `pyobjc-framework-ApplicationServices`) |
| Permissions | **Accessibility** (taps/keystrokes) and **Screen Recording** (capture) for the terminal app that runs `phone-harness` |
User-only steps the harness cannot do for you:
1. Pair iPhone Mirroring with the phone.
2. Toggle Accessibility and Screen Recording in System Settings → Privacy & Security.
Open the privacy panes:
```bash
open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"
```
| Permission | Effect timing | Failure if missing |
|---|---|---|
| Accessibility | Immediate | Taps/keystrokes fail or do nothing |
| Screen Recording | After terminal restart | Blank/black capture; doctor reports capture size too small |
Accessibility and Screen Recording are the permissions `--doctor` currently checks. A fresh Mac may still prompt for additional approvals the first time an action runs. If doctor passes but taps, typing, or capture silently do nothing, watch for a macOS prompt and re-check Privacy & Security for your terminal.
## Canonical install layout
`~/.phone-harness` is the canonical home: source, `SKILL.md`, `src/phone_harness/`, and `agent-workspace/` stay at a fixed path agents and editable installs can rely on.
:::files
~/.phone-harness/
├── phone-harness # checkout launcher (PYTHONPATH=src)
├── pyproject.toml # package + console_scripts entry
├── SKILL.md # skill body for phone-harness skill
├── install.md
├── agent-workspace/
│ └── agent_helpers.py # agent-editable helpers (auto-loaded)
└── src/phone_harness/
├── run.py # CLI: stdin exec, --doctor, skill
├── admin.py # doctor ladder
├── helpers.py # pre-imported script namespace
├── mirror.py # classic backend
├── background.py # default no-focus backend
└── ocr.py
:::
`pip install -e .` binds the `phone-harness` command to that checkout. Keep the folder at `~/.phone-harness`, or re-run `pip install -e .` after relocating.
## Install
```bash
git clone https://github.com/ShawnPana/phone-harness ~/.phone-harness
cd ~/.phone-harness
```
Install the frameworks first so `--no-deps` on the package install can skip network resolution if needed:
```bash
pip install pyobjc-framework-Quartz pyobjc-framework-Vision pyobjc-framework-AppKit
```
`pyproject.toml` also lists `pyobjc-framework-ApplicationServices` (used by doctor via `AXIsProcessTrusted`). Prefer a full editable install with deps when PyPI is reachable:
```bash
pip install -e .
```
When pyobjc is already installed (or PyPI is unreachable for deps):
```bash
pip install -e . --no-deps
```
This registers the console script:
| Item | Value |
|---|---|
| Command | `phone-harness` |
| Entry | `phone_harness.run:main` |
| Build backend | hatchling; wheel packages `src/phone_harness` |
After install, `phone-harness` works from any directory (required for skill-driven use).
Enable **Accessibility** and **Screen Recording** for the terminal (or IDE) that will run the command. Restart that app after granting Screen Recording.
Open **iPhone Mirroring** once manually and complete pairing. Connecting or resuming a session is a physical user action; the harness does not auto-connect past interstitials.
```bash
mkdir -p ~/.claude/skills/phone-harness
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness"
phone-harness skill > "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"
```
`phone-harness skill` prints repo-root `SKILL.md` (UTF-8) to stdout. Skill frontmatter name/description:
| Field | Value |
|---|---|
| `name` | `phone-harness` |
| `description` (trigger) | Control the user's iPhone through the Mac's iPhone Mirroring window: open apps, tap, type, swipe, read the screen. |
Re-run the redirect lines after `git pull` so registered copies stay current.
```bash
phone-harness --doctor
```
Smoke test once doctor is clean:
```bash
phone-harness <<'PY'
print(screen_info())
PY
```
Success: `screen_info()` prints window bounds (and image pixel size). That means the package, window discovery, and capture path are live.
### Checkout launcher (no pip)
From a working tree without installing:
```bash
./phone-harness --doctor
./phone-harness <<'PY'
print(screen_info())
PY
```
The `phone-harness` shell script sets `PYTHONPATH=/src` and runs `python3 -m phone_harness.run`. Day-to-day agent use still expects the PATH command from the editable install.
## CLI surfaces used at install time
| Invocation | Behavior |
|---|---|
| `phone-harness --doctor` / `phone-harness doctor` | Runs `admin.run_doctor()`; process exit `0` if required checks pass, else `1` |
| `phone-harness skill` | Prints `SKILL.md` from the installed package’s repo root |
| `phone-harness -h` / `--help` | Prints usage |
| `phone-harness` + stdin script | `exec` of Python with helpers from `phone_harness.helpers` in scope |
| No args on a TTY, empty stdin, or unexpected args | Exits with usage text |
## Doctor verification ladder
`phone-harness --doctor` prints `phone-harness doctor` then ordered checks. Fix the first `FAIL`; later steps depend on earlier ones.
| Order | Check | Notes |
|---|---|---|
| 1 | pyobjc frameworks (Quartz, Vision, AppKit) | Import failure → exit `1` immediately |
| 2 | Accessibility | `AXIsProcessTrusted()` |
| 3 | Screen Recording | `CGPreflightScreenCaptureAccess()` |
| 4 | iPhone Mirroring installed | Path exists at `/System/Applications/iPhone Mirroring.app` |
| 5 | iPhone Mirroring running | Non-fatal if not running (auto-launch path exists on first use for some backends) |
| 6 | Mirroring window found | Open app and pair if missing |
| 7 | Window capture works | `screencapture` of window id; size must be `> 20_000` bytes |
| 8 | Vision OCR works | Count of recognized text boxes when capture is large enough |
Output lines look like ` [PASS] …` or ` [FAIL] … — `. Closing line is `all clear` or `fix the FAILs above, then re-run`, plus a note that fresh machines may need extra permission prompts.
### Required vs informational checks
- **Required for exit 0** (when applicable): pyobjc, Accessibility, Screen Recording, app installed, non-blank capture when a window exists.
- **Reported but not always fatal**: app not running, window not found (no capture/OCR attempt without a window), OCR box count is informational once capture succeeds.
## Install failure modes
| Symptom | Likely cause | Recovery |
|---|---|---|
| `phone-harness: command not found` | Editable install missing or wrong env | `cd ~/.phone-harness && pip install -e .` (or `--no-deps` after pyobjc) |
| Doctor FAIL on pyobjc | Frameworks not installed in this Python | `pip install pyobjc-framework-Quartz pyobjc-framework-Vision pyobjc-framework-AppKit` |
| Capture blank / size FAIL | Screen Recording granted but terminal not restarted | Quit and reopen the terminal app, re-run doctor |
| Window not found | Not paired, out of range, or connect interstitial | Open iPhone Mirroring manually; user locks/connects phone if “iPhone in Use” |
| Taps do nothing | Accessibility off, or focus stolen (classic backend) | Grant Accessibility; see input-backends and troubleshooting if needed |
| Editable install broken after move | `pip install -e .` bound to old path | Keep `~/.phone-harness` or re-run `pip install -e .` |
## Related configuration (post-install)
Not required for install, but active after the package imports:
| Variable | Default | Role |
|---|---|---|
| `PHONE_HARNESS_BACKGROUND` | `"1"` (on) | Background/SkyLight backend unless `0`/`false`/`no`; falls back to classic `mirror` if load fails |
| `PH_AGENT_WORKSPACE` | `/agent-workspace` | Override path for agent-editable `agent_helpers.py` |
Temp captures land under the system temp dir in a `phone-harness` subdirectory.
## Next
First script: connection state, screen_info, and a short stdin Python block.
Full doctor ladder, exit codes, and fresh-machine permission notes.
Claude Code / Codex install paths, trigger text, and re-sync after pulls.
Entry points, usage errors, and the checkout launcher.
Blank capture, window missing, silent taps, and blocked sessions.
What phone-harness exposes and runtime assumptions.
---
## 03. Quickstart
> First successful script: check connection state, print screen_info, run a short stdin Python block with pre-imported helpers, and read the success signal.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/03-quickstart.md
- Generated: 2026-08-10T20:16:20.081Z
### Source Files
- `SKILL.md`
- `src/phone_harness/run.py`
- `src/phone_harness/helpers.py`
- `install.md`
- `phone-harness`
- `README.md`
---
title: "Quickstart"
description: "First successful script: check connection state, print screen_info, run a short stdin Python block with pre-imported helpers, and read the success signal."
---
`phone-harness` runs a one-shot stdin Python script with every public name from `src/phone_harness/helpers.py` already in scope—no imports, no daemon. The first successful path is: confirm session readiness with `connection_state()`, then print `screen_info()` and verify non-empty window bounds.
Install and permissions must already pass `phone-harness --doctor` (or `./phone-harness --doctor` from a checkout). Pairing iPhone Mirroring and granting Accessibility + Screen Recording are user-only steps—see [Installation](/installation) and [Doctor diagnostics](/doctor-diagnostics).
## Prerequisites
| Requirement | Notes |
|---|---|
| macOS Sequoia+ | iPhone Mirroring app available and paired once |
| `phone-harness` on PATH | From `pip install -e .` at `~/.phone-harness`, or `./phone-harness` in a checkout |
| Terminal permissions | Accessibility (input) and Screen Recording (capture); Screen Recording needs a terminal restart |
| Live session | iPhone Mirroring open with a connected phone (not the connect / “iPhone in Use” interstitial) |
## How the CLI runs a script
With no command args and a non-TTY stdin, `phone-harness` reads Python from stdin, builds a globals dict from every public name in `helpers` (names not starting with `_`), sets `__name__` to `"__main__"`, and `exec`s the code.
```bash
phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```
From a git checkout without install:
```bash
./phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```
| Invocation | Behavior |
|---|---|
| `phone-harness <<'PY' … PY` | Exec script with helpers pre-imported |
| `phone-harness --doctor` / `doctor` | Permission and session ladder; exits non-zero on hard FAIL |
| `phone-harness skill` | Prints `SKILL.md` to stdout |
| `phone-harness -h` / `--help` | Usage text |
| Args, empty stdin, or TTY stdin | Exits with usage (not a free-form Python REPL) |
Public helpers in scope for a first script include session and capture primitives such as `connection_state`, `ensure_mirroring`, `screen_info`, `screenshot`, `ocr`, `tap`, `tap_text`, `home`, `open_app`, `wait_stable`, plus re-exports (`tap`, `long_press`, `drag`, `press`, `type_text`, `activate`, `find_window`). Agent-editable names from `agent-workspace/agent_helpers.py` are auto-loaded into the same namespace at import time.
## First successful script
Call `connection_state()` before any gesture. Only `'ready'` means the mirrored phone content is usable.
```bash
phone-harness <<'PY'
state = connection_state()
print("state:", state)
if state != "ready":
raise SystemExit(f"not ready: {state} — user must connect/lock the phone")
print(screen_info())
PY
```
`screen_info()` captures the mirroring window and returns bounds plus capture pixel size. This is the install.md success signal: if window bounds print, the capture path works.
Success looks like a dict with non-zero `window` geometry and non-zero `img_px`. Failure is a raised error, empty usage exit, or a non-`ready` state string.
### Expected success output
`connection_state()` returns one of:
| Value | Meaning |
|---|---|
| `ready` | App running, window present, OCR does not show a blocked interstitial |
| `blocked` | Connect / “iPhone in Use” / paused interstitial text detected |
| `no-window` | App running but no phone window |
| `not-running` | iPhone Mirroring process not running |
`screen_info()` return shape:
```python
{
"window": {"x": float, "y": float, "w": float, "h": float, "id": int},
"frontmost": bool,
"img_px": [width_px, height_px],
}
```
| Field | Type | Meaning |
|---|---|---|
| `window` | dict | Screen-point bounds and CG window id (`x`, `y`, `w`, `h`, `id`) |
| `frontmost` | bool | Whether iPhone Mirroring is the frontmost app |
| `img_px` | `[int, int]` | Capture image width and height in pixels |
Example console shape (values vary by display layout):
```text
state: ready
{'window': {'x': 100.0, 'y': 80.0, 'w': 393.0, 'h': 852.0, 'id': 12345}, 'frontmost': True, 'img_px': [1179, 2556]}
```
If `state` is `ready` and `screen_info()` prints a `window` with positive `w`/`h` and non-zero `img_px`, the harness can see the phone. That is the first successful script.
## Optional gate: ensure_mirroring
`ensure_mirroring()` is the strict gate used before tasks: if state is `ready`, it activates the window and returns bounds; otherwise it raises `RuntimeError` with user-facing reconnect instructions. It never launches the app, taps Connect/Continue, or poll-waits for reconnect.
```bash
phone-harness <<'PY'
win = ensure_mirroring()
print("window:", win)
print(screen_info())
PY
```
| State | `ensure_mirroring()` |
|---|---|
| `ready` | Activates and returns window bounds |
| `not-running` | Raises — open iPhone Mirroring and connect |
| `no-window` | Raises — connect the phone in the app |
| `blocked` | Raises — user must connect / lock the phone; agent must not tap Connect |
Reconnecting is a physical user action. When state is not `ready`, stop and relay the error. Do not tap Connect/Continue and do not loop-poll; retry only after the user confirms they connected or locked the phone.
## Minimal script patterns
```bash title="State + screen_info (first success)"
phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```
```bash title="Strict gate then OCR sample"
phone-harness <<'PY'
ensure_mirroring()
info = screen_info()
print(info)
print([o["text"] for o in ocr()][:15])
PY
```
```bash title="Checkout launcher"
./phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```
## What counts as failure
| Symptom | Likely cause | Next action |
|---|---|---|
| Usage text only, non-zero exit | TTY stdin, empty script, or unexpected args | Use a heredoc; no extra CLI args on the script path |
| `not-running` | iPhone Mirroring not open | User opens the app |
| `no-window` | No phone window | User pairs/connects in the app |
| `blocked` | “iPhone in Use”, “lock your iphone”, “mirroring ended”, or “to connect” OCR | User locks phone / connects; do not tap interstitial |
| Capture errors / blank image | Screen Recording missing or terminal not restarted | Re-grant permission, restart terminal, re-run `--doctor` |
| Silent no-ops later on taps | Accessibility missing or focus lost | Grant Accessibility; re-activate if needed |
Blocked interstitial detection uses OCR over capture text matched case-insensitively against markers: `iphone in use`, `lock your iphone`, `mirroring ended`, `to connect`.
## Coordinates note for the next script
All helper coordinates are **global screen points**, not image pixels. `window` is in screen points; `img_px` is capture resolution. Convert unlabeled icon taps with image pixels ÷ scale + window origin. Do not cache coordinates across calls—window bounds are re-queried per capture/gesture.
## Consent boundary
This drives the user’s real phone. Navigating and reading for the user’s task is fine; stop and ask before outward-facing or hard-to-reverse actions (messages, posts, purchases, deletes, settings changes). Prefer Mac or web surfaces when the phone is not required.
## Next
ready / blocked / no-window / not-running, ensure_mirroring gates, and user-only reconnect.
OCR-first loop: ocr, tap_text, wait_stable, screenshot as ground truth.
Full pre-imported helper signatures, defaults, and return shapes.
Entry points, usage exits, skill output, and the checkout launcher.
Blank capture, window not found, silent taps, and blocked session recovery.
---
## 04. Connection and session states
> connection_state values ready, blocked, no-window, not-running; ensure_mirroring gates; blocked interstitial markers; and why reconnect is a user physical action.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/04-connection-and-session-states.md
- Generated: 2026-08-10T20:22:32.096Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `SKILL.md`
- `install.md`
- `src/phone_harness/admin.py`
---
title: "Connection and session states"
description: "connection_state values ready, blocked, no-window, not-running; ensure_mirroring gates; blocked interstitial markers; and why reconnect is a user physical action."
---
Session readiness for phone-harness is decided in `connection_state()` and gated by `ensure_mirroring()` in `src/phone_harness/helpers.py`. Both helpers are pre-imported into every `phone-harness` stdin script. The harness discovers the **iPhone Mirroring** app (`com.apple.ScreenContinuity`) and its on-screen window; it never launches the app, never taps **Connect** / **Continue**, and never polls for a reconnect.
## Why the harness does not reconnect
Connecting or resuming iPhone Mirroring is a **physical user action**: open the app, approve pairing/prompts, and — when the interstitial says **iPhone in Use** — **lock the iPhone**. Automation cannot complete those steps. Tapping Connect while the phone is unlocked does nothing useful; loop-polling burns time without progress.
When `ensure_mirroring()` raises, **stop and relay the error text to the user**. Do not tap through the interstitial. Retry only after the user confirms they connected or locked the phone.
## `connection_state()`
Cheap readiness probe. Use it before work, or when a capture looks like a connect screen instead of an app UI.
```python
connection_state() # -> "ready" | "blocked" | "no-window" | "not-running"
```
### Decision order
| Step | Check | Result |
|------|--------|--------|
| 1 | `mirror.running_app()` is `None` | `"not-running"` |
| 2 | `mirror.find_window()` is `None` | `"no-window"` |
| 3 | Capture the window, OCR all text, match blocked markers | `"blocked"` if any marker hits |
| 4 | Otherwise | `"ready"` |
Steps 3–4 require a successful window capture and Vision OCR. Permissions that break capture or OCR also break distinguishing `"blocked"` from `"ready"`.
### State values
| Value | Meaning | Typical user fix |
|-------|---------|------------------|
| `ready` | App running, phone window present, no blocked interstitial text | Proceed |
| `blocked` | Window present, but OCR shows a connect / paused interstitial | Open Mirroring; if **iPhone in Use**, lock the phone |
| `no-window` | App process exists, no qualifying phone window | Connect the phone in the app |
| `not-running` | Bundle `com.apple.ScreenContinuity` not running | Open **iPhone Mirroring** |
```mermaid
stateDiagram-v2
[*] --> not_running: running_app() is None
not_running --> no_window: app starts, no phone window
no_window --> blocked: window exists, interstitial OCR
no_window --> ready: window exists, no markers
blocked --> ready: user connects / locks phone
ready --> blocked: user unlocks phone ("iPhone in Use")
ready --> no_window: window disappears
ready --> not_running: app quits
```
## Blocked interstitial markers
When a window exists, `connection_state()` captures it, runs OCR, joins every box’s `text` (lowercased), and substring-matches against `_BLOCKED_MARKERS`:
| Marker (lowercase) | Role |
|--------------------|------|
| `iphone in use` | Session paused because the physical phone is unlocked |
| `lock your iphone` | Explicit lock instruction on the interstitial |
| `mirroring ended` | Session ended interstitial |
| `to connect` | Connect-flow copy |
Any match → `"blocked"`. Matching is case-insensitive substring on the concatenated OCR string, not exact full-screen equality.
## `ensure_mirroring()`
Hard gate used before tasks that need a live session.
```python
ensure_mirroring() # -> window bounds dict when ready; else raises RuntimeError
```
| `connection_state()` | Behavior |
|----------------------|----------|
| `ready` | Calls `mirror.activate()`, returns `mirror.find_window()` |
| `not-running` | `RuntimeError` — open iPhone Mirroring and connect |
| `no-window` | `RuntimeError` — connect the phone in the app |
| `blocked` | `RuntimeError` — connect; if **iPhone in Use**, lock the iPhone; harness will not tap Connect |
On success the return value is the window bounds dict from `find_window()`: `{x, y, w, h, id}` in global screen points.
`ensure_mirroring()` does **not** launch the app, tap Connect/Continue, or wait/poll for the user. Doc comments in helpers are the contract; treat SKILL phrasing that says it “launches” as outdated relative to this implementation.
### Example gate
```bash
phone-harness <<'PY'
print(connection_state())
win = ensure_mirroring()
print(win)
print(screen_info())
PY
```
Expected when ready: `connection_state()` prints `ready`, `ensure_mirroring()` returns bounds, `screen_info()` includes `window`, `frontmost`, and `img_px`.
## Window presence vs session readiness
| Helper | Module | What it guarantees |
|--------|--------|--------------------|
| `find_window()` | backend (`mirror` / re-export) | On-screen layer-0 window owned by **iPhone Mirroring** with width ≥ 100, or `None` |
| `ensure_window(timeout=5.0)` | backend | Same bounds, or raises if no window after optional activate/retry (classic) or immediate raise (background) |
| `connection_state()` | helpers | App + window + interstitial OCR classification |
| `ensure_mirroring()` | helpers | Full ready session, or user-facing `RuntimeError` |
The mirroring window can exist while the stream shows a paused/connect interstitial. `ensure_window()` only proves geometry; `connection_state()` / `ensure_mirroring()` prove the session is usable.
### Backend differences for `activate` / `ensure_window`
| Behavior | Classic `mirror` backend | Background backend (default when load succeeds) |
|----------|--------------------------|--------------------------------------------------|
| `activate()` | Bring app frontmost; does **not** launch | No-op (never steals focus) |
| `ensure_window()` | May activate and poll up to `timeout` for a window | Immediate raise if no window |
| `ensure_mirroring()` on ready | Calls `activate()` (focuses classic; no-op background) | Same call path |
App identity constants (classic backend): `APP_NAME = "iPhone Mirroring"`, `BUNDLE_ID = "com.apple.ScreenContinuity"`, `APP_PATH = "/System/Applications/iPhone Mirroring.app"`.
## Agent recovery protocol
Call `connection_state()` (or catch `ensure_mirroring()`).
Do not tap **Connect** / **Continue**. Do not loop-poll waiting for the session.
Surface the `RuntimeError` text from `ensure_mirroring()` (or map the state to the same instructions).
User opens Mirroring, connects, and locks the phone if required.
Call `ensure_mirroring()` again only after the user says they finished the physical steps.
## Session lifecycle notes
- **One phone, one session** — product limit; unlocking the physical phone pauses mirroring (`"iPhone in Use"` / blocked).
- **Pairing** is a first-time manual step on the physical device (see install path and doctor “window not found”).
- **`--doctor`** checks installed → running → window found → capture → OCR. It does not call `connection_state()` by name, but the same ladder underpins “window found” vs blank capture failures.
## Failure modes
| Symptom | Likely state / cause | Recovery |
|---------|----------------------|----------|
| `ensure_mirroring` not-running error | App closed | User opens **iPhone Mirroring** |
| `ensure_mirroring` no-window error | No phone window | User connects phone in the app |
| `ensure_mirroring` blocked / “iPhone in Use” error | Interstitial OCR match | User locks phone and reconnects; agent does not tap Connect |
| Window found but work hits connect UI | Agent skipped `ensure_mirroring` or race after unlock | Call `connection_state()` / re-gate |
| Capture blank / tiny | Screen Recording not effective yet | Restart terminal after grant; re-run doctor |
| Doctor: window not found | Unpaired, out of range, or connect screen without usable window | Open app manually once; complete pairing |
## Related pages
First script: check state, print `screen_info()`, confirm a live window.
Ordered ladder for permissions, app running, window, capture, and OCR.
Blank capture, window missing, blocked **iPhone in Use**, silent input.
How `activate()` differs between background and classic mirror backends.
Full signatures for session, capture, OCR, and input helpers.
One session, real phone constraints, and when not to automate.
---
## 05. Coordinates, capture, and OCR
> Global screen-point space, window bounds vs image pixels, capture return shape, Vision OCR boxes with confidence and tap-ready centers, and why coordinates must not be cached.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/05-coordinates-capture-and-ocr.md
- Generated: 2026-08-10T20:15:25.625Z
### Source Files
- `src/phone_harness/ocr.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `src/phone_harness/helpers.py`
- `SKILL.md`
---
title: "Coordinates, capture, and OCR"
description: "Global screen-point space, window bounds vs image pixels, capture return shape, Vision OCR boxes with confidence and tap-ready centers, and why coordinates must not be cached."
---
phone-harness treats the iPhone Mirroring window as a video stream: every public coordinate is a **global macOS screen point** (top-left origin), the same space `screencapture -R` and HID mouse events use. Capture returns a PNG plus live window bounds; Vision OCR maps image pixels back into those bounds so each text box is already tap-ready.
## Coordinate space
| Space | Origin | Units | Used by |
|-------|--------|-------|---------|
| Global screen points | Top-left of the Mac display | Logical points | `tap(x, y)`, OCR `x`/`y`/`w`/`h`, window bounds |
| Capture image pixels | Top-left of the PNG | Pixels | `screen_info()["img_px"]`, Vision raw boxes |
| Vision normalized boxes | Bottom-left of the image | 0–1 fractions | Internal only; flipped and scaled in `ocr.recognize` |
```text
Mac display (global screen points)
┌────────────────────────────────────────────┐
│ window = {x, y, w, h, id} │
│ ┌──────────────────┐ │
│ │ iPhone Mirroring│ ← video stream │
│ │ (no AX tree) │ │
│ └──────────────────┘ │
└────────────────────────────────────────────┘
capture PNG (img_px = [Wpx, Hpx])
scale: sx = w / Wpx , sy = h / Hpx
tap point = (window.x + img_x * sx, window.y + img_y * sy)
```
Window discovery (`find_window`) returns:
| Field | Type | Meaning |
|-------|------|---------|
| `x`, `y` | float | Top-left of the mirroring window in screen points |
| `w`, `h` | float | Window size in screen points |
| `id` | int | `kCGWindowNumber` (used by window-id capture) |
Windows owned by `"iPhone Mirroring"` with layer `0` and width ≥ 100 are accepted; smaller owner panels are ignored.
Never cache `(x, y)` across calls. The window can move, resize, or reappear; every helper re-queries bounds on each capture/gesture. Stale points miss the phone and tap empty desktop or another app.
## Capture
Both backends expose the same contract:
```python
path, win = mirror.capture(path=None, retries=2)
# path: str — PNG on disk
# win: dict — {x, y, w, h, id} at capture time
```
### Defaults and paths
| Backend | Module | Default PNG path | Capture mechanism |
|---------|--------|------------------|-------------------|
| Classic (focus-stealing) | `mirror.py` | `$TMPDIR/phone-harness/window.png` | `screencapture -l `, then `screencapture -R x,y,w,h` after `activate()` |
| Background (default) | `background.py` | `$TMPDIR/phone-harness/background.png` | `CGWindowListCreateImage` by window id (works unfocused / occluded) |
Temp directory: `Path(tempfile.gettempdir()) / "phone-harness"` (created on import).
### Classic capture path
1. Resolve window via `find_window()` or `ensure_window()`.
2. Prefer window-only: `screencapture -x -o -l ` (no shadow; fails if the window is not frontmost/composited).
3. On failure: `activate()`, re-read bounds, region capture `-R x,y,w,h`.
4. Success requires return code 0, file exists, and size **> 1000** bytes.
5. After `retries + 1` attempts (default `retries=2`): `RuntimeError("window capture failed after …")`.
### Background capture path
1. Resolve window (no activation).
2. `CGWindowListCreateImage` with `kCGWindowListOptionIncludingWindow`, `kCGWindowImageBoundsIgnoreFraming | kCGWindowImageNominalResolution`.
3. Encode PNG via Core Graphics destination.
4. Failures: empty image, encode failure → retry; then `RuntimeError("background capture failed: …")`.
### Public helpers over capture
| Helper | Signature | Returns |
|--------|-----------|---------|
| `screenshot` | `screenshot(path=None)` | PNG path only (`str`) |
| `screen_info` | `screen_info()` | `{window, frontmost, img_px}` |
| `ocr` | `ocr(min_confidence=0.3)` | list of OCR boxes (see below) |
| `wait_stable` | `wait_stable(timeout=6.0, interval=0.5, settle=2)` | `True` if consecutive capture digests match; else `False` |
`screen_info` fields:
| Field | Shape | Notes |
|-------|-------|-------|
| `window` | `{x, y, w, h, id}` | Screen-point bounds from the capture used for sizing |
| `frontmost` | `bool` | Whether iPhone Mirroring is the frontmost Mac app |
| `img_px` | `[width, height]` | Pixel dimensions of the PNG (`ocr.image_size`) |
`img_px` and `window` `w`/`h` often differ on Retina displays. Scale factors are:
```text
sx = window["w"] / img_px[0]
sy = window["h"] / img_px[1]
```
## OCR (Vision element tree)
`ocr()` is the preferred read path for anything with a text label. It captures, runs Apple Vision at **Accurate** recognition level, and maps every observation into global screen points.
### Return shape
Each element:
| Field | Type | Meaning |
|-------|------|---------|
| `text` | `str` | Top candidate string |
| `confidence` | `float` | Rounded to 3 decimals (0–1) |
| `x`, `y` | `float` | **Center** of the box in global screen points (rounded to 1 decimal) — pass straight to `tap` |
| `w`, `h` | `float` | Box size in screen points (rounded to 1 decimal) |
Default filter: `min_confidence=0.3`. Boxes below that threshold are dropped by the helper (raw `recognize` returns all candidates).
### Mapping math (`ocr.recognize`)
Vision boxes are normalized with a **bottom-left** origin. Conversion:
1. Pixel box from normalized `boundingBox` (Y flipped to top-left image space).
2. Scale by `sx`, `sy` from window points / image pixels.
3. Offset by `window["x"]`, `window["y"]`.
4. Emit **center**: `(px + pw/2)`, `(py_top + ph/2)`.
```python
# Conceptual — implemented in src/phone_harness/ocr.py
sx = window["w"] / img_w
sy = window["h"] / img_h
x = window["x"] + (px + pw / 2) * sx
y = window["y"] + (py_top + ph / 2) * sy
```
Failures raise `RuntimeError("Vision OCR failed: …")` or `RuntimeError("cannot read image …")` if the PNG cannot be opened.
### Related readers
| Helper | Behavior |
|--------|----------|
| `find_text(query, exact=False)` | Case-insensitive substring (or exact) over a fresh `ocr()` |
| `tap_text(query, index=0, exact=False)` | Tap center of match; on miss, raises with up to 30 visible strings |
| `_content_texts` (scroll internals) | OCR filtered to ~6%–92% vertical band to ignore status bar / home indicator |
## Manual points (icons without labels)
When there is no OCR target (glyph-only UI):
1. `info = screen_info()` — or `path = screenshot()` and keep bounds from a paired capture.
2. Inspect the PNG (human or vision model) in **image pixel** space.
3. Convert to screen points:
```python
info = screen_info()
win = info["window"]
Wpx, Hpx = info["img_px"]
sx = win["w"] / Wpx
sy = win["h"] / Hpx
# img_x, img_y measured in the PNG
tap(win["x"] + img_x * sx, win["y"] + img_y * sy)
```
Home Screen labels are not tappable targets: the icon sits ~35 screen points **above** the label. Use `tap_icon("Weather")` from agent helpers, not raw `tap_text`, for launcher icons.
## Why coordinates must not be cached
| Fact | Implication |
|------|-------------|
| Transport is **stateless** — no daemon holds window geometry | Each `phone-harness` invocation and each helper call re-discovers the window |
| `ocr()`, `swipe()`, `scroll_*`, `screen_info()`, `screenshot()` all capture or re-query bounds | Fresh `(x, y)` every time is the design |
| User can drag/resize the mirroring window mid-task | Cached points from an earlier `ocr()` miss |
| Session pause / reconnect can reposition the window | Re-run `ocr()` after `ensure_mirroring()` succeeds again |
Pattern: **read → act → re-read**. After a tap or gesture, call `wait_stable()` then a new `ocr()` / `screenshot()`. There is no accessibility tree or DOM; the PNG is ground truth.
```bash
phone-harness <<'PY'
print(screen_info())
for o in ocr()[:8]:
print(o["text"], o["confidence"], o["x"], o["y"])
hit = find_text("Settings")
if hit:
tap(hit[0]["x"], hit[0]["y"]) # or tap_text("Settings")
wait_stable()
print([o["text"] for o in ocr()][:12])
PY
```
## Backend note on coordinates
`PHONE_HARNESS_BACKGROUND` selects SkyLight background input vs classic HID + activate. **Both backends share the same global screen-point convention and the same `capture → (path, win)` shape**, so `ocr`, `tap_text`, and scroll helpers do not change coordinate math when the backend switches. Background mouse delivery also stores local window-relative points (`gx - win["x"]`, `gy - win["y"]`) inside the event record; callers still pass global points only.
## Failure modes (capture / OCR)
| Symptom | Likely cause | Check |
|---------|--------------|-------|
| Capture empty / tiny file | Screen Recording not effective until terminal restart | `phone-harness --doctor` (expects size > ~20 KB) |
| `window capture failed` / `background capture failed` | No window, permissions, or encoding failure | `connection_state()`, window presence, Screen Recording |
| `Vision OCR failed` | Vision/pyobjc issue | Doctor OCR step; pyobjc Vision framework installed |
| OCR boxes empty but UI has text | Confidence filter, blocked interstitial, DRM black frame | Lower `min_confidence`, inspect `screenshot()`, check blocked markers |
| Tap misses after saved coords | Window moved or layout changed | Re-run `ocr()`; do not reuse old centers |
| `tap_text` raises with `saw: […]` | Label not present or OCR miss | Read the visible list; screenshot for non-text UI |
## Related pages
OCR-first reading, tap_text, wait_stable, and capture-as-ground-truth after every action.
Background SkyLight capture/input versus classic screencapture + CGEvent activate.
Full signatures for capture, ocr, gestures, and return shapes.
ready / blocked / no-window / not-running gates before capture is meaningful.
Blank capture after Screen Recording grant, silent taps, Home Screen label misses.
---
## 06. Input backends
> Default background backend (SkyLight event records, no focus steal) versus classic mirror backend (CGEvent HID + activate); selection via PHONE_HARNESS_BACKGROUND and automatic fallback.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/06-input-backends.md
- Generated: 2026-08-10T20:17:12.762Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/background.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/run.py`
- `SKILL.md`
---
title: "Input backends"
description: "Default background backend (SkyLight event records, no focus steal) versus classic mirror backend (CGEvent HID + activate); selection via PHONE_HARNESS_BACKGROUND and automatic fallback."
---
`helpers.py` binds every script to one transport module at import time: either `background` (default) or `mirror` (classic). That binding decides how capture and input reach the iPhone Mirroring window (`com.apple.ScreenContinuity`). Higher-level helpers (`ocr`, `tap_text`, `swipe`, `scroll_screen`, `open_app`, …) call the bound module through a shared `mirror` symbol, so the public API stays the same across backends.
## Two backends
| Property | Background (`background.py`) | Classic (`mirror.py`) |
| --- | --- | --- |
| Default | Yes (`PHONE_HARNESS_BACKGROUND` defaults to on) | Only when env forces it, or when background import fails |
| Capture | `CGWindowListCreateImage` by window id | `screencapture -l` / region after activate |
| Mouse input | SkyLight `SLPSPostEventRecordTo` event records | `CGEventPost` at `kCGHIDEventTap` |
| Keyboard | Make-key record + `CGEventPostToPid` | `CGEventPost` HID after activate |
| Focus steal | No (`activate` is a no-op) | Yes (every input path calls `activate`) |
| Scroll lists | Fast vertical flick (wheel events do not land unfocused) | Real scroll-wheel `CGEvent`s |
| Temp capture default | `$TMPDIR/phone-harness/background.png` | `$TMPDIR/phone-harness/window.png` |
Both use **global screen points** for coordinates — the same space as `ocr()` centers and `CGEvent` locations. Helpers re-query window bounds per call; do not cache coordinates across captures.
## Selection and fallback
Selection runs once when `phone_harness.helpers` is imported (including every `phone-harness` stdin script).
```text
PHONE_HARNESS_BACKGROUND
│
▼
default "1" ──► truthy? ──no──► import .mirror
│
yes
▼
try import .background
│
success ──► use background
│
Exception ──► import .mirror, force _BACKGROUND=False
```
Controls transport selection. Default is on. Treated as **off** only when the lowercased value is exactly one of `0`, `false`, or `no`. Any other value (including empty-after-default) keeps background preferred.
| Value | Effect |
| --- | --- |
| unset / `1` / `true` / other | Prefer `background` |
| `0` / `false` / `no` (any case) | Force `mirror` |
| Background import raises | Silent fallback to `mirror`; process continues |
There is no public API to switch backends mid-process. Change the env var and start a new `phone-harness` invocation.
### Force classic mirror
```bash
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
print(screen_info()) # frontmost may flip True when capture/activate runs
PY
```
Use classic when you want the iPhone Mirroring window raised for live observation, when background taps or capture fail on a given macOS build, or when diagnosing focus-related issues.
## Shared public surface
After selection, these names are re-exported from the active backend onto `helpers` (and therefore into the script namespace):
| Symbol | Role |
| --- | --- |
| `tap(x, y)` | Left click / touch at global points |
| `long_press(x, y, duration=0.8)` | Hold then release |
| `drag(x1, y1, x2, y2, duration=0.35, steps=14)` | Touch-drag / swipe |
| `press(combo)` | Key combo, e.g. `cmd+1`, `return` |
| `type_text(text, delay=0.03)` | US-layout HID keycodes |
| `activate()` | Raise app (classic) or no-op (background) |
| `find_window()` | `{x, y, w, h, id}` or `None` |
Session helpers (`connection_state`, `ensure_mirroring`, `screenshot`, gestures, scroll family) always go through the same bound module for `capture`, `ensure_window`, `running_app`, and `activate`.
`background` reuses `mirror.find_window` and `mirror.running_app` (and keycode tables for keyboard). Window discovery itself never activates.
## Background backend
Module: `src/phone_harness/background.py`.
### Capture (eyes)
- `capture(path=None, retries=2)` → `(path, window_bounds)`.
- Uses `Quartz.CGWindowListCreateImage` with `kCGWindowListOptionIncludingWindow` and the mirroring window id.
- Works when the window is not frontmost and when another app occludes it.
- On failure after retries: `RuntimeError("background capture failed: …")`.
### Mouse (hands)
- Builds a `0xf8`-byte SkyLight event record (yabai-style layout): length, CGSEventType (`1` down, `2` up, `6` dragged), global location, window-local location, window id.
- Resolves process PSN via `GetProcessForPID`, then `_SLPSSetFrontProcessWithOptions` + `SLPSPostEventRecordTo`.
- Frontmost app on the Mac does not change for mouse paths.
- Event types and packing stay inside the fixed buffer; bad values fail as no-ops rather than crashes.
### Scroll in background
Mac scroll-wheel events only reach iPhone Mirroring when the app is active. Background `scroll_wheel(dy, x, y, steps=6)` therefore performs a **fast vertical flick** (drag with short per-step delays) from ~72% → ~28% of window height when `dy < 0` (reveal content below / `scroll_screen` `"up"`), and the reverse when `dy > 0`. Higher-level `scroll()`, `scroll_screen`, `scroll_until`, and `scroll_collect` keep calling `mirror.scroll_wheel`; behavior changes by backend without API changes.
### Keyboard
- `press` / `type_text` make the mirroring window **key** with a blanked-location focus record, then post key events with `CGEventPostToPid` — still without raising the app as frontmost.
- Same US keycode constraints as classic: unknown characters raise `ValueError(f"cannot type {ch!r} via keycodes")`.
- Shared keycode / modifier maps live on `mirror` (`_KEYCODES`, `_MODIFIERS`, `_keycode_for`).
### `activate` and `ensure_window`
| Function | Background behavior |
| --- | --- |
| `activate()` | Immediate `None` (documented no-op) |
| `ensure_window(timeout=5.0)` | No poll loop; raises if app not running or no phone window |
`ensure_mirroring()` still calls `mirror.activate()` on the ready path; under background that call does nothing.
## Classic mirror backend
Module: `src/phone_harness/mirror.py`.
### Why focus is required
The mirroring window is a video stream: Accessibility does not expose in-window UI. Classic capture uses `screencapture`, and input uses HID `CGEventPost`. Both paths target the focused app; unfocused events are swallowed silently.
### Capture
1. Prefer `screencapture -x -o -l `.
2. On failure, `activate()`, then region capture `-R x,y,w,h`.
3. After `retries + 1` attempts: `RuntimeError("window capture failed after …")`.
### Input
Every `tap`, `long_press`, `drag`, `scroll_wheel`, `press`, and `type_text` calls `_focus()` → `activate()` first:
- `activate()` uses `NSApplicationActivateIgnoringOtherApps` when the app is running but not frontmost, then sleeps ~0.5s.
- Mouse events: `CGEventCreateMouseEvent` + `CGEventPost(kCGHIDEventTap, …)`.
- Scroll: `CGEventCreateScrollWheelEvent` (pixel unit) at the target location.
- Keys: `CGEventCreateKeyboardEvent` with optional modifier flags.
Unicode string payloads on keyboard events are **not** used: iPhone Mirroring forwards raw HID keycodes only.
### `ensure_window`
If no window is found, classic calls `activate()` and polls until `timeout` (default 5s), then raises if still missing. That can bring the app frontmost even before a successful connection.
## Runtime path
```mermaid
flowchart TB
subgraph cli ["phone-harness CLI"]
RUN["run.main: exec stdin"]
end
subgraph helpers ["helpers.py"]
SEL["PHONE_HARNESS_BACKGROUND + import"]
API["tap / capture / ocr / scroll_* / open_app"]
end
subgraph backends ["Transport modules"]
BG["background.py\nSkyLight + CGWindowListCreateImage"]
MIR["mirror.py\nscreencapture + CGEvent HID"]
end
subgraph os ["macOS / iPhone Mirroring"]
APP["ScreenContinuity window"]
end
RUN --> API
SEL --> BG
SEL --> MIR
API --> SEL
BG --> APP
MIR --> APP
```
## Behavioral differences that matter
| Concern | Background | Classic |
| --- | --- | --- |
| User keeps working in another Mac app | Yes for mouse/capture/keyboard paths here | No — window is raised |
| `screen_info()["frontmost"]` during automation | Often `False` | Typically `True` after actions |
| Mid-run focus steal by another click | Mouse still targets process by pid | Events may be swallowed; re-activate |
| List scrolling feel | Momentum flick | Wheel deltas |
| `ensure_window` on missing window | Fail fast | Activate + poll |
| Private API risk | Depends on SkyLight symbols across macOS builds | Public Quartz / `screencapture` only |
If SkyLight symbols fail to load or the private path misbehaves on a given OS build, import falls back to classic automatically. Silent focus-steal after an upgrade is a signal that fallback engaged — force classic explicitly with `PHONE_HARNESS_BACKGROUND=0` to confirm, or inspect whether `background` imports cleanly in a Python REPL.
## Permissions and silent failures
Both backends still need:
- **Screen Recording** — capture path
- **Accessibility** — synthesized input
Classic additionally requires the mirroring window frontmost for reliable HID delivery. Background is designed so mouse and capture work without frontmost, but Accessibility must still be granted to the terminal that runs `phone-harness`.
| Symptom | Likely cause |
| --- | --- |
| Taps do nothing (classic or after fallback) | Accessibility missing, or window not frontmost |
| Capture empty / fails | Screen Recording not effective until terminal restart |
| Background import fails every run | Private SkyLight load error → automatic classic fallback |
| Keyboard types nowhere | iOS text field not focused first (tap field, wait, then `type_text`) |
## Verify which path you are on
There is no exported `is_background()` flag. Practical checks:
```bash
# Prefer background (default)
phone-harness <<'PY'
info = screen_info()
print(info["frontmost"], info["window"])
# With background, capture need not raise iPhone Mirroring
PY
# Force classic and expect activation
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
print(screen_info()["frontmost"]) # usually True after capture/activate
PY
```
Or probe import directly:
```bash
python3 -c "from phone_harness import helpers; print(helpers.mirror.__name__)"
# phone_harness.background or phone_harness.mirror
```
## Related pages
`PHONE_HARNESS_BACKGROUND` defaults, falsey values, and fallback when SkyLight load fails.
Public helper signatures that re-export backend primitives.
Global screen points shared by both backends and OCR tap centers.
`connection_state` / `ensure_mirroring` gates on the same bound transport.
Silent taps, focus steal, blank capture, and recovery steps.
Ordered permission and session checks for both capture and input.
---
## 07. Consent and limits
> When not to use the phone, outward-facing and irreversible actions that require user consent, and hard product limits (one session, no multi-touch, DRM black frames, keycode typing).
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/07-consent-and-limits.md
- Generated: 2026-08-10T20:14:54.057Z
### Source Files
- `SKILL.md`
- `README.md`
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
---
title: "Consent and limits"
description: "When not to use the phone, outward-facing and irreversible actions that require user consent, and hard product limits (one session, no multi-touch, DRM black frames, keycode typing)."
---
phone-harness drives the user's **real iPhone** through the macOS iPhone Mirroring window. Policy for when to use it, what requires an explicit user yes, and what the transport cannot do lives in `SKILL.md` (agent skill body) and is enforced in code by `ensure_mirroring()` / `connection_state()` in `helpers.py`, single-pointer input in `mirror.py` / `background.py`, and US-layout HID `type_text`.
## When not to use the phone
Prefer Mac or web whenever the task can complete there: a website, an API, or an app with a web equivalent. Use phone-harness only when the work genuinely needs the device:
| Use phone-harness | Prefer Mac / web |
| --- | --- |
| iOS-only apps | Browser or desktop app equivalent |
| Flows tied to the user's phone number or 2FA on-device | Server APIs, email links, desktop auth |
| How something looks or behaves on the phone | Same content available on desktop |
Navigating and reading for the user's own task is allowed. Do not linger in personal content (Messages, Photos, Mail) beyond what the task needs.
## Consent: stop and ask
This is the user's real phone. **Stop and ask before anything outward-facing or hard to reverse**, including:
- Sending a message
- Posting
- Purchasing
- Deleting
- Changing settings
Navigating UI and reading screen content for an in-scope task does not require per-step consent. Destructive or public side effects do.
The harness does not implement a consent API or dry-run mode. Consent is an agent/operator policy surface: halt, describe the intended action, and proceed only after the user confirms.
Setup-time actions that only the user can perform (pairing iPhone Mirroring, granting Accessibility and Screen Recording) are the same class of gate: the agent walks the user through them and waits; it does not bypass prompts.
## Connection is physical — not agent-automatable
Reconnecting or resuming mirroring is a **physical** user action: open iPhone Mirroring, approve prompts, and when the interstitial says **iPhone in Use**, **lock the iPhone**. The harness never connects the phone for the agent.
### Session gates
| Helper | Role |
| --- | --- |
| `connection_state()` | Returns `'ready'`, `'blocked'`, `'no-window'`, or `'not-running'` |
| `ensure_mirroring()` | Proceeds only when `'ready'`; otherwise raises with user-facing instructions |
`connection_state()` treats OCR text matching any of these markers as `'blocked'`:
- `iphone in use`
- `lock your iphone`
- `mirroring ended`
- `to connect`
On non-ready states, `ensure_mirroring()` raises `RuntimeError` (does not launch the app, tap Connect/Continue, or poll). Example messages instruct the user to open the app, connect the phone, or lock the phone when **iPhone in Use** appears.
### Agent rules when gated
1. **STOP** and relay the error message to the user.
2. **Never** tap `Connect` / `Continue`.
3. **Never** loop-poll waiting for connection (burns time; unlock-state resume only the user can fix).
4. Retry **once after the user confirms** they reconnected — not before.
Unlocking the physical phone pauses the session (**iPhone in Use**). Do not tap through the resume interstitial; ask the user to lock/connect.
## Hard product limits
Documented product ceilings (README **Limits**, SKILL **Gotchas**) and the implementations that produce them:
| Limit | Behavior |
| --- | --- |
| One phone, one session | Single iPhone Mirroring window; unlock pauses mirroring |
| No multi-touch | Mouse maps 1:1 to one touch — no pinch, no two-finger gestures |
| No camera / Face ID flows | Not exposed through the mirroring input path |
| DRM video | Protected content captures as black frames |
| Keycode typing only | Mirroring forwards raw HID keycodes; unicode string payloads are ignored |
| OCR is text, not semantics | Labels have coordinates; unlabeled icons need `screenshot()` + vision judgment |
### One session and pause semantics
There is no multi-device or multi-session API. Window bounds and captures are re-queried per call (stateless transport), but the **mirroring session** is still one paired phone. When the user unlocks the physical device, the stream shows a blocked interstitial until they lock/reconnect.
### Input: single pointer only
`tap`, `long_press`, and `drag` post a single left-button mouse stream (`LeftMouseDown` / `Dragged` / `Up`). There is no multi-finger synthesizer — pinches and other multi-touch gestures are out of scope.
### Capture: black DRM frames
Window capture (`mirror.capture` / background `CGWindowListCreateImage`) returns pixels as macOS renders them. DRM-protected video commonly appears black; treat black frames as a content/transport limit, not a permission failure (blank capture after granting Screen Recording without restarting the terminal is a different failure — see diagnostics docs).
### Typing: US HID keycodes
`type_text(text, delay=0.03)` and `press(combo)` map through real keycodes (US layout). Characters without a keycode mapping raise:
```text
ValueError: cannot type '' via keycodes
```
Examples of untypable content: emoji and other non-keycode glyphs. `\n` becomes Return. Prerequisites:
- An iOS text field must be focused first (tap the field, wait for the keyboard, then type).
- Background backend still briefly uses the keyboard path that requires the app path for keys; mouse actions can stay backgrounded.
Supported character set is the US keyboard plus shifted punctuation defined in `mirror.py` (`_KEYCODES`, `_PUNCT_KEYCODES`, `_SHIFTED`). Unknown `press` keys raise `ValueError: unknown key ...`.
## Related operational constraints
These are not consent gates, but they bound safe automation the same way:
| Constraint | Implication |
| --- | --- |
| Window is a video stream | No accessibility tree inside the phone UI; AppleScript `click at` fails silently |
| Coordinates are not stable | Do not cache `(x, y)` across calls; re-run `ocr()` / re-query bounds |
| Unfocused classic input is swallowed | Classic mirror backend needs frontmost window; silent no-ops if focus is stolen |
| Home Screen labels vs icons | `tap_text` hits the label; use `tap_icon` (agent helper) for Home Screen icons |
## Practical decision checklist
Use this order before acting:
1. Can this run on Mac or web? If yes, do not open the phone.
2. Is `connection_state()` `'ready'`? If not, stop and ask the user to connect/lock — never tap Connect.
3. Is the next action outward-facing or hard to reverse? If yes, stop and ask.
4. Will the step need multi-touch, Face ID, camera, DRM-protected video, or non-US/emoji typing? Plan an alternate path or human handoff.
5. After any allowed action: `wait_stable()` then `ocr()` / `screenshot()` — capture is ground truth.
```text
task request
│
├─ Mac/web enough? ──────────────► do not use phone
│
├─ connection_state != ready? ───► stop; user reconnects
│
├─ outward / irreversible? ──────► stop; ask consent
│
└─ within hard limits? ──────────► act → verify capture
```
## Next
`ready` / `blocked` / `no-window` / `not-running`, blocked markers, and why reconnect is physical.
`type_text` US keycode constraints, `press` combos, Home Screen vs in-app taps.
OCR-first reads, post-action verification, and when screenshots are required.
Blank/black capture, silent taps, blocked interstitials, and typing focus failures.
---
## 08. See, act, and verify
> OCR-first reading, tap and tap_text, wait_stable, screenshot for unlabeled icons, and the capture-as-ground-truth verification loop after every action.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/08-see-act-and-verify.md
- Generated: 2026-08-10T20:15:12.511Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/ocr.py`
- `src/phone_harness/mirror.py`
- `agent-workspace/agent_helpers.py`
- `SKILL.md`
---
title: "See, act, and verify"
description: "OCR-first reading, tap and tap_text, wait_stable, screenshot for unlabeled icons, and the capture-as-ground-truth verification loop after every action."
---
Every phone-harness script runs as stdin Python with helpers pre-imported from `phone_harness.helpers`. The control loop is fixed: **read the screen with Vision OCR**, **act at global screen points**, **wait until pixels settle**, then **re-capture to confirm the UI actually changed**. There is no accessibility tree or DOM inside the iPhone Mirroring window — the PNG capture is the only ground truth.
```text
ocr() / screenshot() → see
tap_text() / tap(x, y) → act
wait_stable() → settle animations
ocr() / screenshot() → verify (ground truth)
```
## Prerequisites
| Requirement | Why it matters |
|-------------|----------------|
| Session `connection_state() == "ready"` | Interstitials and disconnects block real UI |
| Screen Recording granted | Capture / OCR fail or return empty |
| Accessibility granted | Classic-backend taps/keys are silent without it |
| Coordinates treated as ephemeral | Window can move between calls |
Gate with `ensure_mirroring()` or check `connection_state()` before the loop. Reconnect is physical (user only) — see [Connection and session](/connection-and-session).
## See: OCR-first reading
Prefer `ocr()` over viewing screenshots whenever a control has a text label. `ocr()` captures the mirroring window, runs Apple Vision at accurate recognition level, and returns boxes with **tap-ready centers in global screen points**.
### `ocr(min_confidence=0.3)`
Captures, recognizes, and filters by confidence.
| Field | Type | Meaning |
|-------|------|---------|
| `text` | `str` | Recognized string |
| `confidence` | `float` | Rounded to 3 decimals; default floor `0.3` |
| `x`, `y` | `float` | Box **center** in screen points (pass to `tap`) |
| `w`, `h` | `float` | Box size in screen points |
Vision uses a bottom-left origin; the harness flips and scales into the window’s top-left screen-point space so centers are immediately tappable.
```bash
phone-harness <<'PY'
boxes = ocr()
print([(o["text"], o["x"], o["y"], o["confidence"]) for o in boxes][:20])
PY
```
Filter in Python before printing large dumps — full screens can be dense.
### `find_text(query, exact=False)`
Returns the subset of `ocr()` hits matching `query` (case-insensitive).
| Mode | Match rule |
|------|------------|
| `exact=False` (default) | Substring: `query.lower() in text.lower()` |
| `exact=True` | Full string equality (case-insensitive) |
```python
hits = find_text("Settings")
exact = find_text("Done", exact=True)
```
### Why not cache OCR coordinates
`ocr()`, `find_text()`, `swipe()`, and capture re-query window bounds each call. The mirroring window can move. Re-read after every navigation or layout change; do not store `(x, y)` across steps.
## Act: tap and tap_text
### `tap_text(query, index=0, exact=False)`
Finds text, taps the selected hit’s center, returns the hit dict.
| Param | Default | Role |
|-------|---------|------|
| `query` | required | Label to find |
| `index` | `0` | Which hit when several match |
| `exact` | `False` | Passed through to `find_text` |
On miss, raises `RuntimeError` including up to 30 currently visible OCR strings so the next step is informed:
```text
RuntimeError: no visible text matches 'New Note'; saw: ['Notes', 'Folders', ...]
```
```bash
phone-harness <<'PY'
hit = tap_text("New Note")
print(hit) # {text, confidence, x, y, w, h}
PY
```
### `tap(x, y)`
Direct tap at global screen points. Re-exported from the active backend (`background` by default, `mirror` on fallback).
| Backend | Delivery | Focus |
|---------|----------|--------|
| Background (default) | SkyLight event records to the mirroring process | Does not steal focus |
| Classic (`PHONE_HARNESS_BACKGROUND=0` or SkyLight load fail) | CGEvent HID + `activate()` | Brings window frontmost |
Both use the same screen-point convention, so `tap_text` / `ocr` stay backend-agnostic. Details: [Input backends](/input-backends).
### Home Screen vs in-app labels
| Surface | Prefer | Reason |
|---------|--------|--------|
| In-app buttons, list rows, nav items | `tap_text("…")` | Label is usually the hit target |
| Home Screen app icons | `tap_icon("Weather")` | Label is **below** the icon; tapping the label is a no-op |
`tap_icon` (from `agent-workspace/agent_helpers.py`, auto-loaded) finds the label then taps at `(x, y - 35)`:
```python
tap_icon("Weather") # not tap_text("Weather") on Home Screen
```
## Wait: `wait_stable`
After any action that may animate (open app, navigate, dismiss sheet), settle before reading.
### Signature
```python
wait_stable(timeout=6.0, interval=0.5, settle=2) -> bool
```
| Param | Default | Meaning |
|-------|---------|---------|
| `timeout` | `6.0` | Max seconds to wait |
| `interval` | `0.5` | Sleep between captures |
| `settle` | `2` | Consecutive identical captures required |
### Behavior
1. Capture the window to PNG.
2. MD5 the file bytes.
3. If this digest matches the previous one, increment a streak; else reset.
4. Return `True` when the streak reaches `settle - 1` (i.e. `settle` identical frames with default `2`).
5. Return `False` if the deadline expires first.
Status-bar clock ticks once a minute, so identical-frame near-misses are rare. `open_app()` already calls `wait_stable()` after Spotlight launch.
```python
tap_text("Done")
if not wait_stable():
# animation still running or capture unstable — still re-read
pass
```
Also available: `wait(seconds=1.0)` for fixed sleeps when pixel settle is the wrong tool.
## Screenshot for unlabeled icons
OCR sees **text**, not semantics. Glyph-only controls need a visual pass.
Call `screenshot()` (optional path; default under the harness temp dir). Returns the PNG path.
Open the image (agent vision model or human). Pick the control in image pixels.
Use `screen_info()` for window bounds and capture size:
```python
info = screen_info()
# info["window"] = {x, y, w, h, id}
# info["img_px"] = [pixel_w, pixel_h]
# info["frontmost"] = bool
```
Map image pixel `(px, py)` to screen points:
```text
x = window["x"] + px * (window["w"] / img_px[0])
y = window["y"] + py * (window["h"] / img_px[1])
```
`tap(x, y)` then `wait_stable()` + `ocr()` / `screenshot()` again.
```bash
phone-harness <<'PY'
path = screenshot()
print(path)
print(screen_info())
# after choosing (px, py) from the image:
# info = screen_info()
# win, (iw, ih) = info["window"], info["img_px"]
# tap(win["x"] + px * win["w"] / iw, win["y"] + py * win["h"] / ih)
# wait_stable()
# print([o["text"] for o in ocr()][:15])
PY
```
Combine `screenshot()` with `ocr()` when you need both geometry context and text anchors.
## Verify: capture as ground truth
There is no DOM, no AX tree inside the stream, and no assert API. **After every action**, re-read:
1. `wait_stable()` — animation done (or timed out).
2. `ocr()` — confirm expected labels / absence of prior labels.
3. `screenshot()` — when the success signal is visual (icons, charts, DRM-free imagery).
```bash
phone-harness <<'PY'
ensure_mirroring()
before = {o["text"] for o in ocr()}
tap_text("Folders") # example in-app label
wait_stable()
after = {o["text"] for o in ocr()}
print("appeared:", sorted(after - before)[:20])
print("left:", sorted(before - after)[:20])
PY
```
Canonical agent loop from the product surface (`SKILL.md`):
```text
ocr() → decide → tap_text / tap → wait_stable() → ocr() / screenshot() → next step
```
If verification still shows the previous screen, do not assume the tap “worked.” Re-check focus/backend permissions, blocked interstitials, and Home Screen label-vs-icon targeting before retrying.
## End-to-end example
```bash
phone-harness <<'PY'
ensure_mirroring()
print(connection_state(), screen_info())
# See
labels = [o["text"] for o in ocr() if o["confidence"] >= 0.5]
print("visible:", labels[:15])
# Act (in-app style)
try:
hit = tap_text("Notes")
except RuntimeError as e:
print(e) # includes what OCR saw
raise
# Settle + verify
ok = wait_stable()
print("stable:", ok)
print([o["text"] for o in ocr()][:15])
PY
```
Home Screen launch pattern (agent helper):
```bash
phone-harness <<'PY'
ensure_mirroring()
home()
wait_stable()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:20]) # expect forecast chrome, not Home icons
PY
```
## Failure modes in this loop
| Symptom | Likely cause | Recovery |
|---------|--------------|----------|
| `tap_text` raises with long `saw: [...]` | Wrong label, wrong screen, or OCR miss | Read exception list; `screenshot()` if unlabeled |
| Tap “succeeds,” UI unchanged | Home Screen label hit; silent focus/Accessibility issue; blocked interstitial | Use `tap_icon` on Home; re-activate / doctor; `connection_state()` |
| `wait_stable` returns `False` | Long animation, capture thrash, or timeout too short | Increase `timeout`; still re-OCR; avoid treating settle as hard fail only |
| OCR empty / low confidence | Screen Recording, black DRM frame, interstitial | `--doctor`; avoid DRM video; check session |
| Wrong target after delay | Cached coordinates | Re-run `ocr()` / `find_text` immediately before `tap` |
Session and permission ladders: [Doctor diagnostics](/doctor-diagnostics), [Troubleshooting](/troubleshooting).
## Related helpers (out of scope here)
| Concern | Helpers | Docs |
|---------|---------|------|
| Capture math, Vision boxes, no-cache coords | `screenshot`, `ocr`, `screen_info`, backends | [Coordinates, capture, and OCR](/coordinates-capture-ocr) |
| App switch, Spotlight, keycodes | `home`, `app_switcher`, `open_app`, `type_text`, `press` | [Navigate apps and type text](/navigate-and-type) |
| Lists and end detection | `scroll_screen`, `scroll_until`, `scroll_collect` | [Scroll and collect lists](/scroll-lists) |
| Full signatures | All public helpers | [Helpers API](/helpers-api) |
## Next
Screen-point space, window bounds vs image pixels, and OCR box shape.
home, Spotlight open_app, type_text constraints, and tap_icon vs tap_text.
Signatures, defaults, return shapes, and errors for every public helper.
Silent taps, blank capture, blocked session, and label misses.
---
## 09. Navigate apps and type text
> home, app_switcher, open_app via Spotlight, press key combos, type_text US keycode constraints, and Home Screen tap_icon versus in-app tap_text.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/09-navigate-apps-and-type-text.md
- Generated: 2026-08-10T20:16:11.904Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `agent-workspace/agent_helpers.py`
- `SKILL.md`
---
title: "Navigate apps and type text"
description: "home, app_switcher, open_app via Spotlight, press key combos, type_text US keycode constraints, and Home Screen tap_icon versus in-app tap_text."
---
Navigation and typing in phone-harness are thin helpers over the active input backend (`src/phone_harness/background.py` by default, or `src/phone_harness/mirror.py` when background load fails or `PHONE_HARNESS_BACKGROUND` is falsey). Core shortcuts live in `src/phone_harness/helpers.py`; `press` and `type_text` are re-exported from the backend; Home Screen icon launching is the agent helper `tap_icon` in `agent-workspace/agent_helpers.py`.
## Prerequisites
- Session is `ready` (`ensure_mirroring()` / `connection_state()`).
- iPhone Mirroring shortcuts map as the Mac app defines them: **Cmd+1** Home, **Cmd+2** App Switcher, **Cmd+3** Spotlight.
- For typing: an iOS text field (or Spotlight search field) must already be focused; `type_text` only posts keycodes.
## Navigation helpers
| Helper | Action | Implementation |
| --- | --- | --- |
| `home()` | Home Screen | `press("cmd+1")`, then `sleep(0.8)` |
| `app_switcher()` | Multitasking switcher | `press("cmd+2")`, then `sleep(0.8)` |
| `open_app(name)` | Open app by name via Spotlight | `cmd+3` → type name → wait → `return` → `wait_stable()` |
These helpers do not verify the resulting UI. After navigation, call `wait_stable()` (already used by `open_app`) and re-read with `ocr()` or `screenshot()`.
### `home()`
```python
home() # press("cmd+1"); sleep(0.8)
```
Use before Home Screen work (icon launch, page swipe). Does not confirm Home is visible.
### `app_switcher()`
```python
app_switcher() # press("cmd+2"); sleep(0.8)
```
Opens the app switcher. Selecting an app still requires a separate `tap` / `tap_text` on the card.
### `open_app(name)`
Spotlight path for launching by name:
```text
press("cmd+3")
sleep(0.9)
type_text(name)
sleep(1.2) # let Spotlight results populate
press("return")
wait_stable()
```
| Parameter | Type | Notes |
| --- | --- | --- |
| `name` | string | Typed with `type_text` (US keycode set only) |
```bash
phone-harness <<'PY'
ensure_mirroring()
open_app("Notes")
print([o["text"] for o in ocr()][:15])
PY
```
`open_app` always commits the first Spotlight hit with Return. Use a distinctive `name` (for example `"Settings"` not a single letter). Characters outside the US keycode map raise before Return is pressed.
Prefer `open_app` when the app is not on the current Home Screen page. Prefer `home()` + `tap_icon(label)` when the icon is visible and you want a direct Home Screen launch.
## `press(combo)`
Posts a single key or modifier chord through the active backend.
| Backend | Behavior |
| --- | --- |
| Classic (`mirror.py`) | `activate()` then `CGEventCreateKeyboardEvent` + `CGEventPost` at HID |
| Background (`background.py`) | Make window key via SkyLight event record, then `CGEventPostToPid` (no frontmost steal for the keystroke path) |
### Signature and grammar
```python
press(combo) # e.g. press("return"), press("cmd+1"), press("cmd+3")
```
- Split on `+` (lowercased): last segment is the key; earlier segments are modifiers.
- Unknown key → `ValueError(f"unknown key {key!r}")`.
- Unknown modifier → `KeyError` (modifier not in `_MODIFIERS`).
### Known keys (`_KEYCODES`)
| Category | Keys |
| --- | --- |
| Control | `return` / `enter`, `tab`, `space`, `delete` / `backspace`, `escape` / `esc` |
| Arrows | `left`, `right`, `down`, `up` |
| Digits | `0`–`9` |
| Letters | `a`–`z` (lowercase key names) |
### Known modifiers (`_MODIFIERS`)
| Name | Flag |
| --- | --- |
| `cmd` | Command |
| `shift` | Shift |
| `alt` / `option` | Alternate |
| `ctrl` | Control |
Navigation chords used by core helpers:
| Combo | Meaning |
| --- | --- |
| `cmd+1` | Home Screen |
| `cmd+2` | App Switcher |
| `cmd+3` | Spotlight |
```python
press("return")
press("cmd+1")
press("escape")
```
## `type_text(text, delay=0.03)`
Types into the **already focused** iOS field via real HID keycodes (US layout). iPhone Mirroring forwards raw keycodes and ignores unicode payloads on keyboard events, so emoji and many non-US characters cannot be typed.
### Behavior
1. Split `text` on `\n`. Between lines, press Return.
2. For each character, resolve `(keycode, needs_shift)` via `_keycode_for`.
3. Missing keycode → `ValueError(f"cannot type {ch!r} via keycodes")`.
4. Per-character delay defaults to `0.03` seconds after each key up/down pair.
| Parameter | Default | Role |
| --- | --- | --- |
| `text` | required | String to type; `\n` becomes Return |
| `delay` | `0.03` | Pause after each character |
### Typable set (US)
| Class | Supported |
| --- | --- |
| Letters | `a`–`z`, `A`–`Z` (shift) |
| Digits | `0`–`9` |
| Shifted digit symbols | `! @ # $ % ^ & * ( )` |
| Punctuation | `. , / ; ' [ ] \ - = \`` and shifted forms `_ + : " < > ? ~ { } \|` |
| Space | `" "` |
| Newline | `\n` → Return |
Not supported: emoji, non-Latin scripts, combining marks, and any glyph without a US keycode entry.
```python
# Field must already be focused
tap_text("Title")
wait_stable()
type_text("hello from the harness")
type_text("line one\nline two") # Return between lines
```
`type_text` does not open a field or wait for the software keyboard. Tap the field first, wait until the UI is stable, then type. Typing with no focused field is a silent no-op or lands in the wrong surface.
## Home Screen `tap_icon` versus in-app `tap_text`
| Surface | Helper | Target | Notes |
| --- | --- | --- | --- |
| Home Screen app icon | `tap_icon(label, index=0)` | Label OCR center, then **y − 35** | Label text is not the launch hit target |
| In-app button / row / control | `tap_text(query, index=0, exact=False)` | OCR box center | Label and control share the same center |
### Why labels fail on the Home Screen
OCR returns the caption under the icon. Tapping that center hits the label, not the icon; launch is a no-op. The agent helper offsets upward:
```python
# agent-workspace/agent_helpers.py
def tap_icon(label, index=0):
hits = find_text(label)
if not hits:
raise RuntimeError(f"no Home-Screen label matching {label!r}")
h = hits[index]
tap(h["x"], h["y"] - 35)
return h
```
| Parameter | Default | Role |
| --- | --- | --- |
| `label` | required | Case-insensitive substring match via `find_text` |
| `index` | `0` | Which match when several labels match |
Raises `RuntimeError` if no OCR hit matches.
`tap_icon` is loaded from `agent-workspace/agent_helpers.py` into the script namespace at helper import time (override workspace with `PH_AGENT_WORKSPACE`). Edit that file for task-specific variants; see [Extend agent helpers](/extend-agent-helpers).
### In-app: use `tap_text`
```python
tap_text("New Note") # substring match
tap_text("Done", exact=True) # exact string
```
On miss, `tap_text` raises with up to 30 currently visible OCR strings so the next step can replan.
## Choose a launch path
```text
Need an app open?
├── Name known, icon may be off-screen → open_app("AppName")
├── Icon visible on Home Screen → home(); tap_icon("AppName")
└── Already inside an app UI → tap_text(...) / tap(x, y)
```
```bash
phone-harness <<'PY'
ensure_mirroring()
home()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:12])
PY
```
```bash
phone-harness <<'PY'
ensure_mirroring()
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
wait_stable()
print([o["text"] for o in ocr()][:10])
PY
```
## Verify after navigation and typing
There is no accessibility tree inside the mirroring window. After `home`, `open_app`, `tap_icon`, or `type_text`:
1. `wait_stable()` (or rely on the settle already in `open_app`)
2. `ocr()` or `screenshot()` for ground truth
3. Retry or replan from what is actually visible
## Failure modes
| Symptom | Likely cause | Recovery |
| --- | --- | --- |
| `ValueError: cannot type '…' via keycodes` | Character outside US keycode map | Use only US typable characters; avoid emoji |
| `ValueError: unknown key '…'` | Bad `press` key name | Use keys from `_KEYCODES` |
| Home Screen tap does nothing | Used `tap_text` on icon label | Use `tap_icon` (~35 pt above label) |
| `RuntimeError: no Home-Screen label matching` | OCR miss / wrong page | `home()`, swipe pages, `screenshot()`, retry label |
| Typed text missing | Field not focused | Tap field, `wait_stable()`, then `type_text` |
| Spotlight opens wrong app | Ambiguous name / first result | More specific `name` |
| Silent no-op on classic backend | Window lost frontmost focus | Re-`activate()` / re-`ensure_mirroring()` |
## Related pages
OCR-first read loop, `tap` / `tap_text`, `wait_stable`, and capture-as-ground-truth after every action.
Full signatures for navigation, timing, gestures, and re-exported backend primitives.
Edit `agent_helpers.py`, `PH_AGENT_WORKSPACE`, and the `tap_icon` Home Screen pattern.
Background SkyLight path versus classic mirror HID + activate, and env selection.
Hard limits including keycode-only typing and when not to drive the phone.
Home Screen label misses, `type_text` field focus, blocked session, and silent input.
---
## 10. Scroll and collect lists
> scroll_screen movement detection, scroll_until predicates, scroll_collect de-dup extraction, wheel versus flick behavior, settle windows, and stop reasons reached-end and max-scrolls.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/10-scroll-and-collect-lists.md
- Generated: 2026-08-10T20:16:53.586Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `src/phone_harness/ocr.py`
- `SKILL.md`
---
title: "Scroll and collect lists"
description: "scroll_screen movement detection, scroll_until predicates, scroll_collect de-dup extraction, wheel versus flick behavior, settle windows, and stop reasons reached-end and max-scrolls."
---
`scroll_screen`, `scroll_until`, and `scroll_collect` in `src/phone_harness/helpers.py` walk iOS lists through the iPhone Mirroring window by scrolling, waiting for a settle window, then judging end-of-list from **OCR text-set overlap** (whether the screen moved), not from whether your extractor found new rows. They call `mirror.scroll_wheel` on the active input backend (`background` by default, `mirror` when background is disabled or fails to load).
## Design rule: movement, not parser output
End-of-list is decided only by whether consecutive content OCR text sets still look like the same screen after a settle window. A dense list, a missed OCR line, or a slow lazy-load must not stop the walk early.
| Signal | Role |
|--------|------|
| Content OCR text set | Before/after comparison input (status bar and bottom chrome cropped out) |
| Jaccard overlap | `len(a ∩ b) / len(a ∪ b)` — ~1.0 same screen, low means it moved |
| `moved` | `overlap < moved_thresh` (default `0.6`) |
| Extractor / `done` | Collects or stops on a match; never ends the walk alone |
Empirical band used by the default threshold: real forward progress often lands under ~0.45 overlap; overscroll bounce at a boundary often sits above ~0.7 and would otherwise look like movement.
## Choose the right gesture
| Helper | Mechanism | Use for |
|--------|-----------|---------|
| `scroll_screen` / `scroll_until` / `scroll_collect` | `scroll_wheel` via the active backend | Long lists, feeds, Settings rows |
| `scroll(amount=300)` | One `scroll_wheel` at window center | Simple one-shot nudge |
| `swipe(direction, distance=0.4)` | Fast short `drag` (`duration=0.12`, `steps=6`) | Home Screen pages, carousels, momentum snaps |
A **slow** touch-drag barely moves an iOS list and often bounces back. List helpers intentionally avoid that path.
### Wheel (classic backend) versus flick (background backend)
Both backends expose the same `scroll_wheel(dy, x, y, steps=...)` name, but they do different work:
| Backend | `PHONE_HARNESS_BACKGROUND` | What `scroll_wheel` actually does |
|---------|----------------------------|-----------------------------------|
| `mirror.py` | `0` / `false` / `no`, or background import failure | Real `CGEventCreateScrollWheelEvent` pixel-wheel events after focusing the window |
| `background.py` | default `1` | **Fast vertical flick** (left-mouse drag), not a Mac wheel event |
Background reason (verified in code comments): Mac scroll-wheel events only reach iPhone Mirroring when the app is active, so background wheel posts produce ~0% movement. A slow drag also barely advances a list. A **fast** flick supplies release velocity and advances content (~29% frame change per flick in the author’s check).
Sign convention (shared by list helpers):
- `direction="up"` → negative `dy` → reveal content **below** (finger flicks up / content moves up)
- `direction="down"` → positive `dy` → reveal content **above**
Background flick geometry (fraction of window height):
- `dy < 0`: drag from `0.72h` → `0.28h`
- `dy ≥ 0`: drag from `0.28h` → `0.72h`
- Inter-step sleep: `0.006s` (velocity is the load-bearing part)
List helpers always call `scroll_wheel(..., steps=10)` with `dy = sign * int(window_height * amount)`.
## Content OCR crop
Before movement checks and collection, helpers OCR only the scrollable middle of the window via `_content_texts`:
| Parameter | Default | Purpose |
|-----------|---------|---------|
| `min_conf` | `0.4` | Drop low-confidence Vision boxes |
| `top_frac` | `0.06` | Exclude status bar (clock / battery) |
| `bottom_frac` | `0.92` | Exclude home indicator / bottom nav strip |
Boxes must satisfy `top < y < bot` in **global screen points**. Status-bar text changing every minute must not flip “settled” or “moved.”
## API surface
### `scroll_screen` — one step + settle + movement verdict
```python
scroll_screen(direction="up", amount=0.6, settle=2.5, moved_thresh=0.6)
```
`"up"` or `"down"`. Any other value raises `ValueError`.
Fraction of window height passed into `scroll_wheel` as pixel magnitude. Prefer values `< 1.0` so consecutive screens overlap and rows are not skipped between captures.
Seconds allowed for the post-scroll settle loop (lazy-load spinner / new rows).
Jaccard threshold: `moved` is `True` when `overlap < moved_thresh`.
**Sequence**
1. Snapshot `before` = content text set.
2. `scroll_wheel` at window center with `steps=10`.
3. Sleep `0.4s`.
4. Poll content OCR every `0.35s` until two consecutive identical text sets **or** the settle deadline.
5. Compare `before` vs settled `after`.
**Return shape**
| Field | Type | Meaning |
|-------|------|---------|
| `moved` | `bool` | Screen advanced past `moved_thresh` |
| `overlap` | `float` | Jaccard overlap, rounded to 3 decimals |
| `before` | `frozenset[str]` | Pre-scroll content texts |
| `after` | `frozenset[str]` | Settled post-scroll content texts |
| `boxes` | `list[dict]` | Settled content OCR boxes (ready for extract / `done`) |
```python
res = scroll_screen("up", amount=0.6)
print(res["moved"], res["overlap"], len(res["boxes"]))
```
### `scroll_until` — predicate stop or confirmed end
```python
scroll_until(done, direction="up", amount=0.6, max_scrolls=60, settle=2.5)
```
`done(boxes) -> truthy | falsy`. Receives current content OCR boxes. Truthy return value is returned to the caller.
Hard cap on scroll steps after the initial pre-check.
**Stop outcomes**
| Outcome | Return value |
|---------|----------------|
| `done(boxes)` truthy on current screen (including before any scroll) | That truthy value |
| Two consecutive non-moving scrolls (`stale >= 2`) after settle | `None` |
| `max_scrolls` exhausted without hit or confirmed end | `None` |
On a non-moving step that has not yet hit `stale >= 2`, the helper sleeps `0.8s`, calls `mirror.activate()`, and retries once so a stalled focus/lazy-load does not false-end the walk.
```python
hit = scroll_until(
lambda boxes: next((b for b in boxes if "Wi-Fi" in b["text"]), None)
)
if hit:
tap(hit["x"], hit["y"])
```
### `scroll_collect` — de-duped extraction to true end
```python
scroll_collect(
extract=None,
key=None,
direction="up",
amount=0.6,
max_scrolls=400,
end_after=3,
settle=2.5,
on_progress=None,
)
```
| Parameter | Default | Behavior |
|-----------|---------|----------|
| `extract` | strip non-empty `text` from each box | `extract(boxes) -> list[item]` for one screen |
| `key` | identity | Hashable de-dup key; first occurrence wins, order preserved |
| `direction` | `"up"` | Same sign convention as `scroll_screen` |
| `amount` | `0.6` | Keep `< 1.0` so screens overlap |
| `max_scrolls` | `400` | Hard cap |
| `end_after` | `3` | Consecutive non-moving scrolls required for `reached-end` |
| `settle` | `2.5` | Passed to each `scroll_screen` |
| `on_progress` | `None` | Optional `on_progress(i, total_items, new_count, moved, overlap)` |
**Algorithm**
1. Ingest the current screen (no scroll yet).
2. For `i` in `1..max_scrolls`: `scroll_screen` → ingest settled boxes → optional progress callback.
3. If `moved`: reset stale counter to `0`.
4. If not `moved`: increment stale; when `stale >= end_after`, stop with `reached-end` (after the same `0.8s` + `activate()` grace used on intermediate stalls).
5. If the loop finishes, stop with `max-scrolls`.
**Return shape**
```python
{"items": [...], "stop": "reached-end" | "max-scrolls", "scrolls": int}
```
| Field | Meaning |
|-------|---------|
| `items` | De-duped items in first-seen order |
| `stop` | Why the walk ended |
| `scrolls` | Number of scroll steps performed (`i` on early exit, or `max_scrolls`) |
## Stop reasons
```text
┌────────────────────┐
│ ingest screen 0 │
└─────────┬──────────┘
▼
┌────────────────────┐ moved=True ┌──────────────┐
│ scroll_screen step │ ──────────────────► │ stale = 0 │──► next step
└─────────┬──────────┘ └──────────────┘
│ moved=False
▼
stale += 1; sleep 0.8; activate()
│
├── stale >= end_after (collect: 3) or >= 2 (until)
│ → stop "reached-end" / return None
│
└── i hits max_scrolls
→ stop "max-scrolls" / return None
```
| Reason | API | Meaning |
|--------|-----|---------|
| `reached-end` | `scroll_collect` | `end_after` consecutive non-moving scrolls after settle |
| `max-scrolls` | `scroll_collect` | Hit `max_scrolls` without that consecutive still streak |
| predicate hit | `scroll_until` | Returns `done(...)` truthy value |
| confirmed still / cap | `scroll_until` | Returns `None` (`stale >= 2` or `max_scrolls`) |
**Important:** zero *new extracted items* on a step is normal (overlap de-dup). That does **not** end the walk. Only `moved == False` for enough consecutive steps does.
## Examples
### Collect every visible text line
```bash
phone-harness <<'PY'
result = scroll_collect()
print(result["stop"], result["scrolls"], len(result["items"]))
print(result["items"][:20])
PY
```
### Structured rows with a stable key
```bash
phone-harness <<'PY'
def extract(boxes):
# Example: keep lines that look like settings rows
return [b["text"].strip() for b in boxes if b["text"].strip()]
result = scroll_collect(
extract=extract,
key=lambda s: s.lower(),
amount=0.55,
end_after=3,
on_progress=lambda i, n, new, moved, ov:
print(f"step={i} total={n} new={new} moved={moved} overlap={ov}"),
)
assert result["stop"] in ("reached-end", "max-scrolls")
print(result["stop"], len(result["items"]))
PY
```
### Stop when a label appears, then tap it
```bash
phone-harness <<'PY'
def done(boxes):
for b in boxes:
if "Airplane Mode" in b["text"]:
return b
return None
row = scroll_until(done, direction="up", amount=0.6, max_scrolls=40)
if row is None:
raise SystemExit("label not found before end of list")
tap(row["x"], row["y"])
wait_stable()
print([o["text"] for o in ocr()][:15])
PY
```
### Single-step inspection
```bash
phone-harness <<'PY'
r = scroll_screen("up", amount=0.6, settle=2.5)
print({"moved": r["moved"], "overlap": r["overlap"], "n_boxes": len(r["boxes"])})
PY
```
## Defaults cheat sheet
| Constant / default | Value | Where |
|--------------------|-------|--------|
| Content OCR confidence | `0.4` | `_content_texts` |
| Top crop | `6%` of height | `_content_texts` |
| Bottom crop | below `92%` of height | `_content_texts` |
| Scroll amount | `0.6` window heights | list helpers |
| Settle window | `2.5s` | list helpers |
| Moved threshold | `0.6` Jaccard | `scroll_screen` |
| Post-scroll sleep | `0.4s` | `scroll_screen` |
| Settle poll interval | `0.35s` | `scroll_screen` |
| Non-move grace | `0.8s` + `activate()` | `scroll_until` / `scroll_collect` |
| End confirmation (`until`) | `2` consecutive stills | `scroll_until` |
| End confirmation (`collect`) | `3` consecutive stills (`end_after`) | `scroll_collect` |
| `scroll_until` cap | `60` | `max_scrolls` |
| `scroll_collect` cap | `400` | `max_scrolls` |
| Wheel/flick steps from lists | `10` | `scroll_screen` → `scroll_wheel` |
## Failure modes and recovery
| Symptom | Likely cause | What to do |
|---------|--------------|------------|
| `stop="max-scrolls"` with still-growing UI | List never went fully still (ads, live clocks inside content crop, infinite feed) | Raise `max_scrolls`, tighten `extract`/`key`, or use `scroll_until` with an explicit stop label |
| Early `reached-end` / `None` | Bounce misread as still, or settle too short for lazy-load | Increase `settle`; keep default `moved_thresh=0.6`; ensure session is `ready` |
| No rows advance, overlap stays high | Wrong surface (non-scrollable screen), blocked interstitial, or input not landing | `connection_state()` / `ensure_mirroring()`; check Accessibility; see input backends |
| Missing rows between screens | `amount` too large (no overlap) | Use `amount` in the ~0.5–0.7 range |
| Collect empty / sparse | Content is icon-only or OCR confidence below `0.4` | Custom `extract` after inspecting `scroll_screen()["boxes"]`; fall back to `screenshot()` for unlabeled chrome |
| `swipe` used on a long list | Slow/short drag path is for pages, not lists | Use `scroll_collect` / `scroll_screen` |
| Background session “scrolls” poorly | Expecting Mac wheel semantics | Background path is a **flick**; classic path is real wheel — both share the helper API |
Never end a list walk because `new == 0` on one screen. De-dup and dense OCR make empty ingest normal. Trust `moved` / stop reasons only.
## Backend selection (affects transport only)
List helper signatures and return shapes are backend-agnostic. Transport selection:
- Default: `PHONE_HARNESS_BACKGROUND=1` → `background.scroll_wheel` (fast flick)
- Force classic: `PHONE_HARNESS_BACKGROUND=0|false|no` → `mirror.scroll_wheel` (HID wheel + focus)
- Import failure of the background module falls back to classic automatically
Keyboard helpers may still activate the window on the background backend; mouse scroll/flick does not require focus steal on that path.
## Related pages
OCR-first reading, wait_stable, and capture-as-ground-truth after each action.
Background SkyLight events versus classic HID wheel, and PHONE_HARNESS_BACKGROUND.
Screen-point boxes, confidence, and why content crop uses global coordinates.
Full helper signatures, defaults, and return shapes including the scroll family.
ready / blocked gates before any scroll walk.
Silent input, blocked mirroring, and OCR miss recovery.
---
## 11. Extend agent helpers
> Edit agent-workspace/agent_helpers.py for task-specific primitives, PH_AGENT_WORKSPACE override, auto-load into the script namespace, and the tap_icon Home Screen pattern.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/11-extend-agent-helpers.md
- Generated: 2026-08-10T20:16:31.414Z
### Source Files
- `agent-workspace/agent_helpers.py`
- `src/phone_harness/helpers.py`
- `src/phone_harness/run.py`
- `SKILL.md`
- `README.md`
---
title: "Extend agent helpers"
description: "Edit agent-workspace/agent_helpers.py for task-specific primitives, PH_AGENT_WORKSPACE override, auto-load into the script namespace, and the tap_icon Home Screen pattern."
---
Task-specific phone primitives live in `agent-workspace/agent_helpers.py`. On import, `phone_harness.helpers` loads that file (or the path from `PH_AGENT_WORKSPACE`) and injects every public name into the helper module globals. The CLI then execs stdin Python with those names already in scope, so new functions work the same way as core helpers without changing `src/phone_harness/`.
## Ownership boundary
| Layer | Path | Who edits |
| --- | --- | --- |
| Protected core | `src/phone_harness/helpers.py` and siblings | Package maintainers; pre-imported primitives |
| Agent workspace | `agent-workspace/agent_helpers.py` | Agents and users; task-specific helpers |
| CLI namespace | `phone_harness.run` | Builds the exec dict from public helper names |
Core helpers stay thin and stable. When a flow needs a reusable pattern (Home Screen icons, a multi-step navigation, a project-specific OCR filter), define it in the agent workspace rather than forking the package.
:::files
repo/
├── agent-workspace/
│ └── agent_helpers.py # edit here; auto-loaded
└── src/phone_harness/
├── helpers.py # _load_agent_helpers() at import
└── run.py # exec(stdin, non-_ helpers)
:::
## Load path and `PH_AGENT_WORKSPACE`
```text
REPO_ROOT = # parent of src/
AGENT_WORKSPACE = $PH_AGENT_WORKSPACE
| default: REPO_ROOT/agent-workspace
load file = AGENT_WORKSPACE/agent_helpers.py
```
| Item | Value |
| --- | --- |
| Env var | `PH_AGENT_WORKSPACE` |
| Default | `/agent-workspace` (resolved from `helpers.py` → `REPO_ROOT`) |
| Loaded file | `$PH_AGENT_WORKSPACE/agent_helpers.py` |
| Module name (importlib) | `phone_harness_agent_helpers` |
Canonical install keeps the tree at `~/.phone-harness`, so the default workspace is `~/.phone-harness/agent-workspace`. Point `PH_AGENT_WORKSPACE` at another directory when you want a per-project helper set without relocating the package.
Directory that contains `agent_helpers.py`. Not the file path itself. Unset → `/agent-workspace`.
## Auto-load rules
`_load_agent_helpers()` runs once at the end of `helpers` import:
1. Resolve `p = AGENT_WORKSPACE / "agent_helpers.py"`.
2. If the file does not exist, return silently (core helpers still work).
3. Load via `importlib.util.spec_from_file_location` / `exec_module`.
4. For each `name, value` in `vars(module)` where `name` does **not** start with `_`, assign `globals()[name] = value` on the helpers module.
The CLI then mirrors that public surface into the script namespace:
```python
from . import helpers
g = {k: v for k, v in vars(helpers).items() if not k.startswith("_")}
g["__name__"] = "__main__"
exec(code, g)
```
Each `phone-harness` process is self-contained. Edits to `agent_helpers.py` apply on the next invocation; there is no in-process hot reload.
### What becomes available
| Defined in `agent_helpers.py` | In `phone-harness` scripts? |
| --- | --- |
| `def tap_icon(...)` | Yes |
| `FOO = 1` | Yes |
| `_private_helper` | No (leading `_` filtered) |
| Imports used only inside the file | Not re-exported as names unless bound at module top level |
Names from agent helpers can shadow core helper names if they collide. Prefer new identifiers.
## Shipped example: `tap_icon`
Home Screen app labels are OCR-visible but not the touch target. Tapping the label center is a no-op; the icon hit box sits about **35 screen points above** the label. The stock agent helper encodes that offset.
```python
# agent-workspace/agent_helpers.py
def tap_icon(label, index=0):
"""Tap a Home-Screen app icon by its label."""
from phone_harness.helpers import find_text, tap
hits = find_text(label)
if not hits:
raise RuntimeError(f"no Home-Screen label matching {label!r}")
h = hits[index]
tap(h["x"], h["y"] - 35)
return h
```
### Signature
| Parameter | Type | Default | Role |
| --- | --- | --- | --- |
| `label` | str | required | Substring match via `find_text` (case-insensitive) |
| `index` | int | `0` | Which OCR hit when several labels match |
### Behavior
| Step | Action |
| --- | --- |
| Find | `find_text(label)` → list of OCR hits with `x`, `y`, … |
| Fail | No hits → `RuntimeError(f"no Home-Screen label matching {label!r}")` |
| Tap | `tap(h["x"], h["y"] - 35)` |
| Return | The selected hit dict `h` |
### Home Screen vs in-app
| Situation | Use |
| --- | --- |
| Home Screen app icon | `tap_icon("Weather")` |
| In-app button, row, or labeled control | `tap_text("New Note")` (core helper; taps text center) |
SKILL guidance: `tap_text("Weather")` on the Home Screen hits the caption and does nothing; use `tap_icon` there.
### Example
```bash
phone-harness <<'PY'
home()
wait_stable()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:15])
PY
```
## How to add a helper
Edit `agent-workspace/agent_helpers.py` under the install root (or `$PH_AGENT_WORKSPACE/agent_helpers.py` if overridden).
Use a non-`_` name. Import core primitives inside the function or at module top as needed:
```python
def dismiss_keyboard():
from phone_harness.helpers import press, wait_stable
press("return") # or a known Done/Hide label via tap_text
wait_stable()
```
Module import runs on every `phone-harness` start. Do not open apps, tap, or capture at import time—only define callables and constants.
```bash
phone-harness <<'PY'
print("dismiss_keyboard" in dir()) # or call your helper
print(tap_icon) # stock helper still present
PY
```
### Patterns that fit the workspace
- **Offset / geometry quirks** — like `tap_icon`’s −35 pt Home Screen correction.
- **Multi-step flows** — e.g. open app → wait → assert OCR, wrapped as one name.
- **Domain extractors** — OCR filters and parsers reused across scroll_collect calls.
- **Project conventions** — fixed app names, safe consent gates before outward actions.
Prefer composing `ocr`, `find_text`, `tap`, `wait_stable`, navigation helpers, and scroll helpers over reimplementing capture or HID input.
## Constraints and failure modes
| Symptom | Cause | Fix |
| --- | --- | --- |
| `NameError: name '…' is not defined` | Helper not loaded (wrong path, leading `_`, or syntax error at import) | Confirm file path; avoid `_` prefix; fix import-time errors |
| Stock `tap_icon` missing | Workspace file missing or replaced without re-exporting it | Restore `agent_helpers.py` or re-add the function |
| Edits seem ignored | Different `PH_AGENT_WORKSPACE` or non-editable install copy | Check env; edit the tree bound by `pip install -e` (often `~/.phone-harness`) |
| Home Screen launch fails with `tap_text` | Label is not the icon hit target | Use `tap_icon` |
| `no Home-Screen label matching …` | OCR miss, wrong page, or app name mismatch | `home()`, swipe pages, print `ocr()` / `find_text` results; pass `index` if duplicates |
Agent helpers run with the same privileges as core helpers: full screen capture and input on the real phone. Do not encode irreversible outward actions (send, purchase, delete) without an explicit consent gate in the calling script.
## Relation to core API
Core primitives stay in `helpers.py` (`connection_state`, `ocr`, `tap_text`, `scroll_collect`, `open_app`, …). Agent helpers do not replace that module; they extend the same namespace for scripts and for anything that does `from phone_harness import helpers` after import-time load.
For the full public surface of core helpers, see the Helpers API page. For `PH_AGENT_WORKSPACE` among other env vars, see Environment variables.
## Related pages
Core pre-imported helpers: signatures, defaults, return shapes, and errors.
home, open_app, type_text, and when to use tap_icon versus tap_text.
PH_AGENT_WORKSPACE path override and other runtime env keys.
stdin exec model that exposes helper names in script scope.
OCR-first loop and wait_stable verification after custom helpers.
Skill text that points agents at agent-workspace for task-specific edits.
---
## 12. Register as an agent skill
> phone-harness skill output, Claude Code and Codex skill install paths, trigger text, and re-sync after pulls so the agent auto-reaches for phone control.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/12-register-as-an-agent-skill.md
- Generated: 2026-08-10T20:16:45.351Z
### Source Files
- `install.md`
- `SKILL.md`
- `src/phone_harness/run.py`
- `README.md`
- `pyproject.toml`
---
title: "Register as an agent skill"
description: "phone-harness skill output, Claude Code and Codex skill install paths, trigger text, and re-sync after pulls so the agent auto-reaches for phone control."
---
`phone-harness skill` is a CLI subcommand that prints the repository root `SKILL.md` to stdout. Install that body under a skill directory named `phone-harness` so Claude Code or Codex can match the skill description and invoke `phone-harness` for real iPhone control.
## What gets registered
| Field | Value |
| --- | --- |
| Skill name | `phone-harness` |
| Body source | repo-root `SKILL.md` |
| Emit command | `phone-harness skill` |
| Trigger / description | `Control the user's iPhone through the Mac's iPhone Mirroring window: open apps, tap, type, swipe, read the screen.` |
The skill file frontmatter is:
```yaml
---
name: phone-harness
description: "Control the user's iPhone through the Mac's iPhone Mirroring window: open apps, tap, type, swipe, read the screen."
---
```
Agents match on that description text. The body is the day-to-day product surface: when not to use the phone, stdin Python usage with pre-imported helpers, screen workflow (OCR → act → verify), consent rules, connection gating, and gotchas.
## CLI surface
```text
phone-harness skill print the phone-harness skill text
```
Behavior:
1. Resolve the package file `src/phone_harness/run.py`.
2. Walk three parents to the checkout root.
3. Read `{repo_root}/SKILL.md` as UTF-8.
4. Print the full file to stdout with no trailing extra newline (`end=""`; the file already ends with `\n`).
No flags, no args, no write path — registration is always a shell redirect into a skill directory.
```text
~/.phone-harness/ (canonical editable checkout)
├── SKILL.md ← source of truth for skill body
├── install.md
├── agent-workspace/
└── src/phone_harness/
└── run.py ← `phone-harness skill` reads ../../../SKILL.md
```
Because install is editable (`pip install -e .` at `~/.phone-harness`), the installed entry point still resolves `SKILL.md` from that checkout root.
## Prerequisites
Before registration:
1. Checkout at the canonical home: `~/.phone-harness` (or re-run `pip install -e .` if relocated).
2. `phone-harness` on `PATH` via editable install.
3. Day-to-day phone work still needs permissions and mirroring; skill registration alone does not grant Accessibility or Screen Recording.
Registration only installs agent discovery text. Runtime control still requires a working `phone-harness` command, iPhone Mirroring, and the permission ladder verified by `phone-harness --doctor`.
## Install paths
Register under the agent’s skill root, in a folder named `phone-harness`, as `SKILL.md`.
```bash
mkdir -p ~/.claude/skills/phone-harness
```
```bash
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
```
```bash
head -5 ~/.claude/skills/phone-harness/SKILL.md
# ---
# name: phone-harness
# description: "Control the user's iPhone through the Mac's iPhone Mirroring window: open apps, tap, type, swipe, read the screen."
# ---
```
```bash
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness"
```
```bash
phone-harness skill > "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"
```
`CODEX_HOME` defaults to `$HOME/.codex` when unset.
```bash
head -5 "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"
```
```bash
mkdir -p ~/.claude/skills/phone-harness
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness"
phone-harness skill > "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"
```
| Target | Directory | File |
| --- | --- | --- |
| Claude Code | `~/.claude/skills/phone-harness/` | `SKILL.md` |
| Codex | `${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/` | `SKILL.md` |
Skill packs here are plain files under local skill directories. They are not a model-provider package, hosted connector, or BYOK secret. Any agent that loads skills from a filesystem path can use the same `phone-harness skill` body by writing it into its own skill root.
## Trigger text
Agents should auto-select the skill when the user asks for iPhone control that maps to the description:
```text
Control the user's iPhone through the Mac's iPhone Mirroring window: open apps,
tap, type, swipe, read the screen.
```
Typical match intents:
- Open or navigate iOS apps through mirroring
- Tap, type, swipe, or scroll on the phone
- Read the mirrored screen (OCR / capture)
The skill body also steers agents **away** from phone use when a Mac or web path exists, and toward consent stops for outward-facing or irreversible actions.
## What the skill body tells the agent
| Topic | Instruction surface in `SKILL.md` |
| --- | --- |
| Invoke | `phone-harness <<'PY' … PY` with helpers pre-imported |
| Coordinates | Global screen points; do not cache across calls |
| Eyes | Prefer `ocr()`; `screenshot()` for unlabeled icons |
| Act | `tap_text`, `tap`, `swipe`, `scroll_*`, `type_text`, `press`, navigation helpers |
| Verify | `wait_stable()` then re-capture after every action |
| Connection | User physical action only; never tap Connect/Continue or poll |
| Consent | Ask before send/post/purchase/delete/settings changes |
| Task-specific code | Edit `agent-workspace/agent_helpers.py` (e.g. `tap_icon`) |
| Setup failures | Read `install.md` |
## Re-sync after pulls
Registered copies are **snapshots**, not live links to the checkout.
After `git pull` (or any edit to repo `SKILL.md`):
```bash
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
phone-harness skill > "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"
```
Re-run only the targets you use. Until you re-sync, agents keep the old trigger text and workflow notes.
## Setup prompt integration
The README setup prompt asks the agent to:
1. Clone to `~/.phone-harness` and follow `install.md`
2. Put `phone-harness` on `PATH`
3. Register skill name `phone-harness` with body from `phone-harness skill`
4. Read `SKILL.md` and `src/phone_harness/helpers.py` for normal use
5. Verify with `./phone-harness --doctor` (or `phone-harness --doctor` after install)
Registration is part of install, not a separate product mode.
## Failure modes
| Symptom | Cause | Fix |
| --- | --- | --- |
| `phone-harness: command not found` | Not on `PATH` / not installed | `cd ~/.phone-harness && pip install -e . --no-deps` |
| Skill file missing frontmatter | Wrong redirect / truncated write | Re-run `phone-harness skill > …/SKILL.md` |
| Agent never picks phone control | Skill not installed for that agent | Register under that agent’s skills path |
| Stale workflow after pull | Registered snapshot out of date | Re-run skill redirects |
| Skill prints, phone still fails | Permissions / session, not skill install | `phone-harness --doctor` and connection flow |
`phone-harness skill` does not validate mirroring or permissions; it only emits text.
## Related pages
Clone to ~/.phone-harness, pyobjc, editable install, PATH, and skill registration in the Fast Path.
phone-harness entry points: stdin exec, --doctor, skill, help, and usage errors.
agent-workspace/agent_helpers.py, PH_AGENT_WORKSPACE, and task-specific primitives like tap_icon.
First successful phone-harness script after install and skill registration.
Ordered permission and session checks that skill registration does not replace.
When the skill body says not to use the phone, and hard product limits.
---
## 13. CLI reference
> phone-harness entry points: stdin Python exec with helpers in scope, --doctor and doctor, skill, help flags, usage errors when args or TTY lack a script, and the ./phone-harness checkout launcher.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/13-cli-reference.md
- Generated: 2026-08-10T20:18:38.379Z
### Source Files
- `src/phone_harness/run.py`
- `pyproject.toml`
- `phone-harness`
- `src/phone_harness/admin.py`
- `src/phone_harness/helpers.py`
- `SKILL.md`
---
title: "CLI reference"
description: "phone-harness entry points: stdin Python exec with helpers in scope, --doctor and doctor, skill, help flags, usage errors when args or TTY lack a script, and the ./phone-harness checkout launcher."
---
`phone-harness` is a single console entry point (`phone_harness.run:main` in `pyproject.toml`) that either runs a named subcommand or `exec`s a Python script from stdin with the public `helpers` namespace already in scope. There is no daemon, REPL, or file-path argument mode: every control session is one process, one stdin body (or one doctor/skill invocation), then exit.
## Entry points
| Surface | How it is invoked | Implementation |
| --- | --- | --- |
| Installed command | `phone-harness …` on `PATH` | `[project.scripts]` → `phone_harness.run:main` |
| Checkout launcher | `./phone-harness …` from the repo root | Shell wrapper: `PYTHONPATH=$DIR/src exec python3 -m phone_harness.run "$@"` |
| Module form | `python3 -m phone_harness.run …` with `src` on `PYTHONPATH` | Same `main()` as the console script |
Both the installed command and the checkout launcher share argument parsing and exit behavior. The launcher exists so a working tree can be used without `pip install`.
## Command dispatch
`main()` reads `sys.argv[1:]` and branches in fixed order:
1. `-h` / `--help` → print usage text, return (exit `0`)
2. `--doctor` / `doctor` → `admin.run_doctor()`, then `sys.exit(code)`
3. `skill` → print repo-root `SKILL.md` to stdout, return (exit `0`)
4. Otherwise → stdin script mode (see below)
```text
argv[0]
├─ -h | --help → print USAGE, exit 0
├─ --doctor|doctor → run_doctor(), exit 0|1
├─ skill → print SKILL.md, exit 0
└─ (no args) → read stdin Python and exec
if TTY / extra args / empty → print USAGE, exit 1
```
There are no long-option parsers, subcommand groups, or positional script paths. Unknown first tokens fall through to the stdin gate and become usage errors.
## Usage text
Printed by help and by usage failures:
```text
Usage:
phone-harness <<'PY'
print(screen_info())
PY
Commands:
phone-harness --doctor diagnose permissions, app, and session state
phone-harness skill print the phone-harness skill text
```
## Stdin Python exec
Default mode when the process is not a TTY and stdin has non-empty code.
### Preconditions
| Condition | Result |
| --- | --- |
| Any argv after the program name (except the known commands above) | Usage error |
| `sys.stdin.isatty()` is true (interactive terminal, no pipe/heredoc) | Usage error |
| Stdin is empty or only whitespace | Usage error |
| Non-empty stdin and no extra args | Script runs |
Typical invocation:
```bash
phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```
Pipes work the same way:
```bash
echo 'print(screen_info())' | phone-harness
```
### Namespace
On success, `run.py`:
1. Imports `phone_harness.helpers` (which may load agent helpers at import time)
2. Builds `g = {k: v for k, v in vars(helpers).items() if not k.startswith("_")}`
3. Sets `g["__name__"] = "__main__"`
4. Calls `exec(code, g)`
So scripts see public helpers as bare names (`screen_info`, `ocr`, `tap`, …), not as `phone_harness.helpers.ocr`. Names starting with `_` are omitted. Modules and constants that do not start with `_` (for example `mirror`, `tap`, `REPO_ROOT`, `AGENT_WORKSPACE`) are also present because they live on the helpers module.
Agent-editable symbols from `agent-workspace/agent_helpers.py` (or `PH_AGENT_WORKSPACE/agent_helpers.py`) are merged into the helpers module at import via `_load_agent_helpers()`, so they appear in the same stdin namespace without an extra import.
Raw Quartz remains available inside the script via a normal import:
```python
import Quartz # escape hatch when helpers are not enough
```
### Runtime characteristics
- **One shot.** No REPL loop; the process ends when `exec` returns or raises.
- **No built-in top-level exception wrapper.** Uncaught exceptions propagate to the Python process exit (typically non-zero).
- **Backend selection** for taps/capture happens when helpers is imported (`PHONE_HARNESS_BACKGROUND`, SkyLight fallback). It is not a CLI flag.
- **No session daemon.** Window bounds and captures are re-queried inside helpers; each CLI invocation is self-contained.
### Minimal script examples
```bash
# Connection gate before work
phone-harness <<'PY'
print(connection_state())
PY
```
```bash
# Capture ground truth
phone-harness <<'PY'
info = screen_info()
print(info)
print(ocr()[:5])
PY
```
```bash
# Gesture after ensure_mirroring
phone-harness <<'PY'
ensure_mirroring()
home()
wait_stable()
print([o["text"] for o in ocr()][:20])
PY
```
## Help flags
| Flag | Behavior | Exit |
| --- | --- | --- |
| `-h` | Print `USAGE` to stdout | `0` |
| `--help` | Same | `0` |
Only the first argument is checked. Combined forms such as `phone-harness --help --doctor` still match help and do not run doctor.
## Doctor
| Form | Equivalent |
| --- | --- |
| `phone-harness --doctor` | Canonical |
| `phone-harness doctor` | Same path |
Both call `admin.run_doctor()` and exit with its return code.
Doctor prints a headed ladder and per-check `[PASS]` / `[FAIL]` lines. Summary:
- Success: `all clear` → exit `0`
- Failure: `fix the FAILs above, then re-run` → exit `1`
Check order (early hard-stop only on missing pyobjc):
| Order | Check | Fatal to overall `ok` |
| --- | --- | --- |
| 1 | pyobjc frameworks (Quartz, Vision, AppKit) | Yes — returns `1` immediately if import fails |
| 2 | Accessibility (`AXIsProcessTrusted`) | Yes |
| 3 | Screen Recording (`CGPreflightScreenCaptureAccess`) | Yes |
| 4 | iPhone Mirroring app installed at known path | Yes |
| 5 | App running | Reported; not fatal by itself |
| 6 | Mirroring window found | Reported |
| 7 | Window capture size `> 20_000` bytes | Yes when a window exists |
| 8 | Vision OCR on that capture | Reported when capture is large enough |
Doctor always ends with a note that a fresh machine may still prompt for extra permissions on first real action even when the ladder passes.
Full ladder semantics and recovery live on the doctor diagnostics page; the CLI only needs the flags, exit codes, and that output is human-oriented text on stdout (not JSON).
## Skill
```bash
phone-harness skill
```
- Resolves the package file to the repository root (`run.py` → `src/phone_harness` → `src` → repo)
- Prints the full contents of `SKILL.md` to stdout (UTF-8, no trailing rewrite)
- Exit `0`
Used to install or re-sync agent skill bodies:
```bash
mkdir -p ~/.claude/skills/phone-harness
phone-harness skill > ~/.claude/skills/phone-harness/SKILL.md
mkdir -p "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness"
phone-harness skill > "${CODEX_HOME:-$HOME/.codex}/skills/phone-harness/SKILL.md"
```
The skill body describes when to use the harness, stdin usage, consent, connection rules, and helper patterns. Re-run the redirect after `git pull` so installed copies stay current.
## Usage errors
`sys.exit(USAGE)` is used when stdin mode cannot start. That prints the usage string and exits with code `1`.
| Trigger | Example |
| --- | --- |
| Extra / unknown argv | `phone-harness foo` |
| Interactive TTY, no script | typing `phone-harness` at a shell prompt |
| Empty stdin | `phone-harness
Clone, pyobjc, editable install, PATH command, and first `--doctor` verification.
First stdin script: connection state, `screen_info`, and success signal.
`phone-harness skill` install paths for Claude Code and Codex, and re-sync after pulls.
Public names available in the stdin `exec` namespace and their return shapes.
Ordered ladder, per-check meaning, and exit behavior beyond the CLI flag.
`PHONE_HARNESS_BACKGROUND`, `PH_AGENT_WORKSPACE`, and backend fallback.
---
## 14. Helpers API
> Pre-imported public helpers: session, capture, ocr, gestures, scroll family, navigation, timing, and re-exported backend primitives with signatures, defaults, return shapes, and raised errors.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/14-helpers-api.md
- Generated: 2026-08-10T20:17:49.928Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `src/phone_harness/ocr.py`
- `agent-workspace/agent_helpers.py`
- `src/phone_harness/run.py`
---
title: "Helpers API"
description: "Pre-imported public helpers: session, capture, ocr, gestures, scroll family, navigation, timing, and re-exported backend primitives with signatures, defaults, return shapes, and raised errors."
---
`phone-harness` injects every non-underscore name from `src/phone_harness/helpers.py` into stdin scripts, plus public names loaded from `agent-workspace/agent_helpers.py`. Backend primitives (`tap`, `long_press`, `drag`, `press`, `type_text`, `activate`, `find_window`) are re-exported from the selected input backend; higher-level helpers compose capture + OCR + those primitives.
## How the namespace is built
```text
phone-harness (stdin Python)
│
▼
run.main() → import helpers
│
├─ select backend (background default, else mirror)
├─ re-export: tap, long_press, drag, press, type_text, activate, find_window
├─ define session / OCR / gesture / scroll / nav / timing helpers
└─ load PH_AGENT_WORKSPACE/agent_helpers.py → globals (e.g. tap_icon)
│
▼
exec(script, {k: v for k,v in vars(helpers) if not k.startswith("_")})
```
| Source | What enters the script namespace |
|--------|----------------------------------|
| `helpers.py` | All public functions and re-exports listed below |
| Active backend (`background` or `mirror`) | Bound through the re-exports; not imported by name in scripts |
| `agent_helpers.py` | Public defs only (names not starting with `_`) |
| Private helpers (`_win`, `_content_texts`, …) | **Not** injected |
Raw Quartz remains available: `import Quartz` inside a script for anything the helpers do not cover.
### Backend selection
| Condition | Module used |
|-----------|-------------|
| `PHONE_HARNESS_BACKGROUND` unset or truthy (default `"1"`) | Prefer `background` (SkyLight, no focus steal) |
| Env is `"0"`, `"false"`, or `"no"` (case-insensitive) | Force `mirror` (CGEvent HID + activate) |
| Background import fails | Fall back to `mirror`; selection flag cleared |
Paths: `src/phone_harness/helpers.py`, `background.py`, `mirror.py`.
## Shared data shapes
### Window bounds
Returned by `find_window()`, `ensure_mirroring()`, and as the second value of backend `capture()`:
```python
{"x": float, "y": float, "w": float, "h": float, "id": int}
```
Coordinates are **global screen points** (top-left origin). OCR centers and `tap(x, y)` use the same space.
### OCR box
```python
{
"text": str,
"confidence": float, # rounded to 3 decimals
"x": float, # box center, screen points
"y": float,
"w": float, # size in screen points
"h": float,
}
```
Vision uses a bottom-left normalized box; `ocr.recognize` flips Y and scales image pixels → window points so `(x, y)` is tap-ready.
### Temp captures
Default PNGs land under `{tempdir}/phone-harness/` (`window.png` for mirror, `background.png` for background).
---
## Session and connection
### `connection_state()`
**Returns:** `"ready"` | `"blocked"` | `"no-window"` | `"not-running"`
| Value | Condition |
|-------|-----------|
| `not-running` | `running_app()` is `None` |
| `no-window` | App running, `find_window()` is `None` |
| `blocked` | Window capturable and OCR text matches a blocked interstitial marker |
| `ready` | Window capturable and not blocked |
Blocked markers (case-insensitive substring over joined OCR text):
- `iphone in use`
- `lock your iphone`
- `mirroring ended`
- `to connect`
Performs a capture + OCR when a window exists. Does not launch the app or tap Connect.
### `ensure_mirroring()`
**Returns:** window bounds dict when state is `ready` (after `activate()`).
**Raises:** `RuntimeError` with a user-facing reconnect message:
| State | Error intent |
|-------|----------------|
| `not-running` | Open iPhone Mirroring and connect the phone |
| `no-window` | Connect the phone in the already-open app |
| `blocked` | User must connect / lock iPhone if "iPhone in Use"; agent must not tap Connect |
Never launches the app, taps Connect/Continue, or polls to reconnect.
### `screen_info()`
**Returns:**
```python
{
"window": {"x", "y", "w", "h", "id"},
"frontmost": bool,
"img_px": [width_px, height_px],
}
```
Captures once, then reads PNG pixel size via Vision-friendly `image_size`.
---
## Capture
### `screenshot(path=None)`
| Param | Default | Notes |
|-------|---------|-------|
| `path` | Backend default under temp `phone-harness/` | Optional PNG path |
**Returns:** `str` path to the PNG (window bounds discarded).
**Raises:** backend `RuntimeError` after retries if capture fails (`window capture failed…` or `background capture failed…`).
Backend `capture(path=None, retries=2)` returns `(path, window)` but is **not** re-exported as a top-level helper name; use `screenshot` or `screen_info` / `ocr`.
---
## Reading the screen (OCR)
### `ocr(min_confidence=0.3)`
Captures, runs Vision Accurate recognition, filters by confidence.
**Returns:** `list[OCR box]` sorted as Vision returns them.
| Param | Default | Notes |
|-------|---------|-------|
| `min_confidence` | `0.3` | Boxes with lower confidence dropped |
**Raises:** `RuntimeError` from capture or `Vision OCR failed: …` / `cannot read image …`.
### `find_text(query, exact=False)`
| Param | Default | Notes |
|-------|---------|-------|
| `query` | required | Case-insensitive |
| `exact` | `False` | `True` → full string equality; else substring |
**Returns:** matching OCR boxes (via a fresh `ocr()` call with default confidence).
### `tap_text(query, index=0, exact=False)`
Finds matches, taps `hits[index]` center with `tap(x, y)`.
**Returns:** the tapped OCR box dict.
**Raises:** `RuntimeError` if no match — message includes up to 30 currently visible text strings for recovery.
---
## Re-exported input primitives
Bound to the active backend at import time. Coordinates are global screen points.
### `tap(x, y)`
Single left-click style touch at `(x, y)`.
| Backend | Behavior |
|---------|----------|
| `background` | SkyLight event record down/up; no focus change |
| `mirror` | Activates app, HID mouse move + down/up |
### `long_press(x, y, duration=0.8)`
Hold down for `duration` seconds, then up.
### `drag(x1, y1, x2, y2, duration=0.35, steps=14)`
Touch-drag (iOS swipe). Intermediate dragged events along the segment.
### `press(combo)`
Keyboard combo string, lowercased and split on `+`.
| Example | Meaning |
|---------|---------|
| `press("return")` | Return / Enter |
| `press("cmd+1")` | Home (also used by `home()`) |
| `press("cmd+2")` | App Switcher |
| `press("cmd+3")` | Spotlight |
**Raises:** `ValueError` if the key part is unknown.
Known keys include: `return`/`enter`, `tab`, `space`, `delete`/`backspace`, `escape`/`esc`, arrows, digits `0`–`9`, letters `a`–`z`.
Modifiers: `cmd`, `shift`, `alt`/`option`, `ctrl`.
### `type_text(text, delay=0.03)`
Types via **US keycodes** (iPhone Mirroring ignores Unicode keyboard payloads). `\n` presses return between lines.
**Raises:** `ValueError` for characters with no keycode mapping (emoji, many non-US symbols).
### `activate()`
| Backend | Behavior |
|---------|----------|
| `mirror` | Bring iPhone Mirroring frontmost; **raises** if app not running; does not launch |
| `background` | No-op (returns `None`) |
### `find_window()`
**Returns:** window dict or `None`. On-screen layer-0 window owned by `"iPhone Mirroring"` with width ≥ 100.
---
## Gestures relative to the phone window
These call `ensure_window()` (or the backend equivalent) so a missing window raises before input.
### `swipe(direction, distance=0.4)`
| Param | Default | Notes |
|-------|---------|-------|
| `direction` | required | `'up'` \| `'down'` \| `'left'` \| `'right'` |
| `distance` | `0.4` | Fraction of window width/height |
Finger motion convention: `swipe("up")` moves the finger up (content moves up / scrolls down). Implemented as a **fast short drag** (`duration=0.12`, `steps=6`) for momentum flicks (Home Screen pages, carousels).
**Raises:** `ValueError` for unknown direction.
### `scroll(amount=300)`
Scroll-gesture at window center via `scroll_wheel`. Positive `amount` scrolls content **down** (trackpad two-finger-up semantics: helper passes `-amount` to the backend).
Prefer `scroll_screen` / `scroll_collect` for lists; use `swipe` when momentum pages matter.
---
## Scroll family (lists)
End-of-list is decided by **screen movement** (OCR text-set Jaccard overlap after a settle window), never solely by whether a parser found new items.
Content OCR for movement uses the middle band of the window (status bar and home/nav strip excluded): roughly top 6% and bottom 8% cropped out, min confidence `0.4`.
### `scroll_screen(direction="up", amount=0.6, settle=2.5, moved_thresh=0.6)`
One scroll step, then wait until content text stabilizes or `settle` seconds elapse.
| Param | Default | Notes |
|-------|---------|-------|
| `direction` | `"up"` | `'up'` reveals content below; `'down'` reveals above |
| `amount` | `0.6` | Fraction of window height for the gesture magnitude |
| `settle` | `2.5` | Seconds to allow lazy-load before judging stillness |
| `moved_thresh` | `0.6` | `moved` is `False` when overlap ≥ threshold |
**Returns:**
```python
{
"moved": bool,
"overlap": float, # Jaccard of before/after text sets, 3 decimals
"before": frozenset[str],
"after": frozenset[str],
"boxes": list[OCR box], # settled content-area OCR
}
```
**Raises:** `ValueError` if direction is not `'up'` or `'down'`.
Implementation notes:
- Mirror backend: true scroll-wheel events at window center.
- Background backend: wheel events do not reach an unfocused app; `scroll_wheel` is implemented as a **fast vertical flick** with the same sign semantics.
### `scroll_until(done, direction="up", amount=0.6, max_scrolls=60, settle=2.5)`
| Param | Default | Notes |
|-------|---------|-------|
| `done` | required | Callable `done(boxes) -> truthy to stop` |
| `max_scrolls` | `60` | Cap on scroll steps after the initial check |
Checks current content OCR first. On non-movement, allows one retry (`stale >= 2`) with a short sleep and `activate()` before treating the end as reached.
**Returns:** truthy value from `done`, or `None` if the list stops moving or `max_scrolls` is exhausted without a hit.
### `scroll_collect(extract=None, key=None, direction="up", amount=0.6, max_scrolls=400, end_after=3, settle=2.5, on_progress=None)`
Scroll while extracting and de-duplicating items.
| Param | Default | Notes |
|-------|---------|-------|
| `extract` | strip each content text line | `extract(boxes) -> list[items]` |
| `key` | identity | `key(item) -> hashable` for de-dup |
| `amount` | `0.6` | Keep `< 1.0` so screens overlap |
| `max_scrolls` | `400` | Hard stop |
| `end_after` | `3` | Consecutive non-moving scrolls ⇒ end |
| `on_progress` | `None` | Optional `on_progress(i, total_items, new_count, moved, overlap)` |
**Returns:**
```python
{
"items": list, # ordered unique items
"stop": "reached-end" | "max-scrolls",
"scrolls": int,
}
```
---
## Navigation
### `home()`
`press("cmd+1")`, then sleep `0.8s`.
### `app_switcher()`
`press("cmd+2")`, then sleep `0.8s`.
### `open_app(name)`
1. `press("cmd+3")` (Spotlight)
2. sleep `0.9s`
3. `type_text(name)`
4. sleep `1.2s` (results populate)
5. `press("return")`
6. `wait_stable()`
---
## Timing
### `wait(seconds=1.0)`
`time.sleep(seconds)`.
### `wait_stable(timeout=6.0, interval=0.5, settle=2)`
Poll captures; MD5 digest of PNG bytes must match for `settle` consecutive samples.
| Param | Default | Notes |
|-------|---------|-------|
| `timeout` | `6.0` | Max wait seconds |
| `interval` | `0.5` | Sleep between captures |
| `settle` | `2` | Identical digests needed (implementation uses `same >= settle - 1`) |
**Returns:** `True` if stable within timeout, else `False`.
---
## Agent-loaded helpers
Loaded at the end of `helpers` import from:
```text
$PH_AGENT_WORKSPACE/agent_helpers.py
# default: /agent-workspace/agent_helpers.py
```
### `tap_icon(label, index=0)` (shipped agent helper)
Home Screen app launch by label. Taps **~35 points above** the OCR label center (label text alone is not tappable on the mirrored Home Screen).
| Param | Default |
|-------|---------|
| `label` | required (passed to `find_text`) |
| `index` | `0` |
**Returns:** the matched OCR box (label location, not the adjusted tap point).
**Raises:** `RuntimeError` if no label matches.
In-app labeled controls should use `tap_text`; Home Screen icons should use `tap_icon`.
---
## Module constants (also injected)
| Name | Meaning |
|------|---------|
| `CORE_DIR` | `src/phone_harness` path |
| `REPO_ROOT` | Repository root (`CORE_DIR.parent.parent`) |
| `AGENT_WORKSPACE` | Resolved agent workspace path |
| `mirror` | The selected backend module object |
---
## Errors reference
| Helper / primitive | Exception | When |
|--------------------|-----------|------|
| `ensure_mirroring` | `RuntimeError` | Not ready (not-running / no-window / blocked) |
| `screenshot` / OCR path | `RuntimeError` | Capture failed after retries |
| `ocr` / Vision path | `RuntimeError` | Image unreadable or Vision request failed |
| `tap_text` | `RuntimeError` | No matching text (includes visible sample) |
| `tap_icon` | `RuntimeError` | No Home Screen label match |
| `swipe` / `scroll_screen` | `ValueError` | Bad `direction` |
| `press` | `ValueError` | Unknown key name |
| `type_text` | `ValueError` | Character has no US keycode |
| `activate` (mirror) | `RuntimeError` | App not running |
| Backend `ensure_window` | `RuntimeError` | App not running or no phone window |
Silent input failure is usually permissions/focus (Accessibility, Screen Recording), not an exception — see doctor and troubleshooting docs.
---
## Minimal script surface
```python
print(connection_state())
print(screen_info())
ensure_mirroring()
boxes = ocr()
tap_text("Settings")
wait_stable()
home()
tap_icon("Weather")
open_app("Messages")
# focus a field first, then:
# type_text("hello")
result = scroll_collect(max_scrolls=20)
print(result["stop"], len(result["items"]))
```
Run as:
```bash
phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```
---
## Related pages
How stdin scripts get the helpers namespace and when usage errors exit.
ready / blocked / no-window / not-running and physical reconnect rules.
Screen-point space, window vs pixels, and why coordinates must not be cached.
Background SkyLight path vs classic mirror CGEvent path.
Movement detection, settle windows, and stop reasons in depth.
home, Spotlight open_app, type_text keycode limits, tap_icon vs tap_text.
Editing agent_helpers.py and PH_AGENT_WORKSPACE override.
PHONE_HARNESS_BACKGROUND, PH_AGENT_WORKSPACE, and capture temp dir.
---
## 15. Environment variables
> PHONE_HARNESS_BACKGROUND default and falsey values, PH_AGENT_WORKSPACE path override, temp capture directory under phone-harness, and how backend selection falls back when SkyLight load fails.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/15-environment-variables.md
- Generated: 2026-08-10T20:19:17.499Z
### Source Files
- `src/phone_harness/helpers.py`
- `src/phone_harness/background.py`
- `src/phone_harness/mirror.py`
- `agent-workspace/agent_helpers.py`
- `src/phone_harness/run.py`
---
title: "Environment variables"
description: "PHONE_HARNESS_BACKGROUND default and falsey values, PH_AGENT_WORKSPACE path override, temp capture directory under phone-harness, and how backend selection falls back when SkyLight load fails."
---
phone-harness reads two environment variables at `helpers` import time: `PHONE_HARNESS_BACKGROUND` selects the input/capture backend (background SkyLight path vs classic mirror), and `PH_AGENT_WORKSPACE` points at the directory whose `agent_helpers.py` is auto-merged into the script namespace. Capture defaults land under a fixed `{tempdir}/phone-harness` directory (not env-configurable). Backend choice is fixed for the process once `phone_harness.helpers` loads.
## Variable reference
| Variable | Default | Read when | Effect |
|---|---|---|---|
| `PHONE_HARNESS_BACKGROUND` | `"1"` | Import of `phone_harness.helpers` | Prefer the background backend (`background.py`) when truthy; force classic mirror (`mirror.py`) when falsey |
| `PH_AGENT_WORKSPACE` | `/agent-workspace` | Same import | Directory containing `agent_helpers.py` for agent-editable helpers |
No other environment variables are consulted by the package for backend selection, workspace loading, or default capture paths. `phone-harness --doctor` imports `mirror` directly and does not honor `PHONE_HARNESS_BACKGROUND`.
## `PHONE_HARNESS_BACKGROUND`
### Selection logic
In `src/phone_harness/helpers.py`:
```python
_BACKGROUND = os.environ.get("PHONE_HARNESS_BACKGROUND", "1").lower() not in (
"0", "false", "no")
if _BACKGROUND:
try:
mirror = importlib.import_module(".background", __package__)
except Exception:
mirror = importlib.import_module(".mirror", __package__)
_BACKGROUND = False
else:
mirror = importlib.import_module(".mirror", __package__)
```
After selection, helpers bind transport primitives from the chosen module:
- `tap`, `long_press`, `drag`, `press`, `type_text`, `activate`, `find_window`
- Higher-level helpers call that module’s `capture`, `ensure_window`, `scroll_wheel`, and related APIs via the local `mirror` alias
### Defaults and falsey values
| Value (after `.lower()`) | Result |
|---|---|
| unset | Background on (default `"1"`) |
| `"1"`, `"true"`, `"yes"`, or any string **not** in the falsey set | Background attempted |
| `"0"`, `"false"`, `"no"` | Classic mirror forced |
| empty string `""` | Treated as truthy (not in the falsey set) |
Matching is case-insensitive only via `.lower()`; there is no stripping of whitespace.
### What each backend does
| Backend | Module | Focus behavior | Capture default path | Input path |
|---|---|---|---|---|
| Background (default) | `background.py` | Does not steal focus (`activate` is a no-op) | `{tempdir}/phone-harness/background.png` | SkyLight `SLPSPostEventRecordTo` mouse records; keyboard via make-key + `CGEventPostToPid` |
| Classic mirror | `mirror.py` | Activates iPhone Mirroring for capture/input | `{tempdir}/phone-harness/window.png` | `screencapture` + HID `CGEvent` posting; window must be frontmost or events are swallowed |
Both backends share the same global screen-point coordinate convention, so helpers such as `tap_text`, `swipe`, and `scroll_collect` keep the same API regardless of backend.
### Automatic fallback when SkyLight load fails
If background is requested and **any** exception is raised while importing `background` (including loading private SkyLight symbols at module top level), helpers:
1. Import `mirror` instead
2. Set `_BACKGROUND = False`
3. Continue without raising to the caller
There is no log line, stderr message, or public flag exposing that fallback occurred. SkyLight symbols are “not guaranteed across macOS builds”; the silent fall-through is intentional so the harness stays usable.
```text
PHONE_HARNESS_BACKGROUND truthy?
│
├─ no ──► import mirror.py (classic)
│
└─ yes ─► try import background.py
│
├─ success ──► use background (no focus steal)
│
└─ Exception ─► import mirror.py, _BACKGROUND=False
```
### Process lifetime
Selection runs once when `helpers` is first imported (for example when `phone-harness` runs `from . import helpers` in `run.py`). Changing the variable in the shell after a long-lived process has already imported helpers does not re-select the backend. Start a new `phone-harness` invocation to apply a new value.
### Common shell patterns
```bash
# Default: background backend (omit the variable)
phone-harness <<'PY'
print(screen_info())
PY
# Force classic mirror (focus-stealing HID path)
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
print(screen_info())
PY
# Explicit background request (same as default when load succeeds)
PHONE_HARNESS_BACKGROUND=1 phone-harness <<'PY'
print(screen_info())
PY
```
`phone-harness --doctor` always uses `from . import mirror` (classic path for its capture probe). A doctor PASS does not prove the background backend loaded successfully for script runs.
## `PH_AGENT_WORKSPACE`
### Resolution
```python
CORE_DIR = Path(__file__).resolve().parent # .../src/phone_harness
REPO_ROOT = CORE_DIR.parent.parent # checkout root
AGENT_WORKSPACE = Path(
os.environ.get("PH_AGENT_WORKSPACE", REPO_ROOT / "agent-workspace"))
```
Default layout (relative to the installed or checked-out repo root):
```text
/
agent-workspace/
agent_helpers.py
src/phone_harness/
helpers.py
```
### Load behavior
At the end of `helpers` import, `_load_agent_helpers()`:
1. Looks for `{AGENT_WORKSPACE}/agent_helpers.py`
2. Returns immediately if the file is missing (no error)
3. Otherwise loads it via `importlib.util.spec_from_file_location`
4. Copies every name that does **not** start with `_` into `helpers` globals
The CLI (`run.py`) builds the script namespace from public names on `helpers`, so agent helpers are available in stdin scripts next to core APIs:
```bash
PH_AGENT_WORKSPACE=/path/to/my-workspace phone-harness <<'PY'
# tap_icon comes from agent_helpers.py when defined there
tap_icon("Weather")
PY
```
Stock `agent-workspace/agent_helpers.py` defines `tap_icon` (Home Screen icon: tap ~35 points above the OCR label). Edit that file (or an alternate workspace) for task-specific primitives.
Override must be a directory path. The loader always appends `agent_helpers.py`; do not point `PH_AGENT_WORKSPACE` at the file itself.
## Temp capture directory
There is **no** environment variable for capture storage. Both backends hardcode:
```python
TMP = Path(tempfile.gettempdir()) / "phone-harness"
TMP.mkdir(exist_ok=True)
```
| Item | Value |
|---|---|
| Root | `{tempfile.gettempdir()}/phone-harness` |
| Classic default file | `window.png` |
| Background default file | `background.png` |
| Directory creation | `mkdir(exist_ok=True)` on module import |
On macOS, `tempfile.gettempdir()` is typically under `/var/folders/.../T` (session-specific), so the full path is machine- and user-session-specific rather than a fixed `/tmp/phone-harness`.
### Overriding the file path (not the directory)
`capture(path=None, ...)` and helper `screenshot(path=None)` accept an explicit path. Only the default path uses `TMP`; callers can write elsewhere:
```python
screenshot("/tmp/my-phone.png")
```
Doctor capture uses a separate `tempfile.NamedTemporaryFile(suffix=".png")` and deletes it after the check; it does not rely on the `phone-harness` temp subdirectory.
## Operational notes
| Concern | Behavior |
|---|---|
| Backend switch mid-session | Not supported; restart the process |
| Background import failure | Silent fall-back to classic mirror |
| Missing `agent_helpers.py` | Silent skip; core helpers still work |
| Invalid agent helper syntax | Import of `helpers` fails (load is not wrapped in try/except) |
| Doctor vs scripts | Doctor always probes via classic `mirror`; scripts use selected backend |
| Permissions | Accessibility + Screen Recording still required for both backends |
## Failure modes
| Symptom | Likely cause | Check |
|---|---|---|
| Automation steals focus despite wanting background | `PHONE_HARNESS_BACKGROUND` is `0`/`false`/`no`, or SkyLight import fell back | Unset or set to `1`; inspect whether private framework load fails on this macOS build |
| Custom helpers not visible in scripts | Wrong `PH_AGENT_WORKSPACE`, missing file, or names start with `_` | Confirm `{PH_AGENT_WORKSPACE}/agent_helpers.py` exists and public names export |
| Capture PNGs hard to find | Looking under `/tmp` only | Resolve `tempfile.gettempdir()` and open `phone-harness/` under it |
| Taps silent on classic path | Window not frontmost or Accessibility denied | Grant Accessibility; classic backend activates; see troubleshooting |
## Related pages
Background SkyLight path versus classic CGEvent mirror path, focus behavior, and when to force either mode.
Editing `agent_helpers.py`, `PH_AGENT_WORKSPACE`, auto-load into the script namespace, and `tap_icon`.
Public helpers pre-imported into `phone-harness` scripts, including capture and gesture signatures.
Stdin Python exec, `--doctor`, `skill`, and how the script namespace is built from `helpers`.
Ordered permission and session checks (classic mirror probe path).
Blank capture, silent taps, blocked interstitials, and focus-related failures.
---
## 16. Doctor diagnostics
> Ordered --doctor ladder: pyobjc, Accessibility, Screen Recording, app installed, running, window, capture size, Vision OCR; exit codes; and the note that fresh machines may need extra prompts.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/16-doctor-diagnostics.md
- Generated: 2026-08-10T20:26:54.457Z
### Source Files
- `src/phone_harness/admin.py`
- `src/phone_harness/run.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/ocr.py`
- `install.md`
---
title: "Doctor diagnostics"
description: "Ordered --doctor ladder: pyobjc, Accessibility, Screen Recording, app installed, running, window, capture size, Vision OCR; exit codes; and the note that fresh machines may need extra prompts."
---
`phone-harness --doctor` (also accepted as `phone-harness doctor`) runs `run_doctor()` in `src/phone_harness/admin.py`: an ordered ladder of environment checks that prints one `[PASS]`/`[FAIL]` line per step and exits `0` only when every fatal check passes. The ladder is dependency-ordered — pyobjc frameworks, then macOS permissions, then iPhone Mirroring app state, then a live window capture and a Vision OCR probe — so the correct repair strategy is always to fix the first `FAIL` and re-run.
## Running the doctor
```bash
phone-harness --doctor
```
Each check prints a mark, a label, and — on failure — a repair hint:
```text
phone-harness doctor
[PASS] pyobjc frameworks (Quartz, Vision, AppKit)
[PASS] Accessibility permission (taps & keystrokes)
[FAIL] Screen Recording permission (seeing the phone) — System Settings > Privacy & Security > Screen Recording: enable your terminal
...
fix the FAILs above, then re-run
```
A fully passing run ends with `all clear` instead. Both outcomes append a standing note that these are the *currently known* required permissions and that a fresh machine may prompt for more (see below).
## The check ladder
Checks run in this fixed order. Later checks depend on earlier ones — a missing window skips the capture and OCR probes entirely.
| # | Check | Implementation | Affects exit code |
|---|-------|----------------|-------------------|
| 1 | pyobjc frameworks (Quartz, Vision, AppKit) | `import Quartz, Vision, AppKit` | Yes — aborts immediately with exit `1` |
| 2 | Accessibility permission | `AXIsProcessTrusted()` from ApplicationServices | Yes |
| 3 | Screen Recording permission | `Quartz.CGPreflightScreenCaptureAccess()` | Yes |
| 4 | iPhone Mirroring installed | `Path("/System/Applications/iPhone Mirroring.app").exists()` | Yes |
| 5 | iPhone Mirroring running | `mirror.running_app()` (lookup by bundle id `com.apple.ScreenContinuity`) | No — advisory |
| 6 | Mirroring window found | `mirror.find_window()` via `CGWindowListCopyWindowInfo` | No — advisory, but gates checks 7–8 |
| 7 | Window capture works | `screencapture -x -o -l ` to a temp PNG; passes when the file exceeds 20,000 bytes | Yes |
| 8 | Vision OCR works | `ocr.recognize()` on the capture; reports the recognized text-box count | Reported informationally when capture passes |
Two checks are deliberately advisory. "App running" is not fatal because iPhone Mirroring auto-launches on first use. "Window found" is not fatal either, but when it fails the hint tells you to open iPhone Mirroring once manually to pair the phone — pairing needs the physical device and cannot be done by the harness (see [Connection and session states](/connection-and-session)).
### Failure hints per step
Hint: `pip install pyobjc-framework-Quartz pyobjc-framework-Vision`, plus the original `ImportError`. This is the only check that short-circuits: without pyobjc none of the later checks can run, so the doctor returns exit code `1` immediately.
Hint: System Settings > Privacy & Security > Accessibility — enable your terminal. Takes effect immediately once toggled.
Hint: System Settings > Privacy & Security > Screen Recording — enable your terminal. Unlike Accessibility, this grant only takes effect after the terminal app restarts.
Hint: requires macOS Sequoia+ with a paired iPhone. The check looks for the app bundle at `/System/Applications/iPhone Mirroring.app`.
Hint: open iPhone Mirroring once manually to pair the phone. The window lookup scans on-screen windows owned by `iPhone Mirroring` at layer 0 and ignores panels narrower than 100 points.
Hint: capture is blank — the Screen Recording permission needs a terminal restart to take effect. A blank capture produces a tiny PNG, which is why the pass threshold is a file size above 20,000 bytes.
## Capture and OCR probes
When a mirroring window exists, the doctor takes a real end-to-end sample rather than trusting the permission preflight alone:
1. It runs `screencapture -x -o -l ` into a temporary `.png` (deleted afterward).
2. It checks the file size. A grant that has not taken effect yet still "succeeds" but produces a near-empty black frame, so the size threshold (`> 20,000` bytes) is the actual pass condition.
3. Only if the capture passes does it call `ocr.recognize(path, win)` — Apple's Vision framework at `VNRequestTextRecognitionLevelAccurate` — and print the number of recognized text boxes, e.g. `[PASS] Vision OCR works (23 text boxes)`.
This mirrors the runtime read path exactly: the same `find_window()` bounds, the same `screencapture` invocation, and the same Vision pipeline used by [Coordinates, capture, and OCR](/coordinates-capture-ocr).
## Exit codes
All fatal checks passed. Printed footer: `all clear`. Advisory checks (app running, window found) may still show `FAIL` on a `0` exit — they do not fold into the result.
At least one fatal check failed (pyobjc, Accessibility, Screen Recording, app installed, or capture size). Printed footer: `fix the FAILs above, then re-run`. A pyobjc import failure returns `1` immediately without running the remaining checks.
## Fresh machines may need extra prompts
The doctor prints this caveat on every run, pass or fail: Accessibility and Screen Recording are the permissions *currently known* to be required — the harness was built on an already-permissive machine, so a clean Mac may raise additional macOS approval prompts the first time an action actually runs. If `--doctor` passes but taps, typing, or capture silently do nothing, watch for a permission prompt and check System Settings > Privacy & Security for a pane asking to approve your terminal. `install.md` notes that newly identified prerequisites will be added to `--doctor` as they are pinned down.
Open the two known permission panes directly:
```bash
open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"
```
## Related pages
Prerequisites, pyobjc install, PATH setup, and the first `phone-harness --doctor` verification run.
Recovery for blank captures after a Screen Recording grant, missing windows, and silent taps.
All `phone-harness` entry points, including `--doctor`, `doctor`, `skill`, and stdin script execution.
Why pairing and reconnecting the phone is a user action the doctor can only point at.
---
## 17. Troubleshooting
> Failure modes and recovery: blank capture after Screen Recording grant, window not found, silent taps without Accessibility or focus, blocked iPhone in Use, Home Screen label misses, and type_text field focus.
- Page Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/pages/17-troubleshooting.md
- Generated: 2026-08-10T20:18:05.047Z
### Source Files
- `install.md`
- `SKILL.md`
- `src/phone_harness/admin.py`
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `agent-workspace/agent_helpers.py`
---
title: "Troubleshooting"
description: "Failure modes and recovery: blank capture after Screen Recording grant, window not found, silent taps without Accessibility or focus, blocked iPhone in Use, Home Screen label misses, and type_text field focus."
---
phone-harness fails in a small set of known ways: permissions that look granted but are not live, a missing or blocked iPhone Mirroring session, input that posts without an effect, and label-vs-icon / field-focus mistakes on the mirrored UI. Start with `phone-harness --doctor` (exit `0` = ladder clear, `1` = at least one hard FAIL), then match the symptom below.
## First step: doctor ladder
```bash
phone-harness --doctor
```
Ordered checks (fix the **first** FAIL; later steps depend on earlier ones):
| Order | Check | Hard fail? |
|-------|--------|------------|
| 1 | pyobjc (`Quartz`, `Vision`, `AppKit`) | Yes — exits early |
| 2 | Accessibility (`AXIsProcessTrusted`) | Yes |
| 3 | Screen Recording (`CGPreflightScreenCaptureAccess`) | Yes |
| 4 | iPhone Mirroring installed at `/System/Applications/iPhone Mirroring.app` | Yes |
| 5 | App running | No — note only |
| 6 | Mirroring window found | No — note only |
| 7 | Window capture size `> 20_000` bytes | Yes when window exists |
| 8 | Vision OCR on that capture | Note (runs when capture passes) |
Open permission panes:
```bash
open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"
```
`--doctor` only asserts the permissions currently known to be required. On a fresh Mac, macOS may still prompt for extra approvals the first time an action runs. If doctor is green but taps, typing, or capture silently do nothing, watch for a System Settings prompt under Privacy & Security.
## Blank or black capture after Screen Recording grant
**Symptom:** Capture “works” but the PNG is empty/black; doctor reports capture FAIL with size not `> 20_000` bytes; `screen_info()` / `ocr()` show no useful content.
**Cause:** Screen Recording takes effect only **after the terminal app restarts**. Granting the toggle without relaunching leaves captures blank. Doctor’s capture hint is explicit: *“capture is blank — Screen Recording permission needs a terminal restart to take effect.”*
**Not the same as:** DRM-protected video inside an app (product limit: those frames render black even with permissions correct).
**Recovery:**
Enable your terminal (or IDE terminal host) under System Settings → Privacy & Security → Screen Recording.
Restart the process that runs `phone-harness`, not only a shell tab.
```bash
phone-harness --doctor
phone-harness <<'PY'
print(screen_info())
print(screenshot())
PY
```
**Verify:** Doctor capture line shows a large byte size; `screen_info()` returns real `window` bounds and non-trivial `img_px`.
## Window not found
**Symptom:** Doctor: `mirroring window found` FAIL; scripts raise from `ensure_window` / `ensure_mirroring` about no phone window or app not running.
| Signal | Typical meaning |
|--------|-----------------|
| `connection_state() == "not-running"` | iPhone Mirroring process not up |
| `connection_state() == "no-window"` | App open, no on-screen phone window |
| RuntimeError: *isn't running* / *no phone window* | Same, from `ensure_window` / `ensure_mirroring` |
**Causes:** Phone not paired, out of Continuity range, or only a connect interstitial is showing. Window discovery uses `CGWindowListCopyWindowInfo` for owner `"iPhone Mirroring"`, layer `0`, width ≥ 100 (toolbars ignored).
**Recovery:**
1. Open **iPhone Mirroring** manually once and complete pairing (needs the physical phone).
2. Keep the phone in range with Continuity requirements met.
3. Do **not** rely on the harness to open/connect the app for you — reconnect is a user physical action.
**Verify:**
```bash
phone-harness <<'PY'
print(connection_state()) # expect "ready" (or "blocked" if interstitial)
print(find_window()) # expect {x,y,w,h,id}
PY
```
## Silent taps (Accessibility missing or focus lost)
**Symptom:** No exception; UI does not change. `tap` / `tap_text` / `swipe` appear to run.
**Causes:**
| Cause | Behavior |
|-------|----------|
| Accessibility off for the terminal | HID events not accepted; Accessibility is immediate once enabled |
| Classic mirror backend + window not frontmost | `CGEventPost` to HID is swallowed when Mirroring is not frontmost |
| Focus stolen mid-task (another app click) | Same as unfocused input on the classic backend |
| AppleScript / AX `click at` | Always silent no-op: the window is a video stream with no accessibility tree |
**Backend notes:**
- Default: background backend (`PHONE_HARNESS_BACKGROUND` default truthy) delivers mouse via SkyLight event records without stealing focus; `activate()` is a no-op.
- Classic backend (`PHONE_HARNESS_BACKGROUND=0`/`false`/`no`, or automatic fallback if SkyLight load fails): every input path calls `activate()` / `_focus()` first; if focus is lost between gestures, re-activate or re-run the helper.
- If SkyLight private symbols fail to load, helpers fall back to `mirror` rather than leaving the harness unusable.
**Recovery:**
System Settings → Privacy & Security → Accessibility → enable the terminal. Re-check with `--doctor`.
Ensure Mirroring is frontmost before acting, or call `activate()` after focus steals.
```bash
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
activate()
tap_text("Notes") # or any visible control
PY
```
Only harness helpers / Quartz HID (or background event records) drive the stream.
**Verify:** After `tap_text` / `tap`, run `wait_stable()` then `ocr()` or `screenshot()` — capture is the only ground truth.
## Blocked session: “iPhone in Use” and connect interstitials
**Symptom:** `connection_state()` is `"blocked"`; `ensure_mirroring()` raises and tells the user to connect / lock the phone; UI shows connect, continue, or “iPhone in Use”.
**Detection:** OCR over the live capture; session is blocked if any of these substrings appear (case-insensitive):
- `iphone in use`
- `lock your iphone`
- `mirroring ended`
- `to connect`
**Cause:** Unlocking the physical phone pauses mirroring. Resuming is physical: open Mirroring, approve prompts, and **lock the iPhone** when the interstitial says “iPhone in Use”. Tapping Connect/Continue while the phone is unlocked does nothing useful.
**Hard rules (agents and scripts):**
- STOP and relay the error to the user.
- Never tap `Connect` / `Continue`.
- Never loop-poll waiting for reconnect.
- Retry only after the user confirms they locked/connected.
**Recovery (user only):**
1. Open iPhone Mirroring.
2. If “iPhone in Use”: **lock the iPhone** so the session can resume.
3. Confirm connection, then re-run the script.
```bash
phone-harness <<'PY'
print(connection_state())
# only proceed when "ready"
ensure_mirroring()
print(screen_info())
PY
```
**Verify:** `connection_state() == "ready"` and `ensure_mirroring()` returns window bounds without raising.
## Home Screen: `tap_text` misses the icon
**Symptom:** `tap_text("Weather")` finds the label and returns success-shaped data, but the app does not launch.
**Cause:** On the Home Screen, OCR hits the **label** under the icon. The tappable icon is about **35 points above** the label center. Verified pattern in `agent-workspace/agent_helpers.py`: label tap is a no-op; icon tap launches.
| Context | Use |
|---------|-----|
| Home Screen app icon | `tap_icon("Weather")` |
| In-app buttons / list rows | `tap_text("…")` is correct |
```bash
phone-harness <<'PY'
home()
wait_stable()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:15])
PY
```
`tap_icon` raises `RuntimeError` if no label matches; on hit it `tap(h["x"], h["y"] - 35)`.
**Verify:** After launch, OCR shows app chrome, not only Home Screen icons.
## `type_text` needs a focused iOS field
**Symptom:** Key events post; no characters appear; or `ValueError: cannot type '…' via keycodes`.
**Causes:**
1. No iOS text field / keyboard focused — must tap the field and wait for the keyboard before typing.
2. Typing is **US keycode only**. Mirroring forwards raw HID keycodes and ignores unicode payloads. Unsupported characters (emoji, non-keycoded glyphs) raise `ValueError`.
3. `\n` in the string becomes `press("return")` between lines.
**Recovery:**
```bash
phone-harness <<'PY'
tap_text("New Note") # or tap the field by coordinates
wait_stable()
type_text("hello from the harness")
press("return")
wait_stable()
print([o["text"] for o in ocr()][:20])
PY
```
**Verify:** OCR shows the typed string (or the field content changed). For Spotlight open: `open_app("Notes")` already types via the same keycode path after Cmd+3.
## Capture / OCR runtime errors
| Error pattern | Likely cause | Action |
|---------------|--------------|--------|
| `window capture failed after N tries: …` | `screencapture -l` failed (window not composited) and region fallback also failed | Ensure app running, window present, Screen Recording live; classic path activates then retries |
| `background capture failed: …` | `CGWindowListCreateImage` empty / PNG encode fail | Window gone or permission; check `find_window()` and doctor |
| `Vision OCR failed: …` | Vision request failed | Confirm capture is a valid PNG; re-run doctor OCR step |
| `cannot read image …` | Bad path / empty file | Fix capture first |
| `no visible text matches …; saw: […]` | Wrong screen or OCR miss | Read the exception’s visible list; `screenshot()` for unlabeled icons |
Classic `mirror.capture` tries window-id capture (`screencapture -x -o -l `), then activates and falls back to region `-R x,y,w,h`. Success requires return code 0, file exists, size `> 1000` bytes.
## Coordinates and other silent-ish failures
| Issue | Behavior | Fix |
|-------|----------|-----|
| Cached coordinates after window move | Taps miss | Re-query every action: `ocr()`, `find_window()`, `swipe` re-read bounds |
| DRM / protected video | Black frames in capture | Product limit; cannot OCR/tap inside black content |
| Multi-touch (pinch, two-finger) | Not supported | Use single-point gestures only |
| List barely moves with slow drag | iOS bounce / no advance | Prefer `scroll` / `scroll_screen` / `scroll_collect` (wheel or background flick), not slow `drag` |
## Quick decision table
| You observe | Check | Fix |
|-------------|-------|-----|
| Black tiny capture | Screen Recording + restart | Quit terminal; re-enable; `--doctor` |
| No window | Pairing / range / app open | User opens Mirroring and connects |
| Taps no-op | Accessibility / focus / backend | Doctor; enable AX; `activate` or background path |
| Connect / “iPhone in Use” UI | `connection_state() == "blocked"` | User locks phone / reconnects; no agent taps |
| Home app won’t open | Label vs icon | `tap_icon` not `tap_text` |
| Typing vanishes | No field / bad chars | Focus field + keyboard; US keycodes only |
| Doctor green, still silent | Extra macOS prompt | Approve new Privacy & Security entry |
## Related pages
Full ladder, exit codes, and fresh-machine permission notes.
`ready` / `blocked` / `no-window` / `not-running` and `ensure_mirroring` gates.
Background SkyLight vs classic HID focus, `PHONE_HARNESS_BACKGROUND`, fallback.
Prerequisites, editable install, and permission panes.
`type_text` keycode rules, `tap_icon` vs `tap_text`, Spotlight open.
DRM black frames, single session, no multi-touch.
---