Phone Harness Documentation

Pages

1. OverviewWhat phone-harness exposes, who it is for, runtime assumptions (macOS Sequoia+, iPhone Mirroring, real phone), and the first docs routes to follow.
2. InstallationPrerequisites, clone to ~/.phone-harness, pyobjc and editable pip install, PATH command, skill registration, and verification with phone-harness --doctor.
3. QuickstartFirst successful script: check connection state, print screen_info, run a short stdin Python block with pre-imported helpers, and read the success signal.
4. Connection and session statesconnection_state values ready, blocked, no-window, not-running; ensure_mirroring gates; blocked interstitial markers; and why reconnect is a user physical action.
5. Coordinates, capture, and OCRGlobal screen-point space, window bounds vs image pixels, capture return shape, Vision OCR boxes with confidence and tap-ready centers, and why coordinates must not be cached.
6. Input backendsDefault background backend (SkyLight event records, no focus steal) versus classic mirror backend (CGEvent HID + activate); selection via PHONE_HARNESS_BACKGROUND and automatic fallback.
7. Consent and limitsWhen not to use the phone, outward-facing and irreversible actions that require user consent, and hard product limits (one session, no multi-touch, DRM black frames, keycode typing).
8. See, act, and verifyOCR-first reading, tap and tap_text, wait_stable, screenshot for unlabeled icons, and the capture-as-ground-truth verification loop after every action.
9. Navigate apps and type texthome, app_switcher, open_app via Spotlight, press key combos, type_text US keycode constraints, and Home Screen tap_icon versus in-app tap_text.
10. Scroll and collect listsscroll_screen movement detection, scroll_until predicates, scroll_collect de-dup extraction, wheel versus flick behavior, settle windows, and stop reasons reached-end and max-scrolls.
11. Extend agent helpersEdit agent-workspace/agent_helpers.py for task-specific primitives, PH_AGENT_WORKSPACE override, auto-load into the script namespace, and the tap_icon Home Screen pattern.
12. Register as an agent skillphone-harness skill output, Claude Code and Codex skill install paths, trigger text, and re-sync after pulls so the agent auto-reaches for phone control.
13. CLI referencephone-harness entry points: stdin Python exec with helpers in scope, --doctor and doctor, skill, help flags, usage errors when args or TTY lack a script, and the ./phone-harness checkout launcher.
14. Helpers APIPre-imported public helpers: session, capture, ocr, gestures, scroll family, navigation, timing, and re-exported backend primitives with signatures, defaults, return shapes, and raised errors.
15. Environment variablesPHONE_HARNESS_BACKGROUND default and falsey values, PH_AGENT_WORKSPACE path override, temp capture directory under phone-harness, and how backend selection falls back when SkyLight load fails.
16. Doctor diagnosticsOrdered --doctor ladder: pyobjc, Accessibility, Screen Recording, app installed, running, window, capture size, Vision OCR; exit codes; and the note that fresh machines may need extra prompts.
17. TroubleshootingFailure modes and recovery: blank capture after Screen Recording grant, window not found, silent taps without Accessibility or focus, blocked iPhone in Use, Home Screen label misses, and type_text field focus.