# Overview
> What phone-harness exposes, who it is for, runtime assumptions (macOS Sequoia+, iPhone Mirroring, real phone), and the first docs routes to follow.
- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt
## Source Files
- `README.md`
- `pyproject.toml`
- `src/phone_harness/run.py`
- `src/phone_harness/helpers.py`
- `SKILL.md`
- `phone-harness`
---
---
title: "Overview"
description: "What phone-harness exposes, who it is for, runtime assumptions (macOS Sequoia+, iPhone Mirroring, real phone), and the first docs routes to follow."
---
`phone-harness` is a local CLI that drives a **real iPhone** through the macOS **iPhone Mirroring** window: stdin Python is `exec`'d with capture, Vision OCR, gesture, navigation, and scroll helpers already in scope. There is no daemon, no WebDriverAgent, and no jailbreak — each invocation re-queries the mirror window, captures pixels, and posts HID-level (or SkyLight background) events against global screen points.
## What it exposes
| Surface | Role |
|--------|------|
| `phone-harness` CLI | Primary entry: run a Python script from stdin with helpers pre-imported |
| `phone-harness --doctor` / `doctor` | Permission and session ladder (pyobjc → Accessibility → Screen Recording → app → window → capture → OCR) |
| `phone-harness skill` | Prints `SKILL.md` for agent skill registration |
| `./phone-harness` | Checkout launcher: `PYTHONPATH=src` + `python3 -m phone_harness.run` without a global install |
| `phone_harness.helpers` | Protected core primitives used by every script |
| `agent-workspace/agent_helpers.py` | Agent-editable extensions auto-loaded into the same script namespace |
| Env: `PHONE_HARNESS_BACKGROUND` | Default `1`: background (SkyLight) backend; `0`/`false`/`no` forces classic mirror + activate |
| Env: `PH_AGENT_WORKSPACE` | Override path for agent helpers (default: repo `agent-workspace/`) |
Console entry point (from `pyproject.toml`): `phone-harness = phone_harness.run:main`. Package requires Python `>=3.10` and PyObjC frameworks: Quartz, Vision, AppKit, ApplicationServices. Install guidance also targets Python 3.12+ with explicit `pip install` of those frameworks.
### Typical invocation
```bash
phone-harness <<'PY'
print(connection_state())
print(screen_info())
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
print([o["text"] for o in ocr()][:10])
PY
```
Helpers such as `ocr`, `tap`, `tap_text`, `scroll_collect`, `home`, `open_app`, and `wait_stable` are injected from `helpers` (plus non-underscore names from `agent_helpers.py`). Raw Quartz remains available via `import Quartz` inside the script for escape hatches the helpers do not cover.
## Who it is for
| Audience | Use |
|----------|-----|
| Coding agents (Claude Code, Codex, or any agent with a skill file) | Automate iOS-only work: open apps, tap, type, swipe, read the mirrored screen |
| Developers integrating local phone control | Thin Python harness over capture + HID events, editable workspace helpers |
| Humans supervising an agent on a real device | Pair once, grant terminal permissions, connect/lock the phone when mirroring pauses |
Use phone-harness only when the task **needs the phone** (iOS-only apps, number/2FA-bound flows, on-device UI). Prefer Mac or web when those suffice. Outward-facing or hard-to-reverse actions (messages, posts, purchases, deletes, settings changes) require explicit user consent; do not linger in personal content beyond the task.
## Runtime assumptions
| Requirement | Detail |
|-------------|--------|
| Host OS | macOS **Sequoia+** with system app **iPhone Mirroring** (`com.apple.ScreenContinuity`, `/System/Applications/iPhone Mirroring.app`) |
| Device | A **paired, real iPhone** — not a simulator. Pairing prompts need the physical phone once |
| Transport | The mirroring **window** is the whole stack: eyes = window capture + Vision OCR; hands = CGEvent HID (classic) or SkyLight event records (background default) |
| Terminal permissions | **Accessibility** (taps/keystrokes; immediate) and **Screen Recording** (capture; often needs terminal restart) |
| Session ownership | Connecting or resuming mirroring is a **physical user action**. The harness never taps Connect/Continue or poll-waits to reconnect |
| Process model | **Stateless** — no long-running service; window bounds and captures are re-queried per call |
```text
agent / shell
│ stdin Python
▼
phone-harness (run.main)
│ exec(helpers + agent_helpers)
▼
helpers ──► background.py (default) or mirror.py
│ │
│ ├─ eyes: CGWindowListCreateImage / screencapture
│ └─ hands: SLPSPostEventRecordTo / CGEvent HID
▼
ocr.py (Vision) → [{text, confidence, x, y, w, h}] global screen points
▼
iPhone Mirroring window → real iPhone
```
Unlocking the physical phone pauses mirroring ("iPhone in Use"). Agents must stop, relay the error from `ensure_mirroring()` / `connection_state()`, and wait for the user to lock or reconnect — never tap through interstitials.
## Control model
The supported agent loop is **see → act → verify**:
1. **See** — Prefer `ocr()` (text + tap-ready centers). Use `screenshot()` + a vision-capable model for unlabeled icons.
2. **Act** — `tap` / `tap_text` / gestures / `open_app` / `type_text` / scroll family. Coordinates are **global screen points**; never cache them across calls (the window moves).
3. **Verify** — `wait_stable()` then `ocr()` or `screenshot()`. There is no DOM; the capture is ground truth.
Connection gate before work:
| `connection_state()` | Meaning |
|----------------------|---------|
| `ready` | Connected; safe to proceed (`ensure_mirroring()` activates and returns the window) |
| `blocked` | Connect / "iPhone in Use" / paused interstitial (OCR markers such as `iphone in use`, `lock your iphone`, `mirroring ended`, `to connect`) |
| `no-window` | App open, no phone window |
| `not-running` | iPhone Mirroring not running |
## Input backends
| Backend | Module | Behavior |
|---------|--------|----------|
| Background (default) | `background.py` | Capture without focus steal; mouse via SkyLight private APIs. Keyboard still briefly activates (record path not fully implemented). Loads unless `PHONE_HARNESS_BACKGROUND` is falsey; falls back to classic if SkyLight load fails |
| Classic mirror | `mirror.py` | `screencapture` + `CGEventPost`; window must be frontmost or events are swallowed |
Both backends share the same coordinate space and helper API (`tap`, `drag`, `type_text`, etc. re-exported from the selected module).
## Hard product limits
- One phone, one session.
- No multi-touch (no pinch / two-finger gestures).
- No camera or Face ID flows; DRM video often captures black.
- Typing uses **HID keycodes** (US layout constraints on `type_text`); unicode key payloads are not the mirror path.
- Home Screen: label text is not the tappable icon — use agent helper `tap_icon(label)` (~35 pt above label); in-app UI uses `tap_text` normally.
- Accessibility tree inside the mirror window is empty — AppleScript `click at` fails silently; only HID/SkyLight input works.
## Repository layout
:::files
phone-harness/
├── phone-harness # Dev launcher (sh → python -m phone_harness.run)
├── pyproject.toml # package phone-harness, console script, pyobjc deps
├── SKILL.md # Agent skill body (phone-harness skill)
├── install.md # Permissions, clone path, skill install
├── README.md # Setup prompt + architecture summary
├── agent-workspace/
│ └── agent_helpers.py # Agent-edited primitives (e.g. tap_icon)
└── src/phone_harness/
├── run.py # CLI: stdin exec, --doctor, skill, help
├── helpers.py # Pre-imported public API + backend select
├── mirror.py # Classic capture + CGEvent transport
├── background.py # Default no-focus backend
├── ocr.py # Vision → screen-point boxes
└── admin.py # --doctor ladder
:::
Canonical install home is `~/.phone-harness` (editable `pip install -e .`) so agents always know where source, `SKILL.md`, and `agent-workspace/` live.
## Verification signal
After install and pairing:
```bash
phone-harness --doctor
phone-harness <<'PY'
print(screen_info())
PY
```
Success: doctor reports clear (or only non-fatal notes), and `screen_info()` returns window bounds in screen points plus `img_px`. Blank capture after granting Screen Recording almost always means the terminal was not restarted.
## Next
Clone to ~/.phone-harness, pyobjc + editable install, PATH, skill registration, and --doctor.
First stdin script: connection_state, screen_info, and a short helper block.
ready / blocked / no-window / not-running, ensure_mirroring, and physical reconnect.
Full pre-imported helper surface: session, OCR, gestures, scroll, navigation, timing.
phone-harness skill output, Claude Code / Codex install paths, and re-sync after pulls.
When not to use the phone, consent gates, and hard product limits.