# Consent and limits

> When not to use the phone, outward-facing and irreversible actions that require user consent, and hard product limits (one session, no multi-touch, DRM black frames, keycode typing).

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `SKILL.md`
- `README.md`
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`

---

---
title: "Consent and limits"
description: "When not to use the phone, outward-facing and irreversible actions that require user consent, and hard product limits (one session, no multi-touch, DRM black frames, keycode typing)."
---

phone-harness drives the user's **real iPhone** through the macOS iPhone Mirroring window. Policy for when to use it, what requires an explicit user yes, and what the transport cannot do lives in `SKILL.md` (agent skill body) and is enforced in code by `ensure_mirroring()` / `connection_state()` in `helpers.py`, single-pointer input in `mirror.py` / `background.py`, and US-layout HID `type_text`.

## When not to use the phone

Prefer Mac or web whenever the task can complete there: a website, an API, or an app with a web equivalent. Use phone-harness only when the work genuinely needs the device:

| Use phone-harness | Prefer Mac / web |
| --- | --- |
| iOS-only apps | Browser or desktop app equivalent |
| Flows tied to the user's phone number or 2FA on-device | Server APIs, email links, desktop auth |
| How something looks or behaves on the phone | Same content available on desktop |

Navigating and reading for the user's own task is allowed. Do not linger in personal content (Messages, Photos, Mail) beyond what the task needs.

## Consent: stop and ask

This is the user's real phone. **Stop and ask before anything outward-facing or hard to reverse**, including:

- Sending a message
- Posting
- Purchasing
- Deleting
- Changing settings

Navigating UI and reading screen content for an in-scope task does not require per-step consent. Destructive or public side effects do.

<Warning>
The harness does not implement a consent API or dry-run mode. Consent is an agent/operator policy surface: halt, describe the intended action, and proceed only after the user confirms.
</Warning>

Setup-time actions that only the user can perform (pairing iPhone Mirroring, granting Accessibility and Screen Recording) are the same class of gate: the agent walks the user through them and waits; it does not bypass prompts.

## Connection is physical — not agent-automatable

Reconnecting or resuming mirroring is a **physical** user action: open iPhone Mirroring, approve prompts, and when the interstitial says **iPhone in Use**, **lock the iPhone**. The harness never connects the phone for the agent.

### Session gates

| Helper | Role |
| --- | --- |
| `connection_state()` | Returns `'ready'`, `'blocked'`, `'no-window'`, or `'not-running'` |
| `ensure_mirroring()` | Proceeds only when `'ready'`; otherwise raises with user-facing instructions |

`connection_state()` treats OCR text matching any of these markers as `'blocked'`:

- `iphone in use`
- `lock your iphone`
- `mirroring ended`
- `to connect`

On non-ready states, `ensure_mirroring()` raises `RuntimeError` (does not launch the app, tap Connect/Continue, or poll). Example messages instruct the user to open the app, connect the phone, or lock the phone when **iPhone in Use** appears.

### Agent rules when gated

1. **STOP** and relay the error message to the user.
2. **Never** tap `Connect` / `Continue`.
3. **Never** loop-poll waiting for connection (burns time; unlock-state resume only the user can fix).
4. Retry **once after the user confirms** they reconnected — not before.

Unlocking the physical phone pauses the session (**iPhone in Use**). Do not tap through the resume interstitial; ask the user to lock/connect.

## Hard product limits

Documented product ceilings (README **Limits**, SKILL **Gotchas**) and the implementations that produce them:

| Limit | Behavior |
| --- | --- |
| One phone, one session | Single iPhone Mirroring window; unlock pauses mirroring |
| No multi-touch | Mouse maps 1:1 to one touch — no pinch, no two-finger gestures |
| No camera / Face ID flows | Not exposed through the mirroring input path |
| DRM video | Protected content captures as black frames |
| Keycode typing only | Mirroring forwards raw HID keycodes; unicode string payloads are ignored |
| OCR is text, not semantics | Labels have coordinates; unlabeled icons need `screenshot()` + vision judgment |

### One session and pause semantics

There is no multi-device or multi-session API. Window bounds and captures are re-queried per call (stateless transport), but the **mirroring session** is still one paired phone. When the user unlocks the physical device, the stream shows a blocked interstitial until they lock/reconnect.

### Input: single pointer only

`tap`, `long_press`, and `drag` post a single left-button mouse stream (`LeftMouseDown` / `Dragged` / `Up`). There is no multi-finger synthesizer — pinches and other multi-touch gestures are out of scope.

### Capture: black DRM frames

Window capture (`mirror.capture` / background `CGWindowListCreateImage`) returns pixels as macOS renders them. DRM-protected video commonly appears black; treat black frames as a content/transport limit, not a permission failure (blank capture after granting Screen Recording without restarting the terminal is a different failure — see diagnostics docs).

### Typing: US HID keycodes

`type_text(text, delay=0.03)` and `press(combo)` map through real keycodes (US layout). Characters without a keycode mapping raise:

```text
ValueError: cannot type '<char>' via keycodes
```

Examples of untypable content: emoji and other non-keycode glyphs. `\n` becomes Return. Prerequisites:

- An iOS text field must be focused first (tap the field, wait for the keyboard, then type).
- Background backend still briefly uses the keyboard path that requires the app path for keys; mouse actions can stay backgrounded.

Supported character set is the US keyboard plus shifted punctuation defined in `mirror.py` (`_KEYCODES`, `_PUNCT_KEYCODES`, `_SHIFTED`). Unknown `press` keys raise `ValueError: unknown key ...`.

## Related operational constraints

These are not consent gates, but they bound safe automation the same way:

| Constraint | Implication |
| --- | --- |
| Window is a video stream | No accessibility tree inside the phone UI; AppleScript `click at` fails silently |
| Coordinates are not stable | Do not cache `(x, y)` across calls; re-run `ocr()` / re-query bounds |
| Unfocused classic input is swallowed | Classic mirror backend needs frontmost window; silent no-ops if focus is stolen |
| Home Screen labels vs icons | `tap_text` hits the label; use `tap_icon` (agent helper) for Home Screen icons |

## Practical decision checklist

Use this order before acting:

1. Can this run on Mac or web? If yes, do not open the phone.
2. Is `connection_state()` `'ready'`? If not, stop and ask the user to connect/lock — never tap Connect.
3. Is the next action outward-facing or hard to reverse? If yes, stop and ask.
4. Will the step need multi-touch, Face ID, camera, DRM-protected video, or non-US/emoji typing? Plan an alternate path or human handoff.
5. After any allowed action: `wait_stable()` then `ocr()` / `screenshot()` — capture is ground truth.

```text
  task request
       │
       ├─ Mac/web enough? ──────────────► do not use phone
       │
       ├─ connection_state != ready? ───► stop; user reconnects
       │
       ├─ outward / irreversible? ──────► stop; ask consent
       │
       └─ within hard limits? ──────────► act → verify capture
```

## Next

<CardGroup>
  <Card title="Connection and session states" href="/connection-and-session">
    `ready` / `blocked` / `no-window` / `not-running`, blocked markers, and why reconnect is physical.
  </Card>
  <Card title="Navigate apps and type text" href="/navigate-and-type">
    `type_text` US keycode constraints, `press` combos, Home Screen vs in-app taps.
  </Card>
  <Card title="See, act, and verify" href="/see-act-verify">
    OCR-first reads, post-action verification, and when screenshots are required.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Blank/black capture, silent taps, blocked interstitials, and typing focus failures.
  </Card>
</CardGroup>