# Troubleshooting

> Failure modes and recovery: blank capture after Screen Recording grant, window not found, silent taps without Accessibility or focus, blocked iPhone in Use, Home Screen label misses, and type_text field focus.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `install.md`
- `SKILL.md`
- `src/phone_harness/admin.py`
- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `agent-workspace/agent_helpers.py`

---

---
title: "Troubleshooting"
description: "Failure modes and recovery: blank capture after Screen Recording grant, window not found, silent taps without Accessibility or focus, blocked iPhone in Use, Home Screen label misses, and type_text field focus."
---

phone-harness fails in a small set of known ways: permissions that look granted but are not live, a missing or blocked iPhone Mirroring session, input that posts without an effect, and label-vs-icon / field-focus mistakes on the mirrored UI. Start with `phone-harness --doctor` (exit `0` = ladder clear, `1` = at least one hard FAIL), then match the symptom below.

## First step: doctor ladder

```bash
phone-harness --doctor
```

Ordered checks (fix the **first** FAIL; later steps depend on earlier ones):

| Order | Check | Hard fail? |
|-------|--------|------------|
| 1 | pyobjc (`Quartz`, `Vision`, `AppKit`) | Yes — exits early |
| 2 | Accessibility (`AXIsProcessTrusted`) | Yes |
| 3 | Screen Recording (`CGPreflightScreenCaptureAccess`) | Yes |
| 4 | iPhone Mirroring installed at `/System/Applications/iPhone Mirroring.app` | Yes |
| 5 | App running | No — note only |
| 6 | Mirroring window found | No — note only |
| 7 | Window capture size `> 20_000` bytes | Yes when window exists |
| 8 | Vision OCR on that capture | Note (runs when capture passes) |

Open permission panes:

```bash
open "x-apple.systempreferences:com.apple.preference.security?Privacy_Accessibility"
open "x-apple.systempreferences:com.apple.preference.security?Privacy_ScreenCapture"
```

<Warning>
`--doctor` only asserts the permissions currently known to be required. On a fresh Mac, macOS may still prompt for extra approvals the first time an action runs. If doctor is green but taps, typing, or capture silently do nothing, watch for a System Settings prompt under Privacy & Security.
</Warning>

## Blank or black capture after Screen Recording grant

**Symptom:** Capture “works” but the PNG is empty/black; doctor reports capture FAIL with size not `> 20_000` bytes; `screen_info()` / `ocr()` show no useful content.

**Cause:** Screen Recording takes effect only **after the terminal app restarts**. Granting the toggle without relaunching leaves captures blank. Doctor’s capture hint is explicit: *“capture is blank — Screen Recording permission needs a terminal restart to take effect.”*

**Not the same as:** DRM-protected video inside an app (product limit: those frames render black even with permissions correct).

**Recovery:**

<Steps>
  <Step title="Confirm Screen Recording is on">
    Enable your terminal (or IDE terminal host) under System Settings → Privacy & Security → Screen Recording.
  </Step>
  <Step title="Fully quit and reopen the terminal">
    Restart the process that runs `phone-harness`, not only a shell tab.
  </Step>
  <Step title="Re-run doctor and a capture">
```bash
phone-harness --doctor
phone-harness <<'PY'
print(screen_info())
print(screenshot())
PY
```
  </Step>
</Steps>

**Verify:** Doctor capture line shows a large byte size; `screen_info()` returns real `window` bounds and non-trivial `img_px`.

## Window not found

**Symptom:** Doctor: `mirroring window found` FAIL; scripts raise from `ensure_window` / `ensure_mirroring` about no phone window or app not running.

| Signal | Typical meaning |
|--------|-----------------|
| `connection_state() == "not-running"` | iPhone Mirroring process not up |
| `connection_state() == "no-window"` | App open, no on-screen phone window |
| RuntimeError: *isn't running* / *no phone window* | Same, from `ensure_window` / `ensure_mirroring` |

**Causes:** Phone not paired, out of Continuity range, or only a connect interstitial is showing. Window discovery uses `CGWindowListCopyWindowInfo` for owner `"iPhone Mirroring"`, layer `0`, width ≥ 100 (toolbars ignored).

**Recovery:**

1. Open **iPhone Mirroring** manually once and complete pairing (needs the physical phone).
2. Keep the phone in range with Continuity requirements met.
3. Do **not** rely on the harness to open/connect the app for you — reconnect is a user physical action.

**Verify:**

```bash
phone-harness <<'PY'
print(connection_state())  # expect "ready" (or "blocked" if interstitial)
print(find_window())       # expect {x,y,w,h,id}
PY
```

## Silent taps (Accessibility missing or focus lost)

**Symptom:** No exception; UI does not change. `tap` / `tap_text` / `swipe` appear to run.

**Causes:**

| Cause | Behavior |
|-------|----------|
| Accessibility off for the terminal | HID events not accepted; Accessibility is immediate once enabled |
| Classic mirror backend + window not frontmost | `CGEventPost` to HID is swallowed when Mirroring is not frontmost |
| Focus stolen mid-task (another app click) | Same as unfocused input on the classic backend |
| AppleScript / AX `click at` | Always silent no-op: the window is a video stream with no accessibility tree |

**Backend notes:**

- Default: background backend (`PHONE_HARNESS_BACKGROUND` default truthy) delivers mouse via SkyLight event records without stealing focus; `activate()` is a no-op.
- Classic backend (`PHONE_HARNESS_BACKGROUND=0`/`false`/`no`, or automatic fallback if SkyLight load fails): every input path calls `activate()` / `_focus()` first; if focus is lost between gestures, re-activate or re-run the helper.
- If SkyLight private symbols fail to load, helpers fall back to `mirror` rather than leaving the harness unusable.

**Recovery:**

<Steps>
  <Step title="Enable Accessibility">
    System Settings → Privacy & Security → Accessibility → enable the terminal. Re-check with `--doctor`.
  </Step>
  <Step title="If using classic mirror input">
    Ensure Mirroring is frontmost before acting, or call `activate()` after focus steals.
  </Step>
  <Step title="Force classic backend if background gestures misbehave">
```bash
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
activate()
tap_text("Notes")  # or any visible control
PY
```
  </Step>
  <Step title="Never use AppleScript clicks">
    Only harness helpers / Quartz HID (or background event records) drive the stream.
  </Step>
</Steps>

**Verify:** After `tap_text` / `tap`, run `wait_stable()` then `ocr()` or `screenshot()` — capture is the only ground truth.

## Blocked session: “iPhone in Use” and connect interstitials

**Symptom:** `connection_state()` is `"blocked"`; `ensure_mirroring()` raises and tells the user to connect / lock the phone; UI shows connect, continue, or “iPhone in Use”.

**Detection:** OCR over the live capture; session is blocked if any of these substrings appear (case-insensitive):

- `iphone in use`
- `lock your iphone`
- `mirroring ended`
- `to connect`

**Cause:** Unlocking the physical phone pauses mirroring. Resuming is physical: open Mirroring, approve prompts, and **lock the iPhone** when the interstitial says “iPhone in Use”. Tapping Connect/Continue while the phone is unlocked does nothing useful.

**Hard rules (agents and scripts):**

- STOP and relay the error to the user.
- Never tap `Connect` / `Continue`.
- Never loop-poll waiting for reconnect.
- Retry only after the user confirms they locked/connected.

**Recovery (user only):**

1. Open iPhone Mirroring.
2. If “iPhone in Use”: **lock the iPhone** so the session can resume.
3. Confirm connection, then re-run the script.

```bash
phone-harness <<'PY'
print(connection_state())
# only proceed when "ready"
ensure_mirroring()
print(screen_info())
PY
```

**Verify:** `connection_state() == "ready"` and `ensure_mirroring()` returns window bounds without raising.

## Home Screen: `tap_text` misses the icon

**Symptom:** `tap_text("Weather")` finds the label and returns success-shaped data, but the app does not launch.

**Cause:** On the Home Screen, OCR hits the **label** under the icon. The tappable icon is about **35 points above** the label center. Verified pattern in `agent-workspace/agent_helpers.py`: label tap is a no-op; icon tap launches.

| Context | Use |
|---------|-----|
| Home Screen app icon | `tap_icon("Weather")` |
| In-app buttons / list rows | `tap_text("…")` is correct |

```bash
phone-harness <<'PY'
home()
wait_stable()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:15])
PY
```

`tap_icon` raises `RuntimeError` if no label matches; on hit it `tap(h["x"], h["y"] - 35)`.

**Verify:** After launch, OCR shows app chrome, not only Home Screen icons.

## `type_text` needs a focused iOS field

**Symptom:** Key events post; no characters appear; or `ValueError: cannot type '…' via keycodes`.

**Causes:**

1. No iOS text field / keyboard focused — must tap the field and wait for the keyboard before typing.
2. Typing is **US keycode only**. Mirroring forwards raw HID keycodes and ignores unicode payloads. Unsupported characters (emoji, non-keycoded glyphs) raise `ValueError`.
3. `\n` in the string becomes `press("return")` between lines.

**Recovery:**

```bash
phone-harness <<'PY'
tap_text("New Note")   # or tap the field by coordinates
wait_stable()
type_text("hello from the harness")
press("return")
wait_stable()
print([o["text"] for o in ocr()][:20])
PY
```

**Verify:** OCR shows the typed string (or the field content changed). For Spotlight open: `open_app("Notes")` already types via the same keycode path after Cmd+3.

## Capture / OCR runtime errors

| Error pattern | Likely cause | Action |
|---------------|--------------|--------|
| `window capture failed after N tries: …` | `screencapture -l` failed (window not composited) and region fallback also failed | Ensure app running, window present, Screen Recording live; classic path activates then retries |
| `background capture failed: …` | `CGWindowListCreateImage` empty / PNG encode fail | Window gone or permission; check `find_window()` and doctor |
| `Vision OCR failed: …` | Vision request failed | Confirm capture is a valid PNG; re-run doctor OCR step |
| `cannot read image …` | Bad path / empty file | Fix capture first |
| `no visible text matches …; saw: […]` | Wrong screen or OCR miss | Read the exception’s visible list; `screenshot()` for unlabeled icons |

Classic `mirror.capture` tries window-id capture (`screencapture -x -o -l <id>`), then activates and falls back to region `-R x,y,w,h`. Success requires return code 0, file exists, size `> 1000` bytes.

## Coordinates and other silent-ish failures

| Issue | Behavior | Fix |
|-------|----------|-----|
| Cached coordinates after window move | Taps miss | Re-query every action: `ocr()`, `find_window()`, `swipe` re-read bounds |
| DRM / protected video | Black frames in capture | Product limit; cannot OCR/tap inside black content |
| Multi-touch (pinch, two-finger) | Not supported | Use single-point gestures only |
| List barely moves with slow drag | iOS bounce / no advance | Prefer `scroll` / `scroll_screen` / `scroll_collect` (wheel or background flick), not slow `drag` |

## Quick decision table

| You observe | Check | Fix |
|-------------|-------|-----|
| Black tiny capture | Screen Recording + restart | Quit terminal; re-enable; `--doctor` |
| No window | Pairing / range / app open | User opens Mirroring and connects |
| Taps no-op | Accessibility / focus / backend | Doctor; enable AX; `activate` or background path |
| Connect / “iPhone in Use” UI | `connection_state() == "blocked"` | User locks phone / reconnects; no agent taps |
| Home app won’t open | Label vs icon | `tap_icon` not `tap_text` |
| Typing vanishes | No field / bad chars | Focus field + keyboard; US keycodes only |
| Doctor green, still silent | Extra macOS prompt | Approve new Privacy & Security entry |

## Related pages

<CardGroup>
  <Card title="Doctor diagnostics" href="/doctor-diagnostics">
    Full ladder, exit codes, and fresh-machine permission notes.
  </Card>
  <Card title="Connection and session states" href="/connection-and-session">
    `ready` / `blocked` / `no-window` / `not-running` and `ensure_mirroring` gates.
  </Card>
  <Card title="Input backends" href="/input-backends">
    Background SkyLight vs classic HID focus, `PHONE_HARNESS_BACKGROUND`, fallback.
  </Card>
  <Card title="Installation" href="/installation">
    Prerequisites, editable install, and permission panes.
  </Card>
  <Card title="Navigate apps and type text" href="/navigate-and-type">
    `type_text` keycode rules, `tap_icon` vs `tap_text`, Spotlight open.
  </Card>
  <Card title="Consent and limits" href="/consent-and-limits">
    DRM black frames, single session, no multi-touch.
  </Card>
</CardGroup>