# See, act, and verify

> OCR-first reading, tap and tap_text, wait_stable, screenshot for unlabeled icons, and the capture-as-ground-truth verification loop after every action.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `src/phone_harness/helpers.py`
- `src/phone_harness/ocr.py`
- `src/phone_harness/mirror.py`
- `agent-workspace/agent_helpers.py`
- `SKILL.md`

---

---
title: "See, act, and verify"
description: "OCR-first reading, tap and tap_text, wait_stable, screenshot for unlabeled icons, and the capture-as-ground-truth verification loop after every action."
---

Every phone-harness script runs as stdin Python with helpers pre-imported from `phone_harness.helpers`. The control loop is fixed: **read the screen with Vision OCR**, **act at global screen points**, **wait until pixels settle**, then **re-capture to confirm the UI actually changed**. There is no accessibility tree or DOM inside the iPhone Mirroring window — the PNG capture is the only ground truth.

```text
  ocr() / screenshot()     →  see
  tap_text() / tap(x, y)   →  act
  wait_stable()            →  settle animations
  ocr() / screenshot()     →  verify (ground truth)
```

## Prerequisites

| Requirement | Why it matters |
|-------------|----------------|
| Session `connection_state() == "ready"` | Interstitials and disconnects block real UI |
| Screen Recording granted | Capture / OCR fail or return empty |
| Accessibility granted | Classic-backend taps/keys are silent without it |
| Coordinates treated as ephemeral | Window can move between calls |

Gate with `ensure_mirroring()` or check `connection_state()` before the loop. Reconnect is physical (user only) — see [Connection and session](/connection-and-session).

## See: OCR-first reading

Prefer `ocr()` over viewing screenshots whenever a control has a text label. `ocr()` captures the mirroring window, runs Apple Vision at accurate recognition level, and returns boxes with **tap-ready centers in global screen points**.

### `ocr(min_confidence=0.3)`

Captures, recognizes, and filters by confidence.

| Field | Type | Meaning |
|-------|------|---------|
| `text` | `str` | Recognized string |
| `confidence` | `float` | Rounded to 3 decimals; default floor `0.3` |
| `x`, `y` | `float` | Box **center** in screen points (pass to `tap`) |
| `w`, `h` | `float` | Box size in screen points |

Vision uses a bottom-left origin; the harness flips and scales into the window’s top-left screen-point space so centers are immediately tappable.

```bash
phone-harness <<'PY'
boxes = ocr()
print([(o["text"], o["x"], o["y"], o["confidence"]) for o in boxes][:20])
PY
```

Filter in Python before printing large dumps — full screens can be dense.

### `find_text(query, exact=False)`

Returns the subset of `ocr()` hits matching `query` (case-insensitive).

| Mode | Match rule |
|------|------------|
| `exact=False` (default) | Substring: `query.lower() in text.lower()` |
| `exact=True` | Full string equality (case-insensitive) |

```python
hits = find_text("Settings")
exact = find_text("Done", exact=True)
```

### Why not cache OCR coordinates

`ocr()`, `find_text()`, `swipe()`, and capture re-query window bounds each call. The mirroring window can move. Re-read after every navigation or layout change; do not store `(x, y)` across steps.

## Act: tap and tap_text

### `tap_text(query, index=0, exact=False)`

Finds text, taps the selected hit’s center, returns the hit dict.

| Param | Default | Role |
|-------|---------|------|
| `query` | required | Label to find |
| `index` | `0` | Which hit when several match |
| `exact` | `False` | Passed through to `find_text` |

On miss, raises `RuntimeError` including up to 30 currently visible OCR strings so the next step is informed:

```text
RuntimeError: no visible text matches 'New Note'; saw: ['Notes', 'Folders', ...]
```

```bash
phone-harness <<'PY'
hit = tap_text("New Note")
print(hit)  # {text, confidence, x, y, w, h}
PY
```

### `tap(x, y)`

Direct tap at global screen points. Re-exported from the active backend (`background` by default, `mirror` on fallback).

| Backend | Delivery | Focus |
|---------|----------|--------|
| Background (default) | SkyLight event records to the mirroring process | Does not steal focus |
| Classic (`PHONE_HARNESS_BACKGROUND=0` or SkyLight load fail) | CGEvent HID + `activate()` | Brings window frontmost |

Both use the same screen-point convention, so `tap_text` / `ocr` stay backend-agnostic. Details: [Input backends](/input-backends).

### Home Screen vs in-app labels

| Surface | Prefer | Reason |
|---------|--------|--------|
| In-app buttons, list rows, nav items | `tap_text("…")` | Label is usually the hit target |
| Home Screen app icons | `tap_icon("Weather")` | Label is **below** the icon; tapping the label is a no-op |

`tap_icon` (from `agent-workspace/agent_helpers.py`, auto-loaded) finds the label then taps at `(x, y - 35)`:

```python
tap_icon("Weather")  # not tap_text("Weather") on Home Screen
```

## Wait: `wait_stable`

After any action that may animate (open app, navigate, dismiss sheet), settle before reading.

### Signature

```python
wait_stable(timeout=6.0, interval=0.5, settle=2) -> bool
```

| Param | Default | Meaning |
|-------|---------|---------|
| `timeout` | `6.0` | Max seconds to wait |
| `interval` | `0.5` | Sleep between captures |
| `settle` | `2` | Consecutive identical captures required |

### Behavior

1. Capture the window to PNG.
2. MD5 the file bytes.
3. If this digest matches the previous one, increment a streak; else reset.
4. Return `True` when the streak reaches `settle - 1` (i.e. `settle` identical frames with default `2`).
5. Return `False` if the deadline expires first.

Status-bar clock ticks once a minute, so identical-frame near-misses are rare. `open_app()` already calls `wait_stable()` after Spotlight launch.

```python
tap_text("Done")
if not wait_stable():
    # animation still running or capture unstable — still re-read
    pass
```

Also available: `wait(seconds=1.0)` for fixed sleeps when pixel settle is the wrong tool.

## Screenshot for unlabeled icons

OCR sees **text**, not semantics. Glyph-only controls need a visual pass.

<Steps>
  <Step title="Capture">
    Call `screenshot()` (optional path; default under the harness temp dir). Returns the PNG path.
  </Step>
  <Step title="View and choose a point">
    Open the image (agent vision model or human). Pick the control in image pixels.
  </Step>
  <Step title="Convert to screen points">
    Use `screen_info()` for window bounds and capture size:

    ```python
    info = screen_info()
    # info["window"] = {x, y, w, h, id}
    # info["img_px"] = [pixel_w, pixel_h]
    # info["frontmost"] = bool
    ```

    Map image pixel `(px, py)` to screen points:

    ```text
    x = window["x"] + px * (window["w"] / img_px[0])
    y = window["y"] + py * (window["h"] / img_px[1])
    ```
  </Step>
  <Step title="Tap and verify">
    `tap(x, y)` then `wait_stable()` + `ocr()` / `screenshot()` again.
  </Step>
</Steps>

```bash
phone-harness <<'PY'
path = screenshot()
print(path)
print(screen_info())
# after choosing (px, py) from the image:
# info = screen_info()
# win, (iw, ih) = info["window"], info["img_px"]
# tap(win["x"] + px * win["w"] / iw, win["y"] + py * win["h"] / ih)
# wait_stable()
# print([o["text"] for o in ocr()][:15])
PY
```

Combine `screenshot()` with `ocr()` when you need both geometry context and text anchors.

## Verify: capture as ground truth

There is no DOM, no AX tree inside the stream, and no assert API. **After every action**, re-read:

1. `wait_stable()` — animation done (or timed out).
2. `ocr()` — confirm expected labels / absence of prior labels.
3. `screenshot()` — when the success signal is visual (icons, charts, DRM-free imagery).

```bash
phone-harness <<'PY'
ensure_mirroring()
before = {o["text"] for o in ocr()}
tap_text("Folders")  # example in-app label
wait_stable()
after = {o["text"] for o in ocr()}
print("appeared:", sorted(after - before)[:20])
print("left:", sorted(before - after)[:20])
PY
```

Canonical agent loop from the product surface (`SKILL.md`):

```text
ocr() → decide → tap_text / tap → wait_stable() → ocr() / screenshot() → next step
```

<Warning>
If verification still shows the previous screen, do not assume the tap “worked.” Re-check focus/backend permissions, blocked interstitials, and Home Screen label-vs-icon targeting before retrying.
</Warning>

## End-to-end example

```bash
phone-harness <<'PY'
ensure_mirroring()
print(connection_state(), screen_info())

# See
labels = [o["text"] for o in ocr() if o["confidence"] >= 0.5]
print("visible:", labels[:15])

# Act (in-app style)
try:
    hit = tap_text("Notes")
except RuntimeError as e:
    print(e)  # includes what OCR saw
    raise

# Settle + verify
ok = wait_stable()
print("stable:", ok)
print([o["text"] for o in ocr()][:15])
PY
```

Home Screen launch pattern (agent helper):

```bash
phone-harness <<'PY'
ensure_mirroring()
home()
wait_stable()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:20])  # expect forecast chrome, not Home icons
PY
```

## Failure modes in this loop

| Symptom | Likely cause | Recovery |
|---------|--------------|----------|
| `tap_text` raises with long `saw: [...]` | Wrong label, wrong screen, or OCR miss | Read exception list; `screenshot()` if unlabeled |
| Tap “succeeds,” UI unchanged | Home Screen label hit; silent focus/Accessibility issue; blocked interstitial | Use `tap_icon` on Home; re-activate / doctor; `connection_state()` |
| `wait_stable` returns `False` | Long animation, capture thrash, or timeout too short | Increase `timeout`; still re-OCR; avoid treating settle as hard fail only |
| OCR empty / low confidence | Screen Recording, black DRM frame, interstitial | `--doctor`; avoid DRM video; check session |
| Wrong target after delay | Cached coordinates | Re-run `ocr()` / `find_text` immediately before `tap` |

Session and permission ladders: [Doctor diagnostics](/doctor-diagnostics), [Troubleshooting](/troubleshooting).

## Related helpers (out of scope here)

| Concern | Helpers | Docs |
|---------|---------|------|
| Capture math, Vision boxes, no-cache coords | `screenshot`, `ocr`, `screen_info`, backends | [Coordinates, capture, and OCR](/coordinates-capture-ocr) |
| App switch, Spotlight, keycodes | `home`, `app_switcher`, `open_app`, `type_text`, `press` | [Navigate apps and type text](/navigate-and-type) |
| Lists and end detection | `scroll_screen`, `scroll_until`, `scroll_collect` | [Scroll and collect lists](/scroll-lists) |
| Full signatures | All public helpers | [Helpers API](/helpers-api) |

## Next

<CardGroup>
  <Card title="Coordinates, capture, and OCR" href="/coordinates-capture-ocr">
    Screen-point space, window bounds vs image pixels, and OCR box shape.
  </Card>
  <Card title="Navigate apps and type text" href="/navigate-and-type">
    home, Spotlight open_app, type_text constraints, and tap_icon vs tap_text.
  </Card>
  <Card title="Helpers API" href="/helpers-api">
    Signatures, defaults, return shapes, and errors for every public helper.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Silent taps, blank capture, blocked session, and label misses.
  </Card>
</CardGroup>