# Coordinates, capture, and OCR

> Global screen-point space, window bounds vs image pixels, capture return shape, Vision OCR boxes with confidence and tap-ready centers, and why coordinates must not be cached.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `src/phone_harness/ocr.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `src/phone_harness/helpers.py`
- `SKILL.md`

---

---
title: "Coordinates, capture, and OCR"
description: "Global screen-point space, window bounds vs image pixels, capture return shape, Vision OCR boxes with confidence and tap-ready centers, and why coordinates must not be cached."
---

phone-harness treats the iPhone Mirroring window as a video stream: every public coordinate is a **global macOS screen point** (top-left origin), the same space `screencapture -R` and HID mouse events use. Capture returns a PNG plus live window bounds; Vision OCR maps image pixels back into those bounds so each text box is already tap-ready.

## Coordinate space

| Space | Origin | Units | Used by |
|-------|--------|-------|---------|
| Global screen points | Top-left of the Mac display | Logical points | `tap(x, y)`, OCR `x`/`y`/`w`/`h`, window bounds |
| Capture image pixels | Top-left of the PNG | Pixels | `screen_info()["img_px"]`, Vision raw boxes |
| Vision normalized boxes | Bottom-left of the image | 0–1 fractions | Internal only; flipped and scaled in `ocr.recognize` |

```text
Mac display (global screen points)
┌────────────────────────────────────────────┐
│  window = {x, y, w, h, id}                 │
│       ┌──────────────────┐                 │
│       │  iPhone Mirroring│  ← video stream │
│       │  (no AX tree)    │                 │
│       └──────────────────┘                 │
└────────────────────────────────────────────┘

capture PNG (img_px = [Wpx, Hpx])
  scale: sx = w / Wpx , sy = h / Hpx
  tap point = (window.x + img_x * sx, window.y + img_y * sy)
```

Window discovery (`find_window`) returns:

| Field | Type | Meaning |
|-------|------|---------|
| `x`, `y` | float | Top-left of the mirroring window in screen points |
| `w`, `h` | float | Window size in screen points |
| `id` | int | `kCGWindowNumber` (used by window-id capture) |

Windows owned by `"iPhone Mirroring"` with layer `0` and width ≥ 100 are accepted; smaller owner panels are ignored.

<Warning>
Never cache `(x, y)` across calls. The window can move, resize, or reappear; every helper re-queries bounds on each capture/gesture. Stale points miss the phone and tap empty desktop or another app.
</Warning>

## Capture

Both backends expose the same contract:

```python
path, win = mirror.capture(path=None, retries=2)
# path: str  — PNG on disk
# win:  dict — {x, y, w, h, id} at capture time
```

### Defaults and paths

| Backend | Module | Default PNG path | Capture mechanism |
|---------|--------|------------------|-------------------|
| Classic (focus-stealing) | `mirror.py` | `$TMPDIR/phone-harness/window.png` | `screencapture -l <id>`, then `screencapture -R x,y,w,h` after `activate()` |
| Background (default) | `background.py` | `$TMPDIR/phone-harness/background.png` | `CGWindowListCreateImage` by window id (works unfocused / occluded) |

Temp directory: `Path(tempfile.gettempdir()) / "phone-harness"` (created on import).

### Classic capture path

1. Resolve window via `find_window()` or `ensure_window()`.
2. Prefer window-only: `screencapture -x -o -l <id>` (no shadow; fails if the window is not frontmost/composited).
3. On failure: `activate()`, re-read bounds, region capture `-R x,y,w,h`.
4. Success requires return code 0, file exists, and size **> 1000** bytes.
5. After `retries + 1` attempts (default `retries=2`): `RuntimeError("window capture failed after …")`.

### Background capture path

1. Resolve window (no activation).
2. `CGWindowListCreateImage` with `kCGWindowListOptionIncludingWindow`, `kCGWindowImageBoundsIgnoreFraming | kCGWindowImageNominalResolution`.
3. Encode PNG via Core Graphics destination.
4. Failures: empty image, encode failure → retry; then `RuntimeError("background capture failed: …")`.

### Public helpers over capture

| Helper | Signature | Returns |
|--------|-----------|---------|
| `screenshot` | `screenshot(path=None)` | PNG path only (`str`) |
| `screen_info` | `screen_info()` | `{window, frontmost, img_px}` |
| `ocr` | `ocr(min_confidence=0.3)` | list of OCR boxes (see below) |
| `wait_stable` | `wait_stable(timeout=6.0, interval=0.5, settle=2)` | `True` if consecutive capture digests match; else `False` |

`screen_info` fields:

| Field | Shape | Notes |
|-------|-------|-------|
| `window` | `{x, y, w, h, id}` | Screen-point bounds from the capture used for sizing |
| `frontmost` | `bool` | Whether iPhone Mirroring is the frontmost Mac app |
| `img_px` | `[width, height]` | Pixel dimensions of the PNG (`ocr.image_size`) |

`img_px` and `window` `w`/`h` often differ on Retina displays. Scale factors are:

```text
sx = window["w"] / img_px[0]
sy = window["h"] / img_px[1]
```

## OCR (Vision element tree)

`ocr()` is the preferred read path for anything with a text label. It captures, runs Apple Vision at **Accurate** recognition level, and maps every observation into global screen points.

### Return shape

Each element:

| Field | Type | Meaning |
|-------|------|---------|
| `text` | `str` | Top candidate string |
| `confidence` | `float` | Rounded to 3 decimals (0–1) |
| `x`, `y` | `float` | **Center** of the box in global screen points (rounded to 1 decimal) — pass straight to `tap` |
| `w`, `h` | `float` | Box size in screen points (rounded to 1 decimal) |

Default filter: `min_confidence=0.3`. Boxes below that threshold are dropped by the helper (raw `recognize` returns all candidates).

### Mapping math (`ocr.recognize`)

Vision boxes are normalized with a **bottom-left** origin. Conversion:

1. Pixel box from normalized `boundingBox` (Y flipped to top-left image space).
2. Scale by `sx`, `sy` from window points / image pixels.
3. Offset by `window["x"]`, `window["y"]`.
4. Emit **center**: `(px + pw/2)`, `(py_top + ph/2)`.

```python
# Conceptual — implemented in src/phone_harness/ocr.py
sx = window["w"] / img_w
sy = window["h"] / img_h
x = window["x"] + (px + pw / 2) * sx
y = window["y"] + (py_top + ph / 2) * sy
```

Failures raise `RuntimeError("Vision OCR failed: …")` or `RuntimeError("cannot read image …")` if the PNG cannot be opened.

### Related readers

| Helper | Behavior |
|--------|----------|
| `find_text(query, exact=False)` | Case-insensitive substring (or exact) over a fresh `ocr()` |
| `tap_text(query, index=0, exact=False)` | Tap center of match; on miss, raises with up to 30 visible strings |
| `_content_texts` (scroll internals) | OCR filtered to ~6%–92% vertical band to ignore status bar / home indicator |

## Manual points (icons without labels)

When there is no OCR target (glyph-only UI):

1. `info = screen_info()` — or `path = screenshot()` and keep bounds from a paired capture.
2. Inspect the PNG (human or vision model) in **image pixel** space.
3. Convert to screen points:

```python
info = screen_info()
win = info["window"]
Wpx, Hpx = info["img_px"]
sx = win["w"] / Wpx
sy = win["h"] / Hpx
# img_x, img_y measured in the PNG
tap(win["x"] + img_x * sx, win["y"] + img_y * sy)
```

Home Screen labels are not tappable targets: the icon sits ~35 screen points **above** the label. Use `tap_icon("Weather")` from agent helpers, not raw `tap_text`, for launcher icons.

## Why coordinates must not be cached

| Fact | Implication |
|------|-------------|
| Transport is **stateless** — no daemon holds window geometry | Each `phone-harness` invocation and each helper call re-discovers the window |
| `ocr()`, `swipe()`, `scroll_*`, `screen_info()`, `screenshot()` all capture or re-query bounds | Fresh `(x, y)` every time is the design |
| User can drag/resize the mirroring window mid-task | Cached points from an earlier `ocr()` miss |
| Session pause / reconnect can reposition the window | Re-run `ocr()` after `ensure_mirroring()` succeeds again |

Pattern: **read → act → re-read**. After a tap or gesture, call `wait_stable()` then a new `ocr()` / `screenshot()`. There is no accessibility tree or DOM; the PNG is ground truth.

```bash
phone-harness <<'PY'
print(screen_info())
for o in ocr()[:8]:
    print(o["text"], o["confidence"], o["x"], o["y"])
hit = find_text("Settings")
if hit:
    tap(hit[0]["x"], hit[0]["y"])  # or tap_text("Settings")
wait_stable()
print([o["text"] for o in ocr()][:12])
PY
```

## Backend note on coordinates

`PHONE_HARNESS_BACKGROUND` selects SkyLight background input vs classic HID + activate. **Both backends share the same global screen-point convention and the same `capture → (path, win)` shape**, so `ocr`, `tap_text`, and scroll helpers do not change coordinate math when the backend switches. Background mouse delivery also stores local window-relative points (`gx - win["x"]`, `gy - win["y"]`) inside the event record; callers still pass global points only.

## Failure modes (capture / OCR)

| Symptom | Likely cause | Check |
|---------|--------------|-------|
| Capture empty / tiny file | Screen Recording not effective until terminal restart | `phone-harness --doctor` (expects size > ~20 KB) |
| `window capture failed` / `background capture failed` | No window, permissions, or encoding failure | `connection_state()`, window presence, Screen Recording |
| `Vision OCR failed` | Vision/pyobjc issue | Doctor OCR step; pyobjc Vision framework installed |
| OCR boxes empty but UI has text | Confidence filter, blocked interstitial, DRM black frame | Lower `min_confidence`, inspect `screenshot()`, check blocked markers |
| Tap misses after saved coords | Window moved or layout changed | Re-run `ocr()`; do not reuse old centers |
| `tap_text` raises with `saw: […]` | Label not present or OCR miss | Read the visible list; screenshot for non-text UI |

## Related pages

<CardGroup>
  <Card title="See, act, and verify" href="/see-act-verify">
    OCR-first reading, tap_text, wait_stable, and capture-as-ground-truth after every action.
  </Card>
  <Card title="Input backends" href="/input-backends">
    Background SkyLight capture/input versus classic screencapture + CGEvent activate.
  </Card>
  <Card title="Helpers API" href="/helpers-api">
    Full signatures for capture, ocr, gestures, and return shapes.
  </Card>
  <Card title="Connection and session states" href="/connection-and-session">
    ready / blocked / no-window / not-running gates before capture is meaningful.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Blank capture after Screen Recording grant, silent taps, Home Screen label misses.
  </Card>
</CardGroup>