# Helpers API

> Pre-imported public helpers: session, capture, ocr, gestures, scroll family, navigation, timing, and re-exported backend primitives with signatures, defaults, return shapes, and raised errors.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `src/phone_harness/ocr.py`
- `agent-workspace/agent_helpers.py`
- `src/phone_harness/run.py`

---

---
title: "Helpers API"
description: "Pre-imported public helpers: session, capture, ocr, gestures, scroll family, navigation, timing, and re-exported backend primitives with signatures, defaults, return shapes, and raised errors."
---

`phone-harness` injects every non-underscore name from `src/phone_harness/helpers.py` into stdin scripts, plus public names loaded from `agent-workspace/agent_helpers.py`. Backend primitives (`tap`, `long_press`, `drag`, `press`, `type_text`, `activate`, `find_window`) are re-exported from the selected input backend; higher-level helpers compose capture + OCR + those primitives.

## How the namespace is built

```text
phone-harness (stdin Python)
        │
        ▼
run.main()  →  import helpers
        │
        ├─ select backend (background default, else mirror)
        ├─ re-export: tap, long_press, drag, press, type_text, activate, find_window
        ├─ define session / OCR / gesture / scroll / nav / timing helpers
        └─ load PH_AGENT_WORKSPACE/agent_helpers.py → globals (e.g. tap_icon)
        │
        ▼
exec(script, {k: v for k,v in vars(helpers) if not k.startswith("_")})
```

| Source | What enters the script namespace |
|--------|----------------------------------|
| `helpers.py` | All public functions and re-exports listed below |
| Active backend (`background` or `mirror`) | Bound through the re-exports; not imported by name in scripts |
| `agent_helpers.py` | Public defs only (names not starting with `_`) |
| Private helpers (`_win`, `_content_texts`, …) | **Not** injected |

<Note>
Raw Quartz remains available: `import Quartz` inside a script for anything the helpers do not cover.
</Note>

### Backend selection

| Condition | Module used |
|-----------|-------------|
| `PHONE_HARNESS_BACKGROUND` unset or truthy (default `"1"`) | Prefer `background` (SkyLight, no focus steal) |
| Env is `"0"`, `"false"`, or `"no"` (case-insensitive) | Force `mirror` (CGEvent HID + activate) |
| Background import fails | Fall back to `mirror`; selection flag cleared |

Paths: `src/phone_harness/helpers.py`, `background.py`, `mirror.py`.

## Shared data shapes

### Window bounds

Returned by `find_window()`, `ensure_mirroring()`, and as the second value of backend `capture()`:

```python
{"x": float, "y": float, "w": float, "h": float, "id": int}
```

Coordinates are **global screen points** (top-left origin). OCR centers and `tap(x, y)` use the same space.

### OCR box

```python
{
  "text": str,
  "confidence": float,  # rounded to 3 decimals
  "x": float,           # box center, screen points
  "y": float,
  "w": float,           # size in screen points
  "h": float,
}
```

Vision uses a bottom-left normalized box; `ocr.recognize` flips Y and scales image pixels → window points so `(x, y)` is tap-ready.

### Temp captures

Default PNGs land under `{tempdir}/phone-harness/` (`window.png` for mirror, `background.png` for background).

---

## Session and connection

### `connection_state()`

**Returns:** `"ready"` | `"blocked"` | `"no-window"` | `"not-running"`

| Value | Condition |
|-------|-----------|
| `not-running` | `running_app()` is `None` |
| `no-window` | App running, `find_window()` is `None` |
| `blocked` | Window capturable and OCR text matches a blocked interstitial marker |
| `ready` | Window capturable and not blocked |

Blocked markers (case-insensitive substring over joined OCR text):

- `iphone in use`
- `lock your iphone`
- `mirroring ended`
- `to connect`

Performs a capture + OCR when a window exists. Does not launch the app or tap Connect.

### `ensure_mirroring()`

**Returns:** window bounds dict when state is `ready` (after `activate()`).

**Raises:** `RuntimeError` with a user-facing reconnect message:

| State | Error intent |
|-------|----------------|
| `not-running` | Open iPhone Mirroring and connect the phone |
| `no-window` | Connect the phone in the already-open app |
| `blocked` | User must connect / lock iPhone if "iPhone in Use"; agent must not tap Connect |

Never launches the app, taps Connect/Continue, or polls to reconnect.

### `screen_info()`

**Returns:**

```python
{
  "window": {"x", "y", "w", "h", "id"},
  "frontmost": bool,
  "img_px": [width_px, height_px],
}
```

Captures once, then reads PNG pixel size via Vision-friendly `image_size`.

---

## Capture

### `screenshot(path=None)`

| Param | Default | Notes |
|-------|---------|-------|
| `path` | Backend default under temp `phone-harness/` | Optional PNG path |

**Returns:** `str` path to the PNG (window bounds discarded).

**Raises:** backend `RuntimeError` after retries if capture fails (`window capture failed…` or `background capture failed…`).

Backend `capture(path=None, retries=2)` returns `(path, window)` but is **not** re-exported as a top-level helper name; use `screenshot` or `screen_info` / `ocr`.

---

## Reading the screen (OCR)

### `ocr(min_confidence=0.3)`

Captures, runs Vision Accurate recognition, filters by confidence.

**Returns:** `list[OCR box]` sorted as Vision returns them.

| Param | Default | Notes |
|-------|---------|-------|
| `min_confidence` | `0.3` | Boxes with lower confidence dropped |

**Raises:** `RuntimeError` from capture or `Vision OCR failed: …` / `cannot read image …`.

### `find_text(query, exact=False)`

| Param | Default | Notes |
|-------|---------|-------|
| `query` | required | Case-insensitive |
| `exact` | `False` | `True` → full string equality; else substring |

**Returns:** matching OCR boxes (via a fresh `ocr()` call with default confidence).

### `tap_text(query, index=0, exact=False)`

Finds matches, taps `hits[index]` center with `tap(x, y)`.

**Returns:** the tapped OCR box dict.

**Raises:** `RuntimeError` if no match — message includes up to 30 currently visible text strings for recovery.

---

## Re-exported input primitives

Bound to the active backend at import time. Coordinates are global screen points.

### `tap(x, y)`

Single left-click style touch at `(x, y)`.

| Backend | Behavior |
|---------|----------|
| `background` | SkyLight event record down/up; no focus change |
| `mirror` | Activates app, HID mouse move + down/up |

### `long_press(x, y, duration=0.8)`

Hold down for `duration` seconds, then up.

### `drag(x1, y1, x2, y2, duration=0.35, steps=14)`

Touch-drag (iOS swipe). Intermediate dragged events along the segment.

### `press(combo)`

Keyboard combo string, lowercased and split on `+`.

| Example | Meaning |
|---------|---------|
| `press("return")` | Return / Enter |
| `press("cmd+1")` | Home (also used by `home()`) |
| `press("cmd+2")` | App Switcher |
| `press("cmd+3")` | Spotlight |

**Raises:** `ValueError` if the key part is unknown.

Known keys include: `return`/`enter`, `tab`, `space`, `delete`/`backspace`, `escape`/`esc`, arrows, digits `0`–`9`, letters `a`–`z`.

Modifiers: `cmd`, `shift`, `alt`/`option`, `ctrl`.

### `type_text(text, delay=0.03)`

Types via **US keycodes** (iPhone Mirroring ignores Unicode keyboard payloads). `\n` presses return between lines.

**Raises:** `ValueError` for characters with no keycode mapping (emoji, many non-US symbols).

### `activate()`

| Backend | Behavior |
|---------|----------|
| `mirror` | Bring iPhone Mirroring frontmost; **raises** if app not running; does not launch |
| `background` | No-op (returns `None`) |

### `find_window()`

**Returns:** window dict or `None`. On-screen layer-0 window owned by `"iPhone Mirroring"` with width ≥ 100.

---

## Gestures relative to the phone window

These call `ensure_window()` (or the backend equivalent) so a missing window raises before input.

### `swipe(direction, distance=0.4)`

| Param | Default | Notes |
|-------|---------|-------|
| `direction` | required | `'up'` \| `'down'` \| `'left'` \| `'right'` |
| `distance` | `0.4` | Fraction of window width/height |

Finger motion convention: `swipe("up")` moves the finger up (content moves up / scrolls down). Implemented as a **fast short drag** (`duration=0.12`, `steps=6`) for momentum flicks (Home Screen pages, carousels).

**Raises:** `ValueError` for unknown direction.

### `scroll(amount=300)`

Scroll-gesture at window center via `scroll_wheel`. Positive `amount` scrolls content **down** (trackpad two-finger-up semantics: helper passes `-amount` to the backend).

Prefer `scroll_screen` / `scroll_collect` for lists; use `swipe` when momentum pages matter.

---

## Scroll family (lists)

End-of-list is decided by **screen movement** (OCR text-set Jaccard overlap after a settle window), never solely by whether a parser found new items.

Content OCR for movement uses the middle band of the window (status bar and home/nav strip excluded): roughly top 6% and bottom 8% cropped out, min confidence `0.4`.

### `scroll_screen(direction="up", amount=0.6, settle=2.5, moved_thresh=0.6)`

One scroll step, then wait until content text stabilizes or `settle` seconds elapse.

| Param | Default | Notes |
|-------|---------|-------|
| `direction` | `"up"` | `'up'` reveals content below; `'down'` reveals above |
| `amount` | `0.6` | Fraction of window height for the gesture magnitude |
| `settle` | `2.5` | Seconds to allow lazy-load before judging stillness |
| `moved_thresh` | `0.6` | `moved` is `False` when overlap ≥ threshold |

**Returns:**

```python
{
  "moved": bool,
  "overlap": float,   # Jaccard of before/after text sets, 3 decimals
  "before": frozenset[str],
  "after": frozenset[str],
  "boxes": list[OCR box],  # settled content-area OCR
}
```

**Raises:** `ValueError` if direction is not `'up'` or `'down'`.

Implementation notes:

- Mirror backend: true scroll-wheel events at window center.
- Background backend: wheel events do not reach an unfocused app; `scroll_wheel` is implemented as a **fast vertical flick** with the same sign semantics.

### `scroll_until(done, direction="up", amount=0.6, max_scrolls=60, settle=2.5)`

| Param | Default | Notes |
|-------|---------|-------|
| `done` | required | Callable `done(boxes) -> truthy to stop` |
| `max_scrolls` | `60` | Cap on scroll steps after the initial check |

Checks current content OCR first. On non-movement, allows one retry (`stale >= 2`) with a short sleep and `activate()` before treating the end as reached.

**Returns:** truthy value from `done`, or `None` if the list stops moving or `max_scrolls` is exhausted without a hit.

### `scroll_collect(extract=None, key=None, direction="up", amount=0.6, max_scrolls=400, end_after=3, settle=2.5, on_progress=None)`

Scroll while extracting and de-duplicating items.

| Param | Default | Notes |
|-------|---------|-------|
| `extract` | strip each content text line | `extract(boxes) -> list[items]` |
| `key` | identity | `key(item) -> hashable` for de-dup |
| `amount` | `0.6` | Keep `< 1.0` so screens overlap |
| `max_scrolls` | `400` | Hard stop |
| `end_after` | `3` | Consecutive non-moving scrolls ⇒ end |
| `on_progress` | `None` | Optional `on_progress(i, total_items, new_count, moved, overlap)` |

**Returns:**

```python
{
  "items": list,          # ordered unique items
  "stop": "reached-end" | "max-scrolls",
  "scrolls": int,
}
```

---

## Navigation

### `home()`

`press("cmd+1")`, then sleep `0.8s`.

### `app_switcher()`

`press("cmd+2")`, then sleep `0.8s`.

### `open_app(name)`

1. `press("cmd+3")` (Spotlight)
2. sleep `0.9s`
3. `type_text(name)`
4. sleep `1.2s` (results populate)
5. `press("return")`
6. `wait_stable()`

---

## Timing

### `wait(seconds=1.0)`

`time.sleep(seconds)`.

### `wait_stable(timeout=6.0, interval=0.5, settle=2)`

Poll captures; MD5 digest of PNG bytes must match for `settle` consecutive samples.

| Param | Default | Notes |
|-------|---------|-------|
| `timeout` | `6.0` | Max wait seconds |
| `interval` | `0.5` | Sleep between captures |
| `settle` | `2` | Identical digests needed (implementation uses `same >= settle - 1`) |

**Returns:** `True` if stable within timeout, else `False`.

---

## Agent-loaded helpers

Loaded at the end of `helpers` import from:

```text
$PH_AGENT_WORKSPACE/agent_helpers.py
# default: <repo>/agent-workspace/agent_helpers.py
```

### `tap_icon(label, index=0)` (shipped agent helper)

Home Screen app launch by label. Taps **~35 points above** the OCR label center (label text alone is not tappable on the mirrored Home Screen).

| Param | Default |
|-------|---------|
| `label` | required (passed to `find_text`) |
| `index` | `0` |

**Returns:** the matched OCR box (label location, not the adjusted tap point).

**Raises:** `RuntimeError` if no label matches.

In-app labeled controls should use `tap_text`; Home Screen icons should use `tap_icon`.

---

## Module constants (also injected)

| Name | Meaning |
|------|---------|
| `CORE_DIR` | `src/phone_harness` path |
| `REPO_ROOT` | Repository root (`CORE_DIR.parent.parent`) |
| `AGENT_WORKSPACE` | Resolved agent workspace path |
| `mirror` | The selected backend module object |

---

## Errors reference

| Helper / primitive | Exception | When |
|--------------------|-----------|------|
| `ensure_mirroring` | `RuntimeError` | Not ready (not-running / no-window / blocked) |
| `screenshot` / OCR path | `RuntimeError` | Capture failed after retries |
| `ocr` / Vision path | `RuntimeError` | Image unreadable or Vision request failed |
| `tap_text` | `RuntimeError` | No matching text (includes visible sample) |
| `tap_icon` | `RuntimeError` | No Home Screen label match |
| `swipe` / `scroll_screen` | `ValueError` | Bad `direction` |
| `press` | `ValueError` | Unknown key name |
| `type_text` | `ValueError` | Character has no US keycode |
| `activate` (mirror) | `RuntimeError` | App not running |
| Backend `ensure_window` | `RuntimeError` | App not running or no phone window |

Silent input failure is usually permissions/focus (Accessibility, Screen Recording), not an exception — see doctor and troubleshooting docs.

---

## Minimal script surface

```python
print(connection_state())
print(screen_info())
ensure_mirroring()

boxes = ocr()
tap_text("Settings")
wait_stable()

home()
tap_icon("Weather")

open_app("Messages")
# focus a field first, then:
# type_text("hello")

result = scroll_collect(max_scrolls=20)
print(result["stop"], len(result["items"]))
```

Run as:

```bash
phone-harness <<'PY'
print(connection_state())
print(screen_info())
PY
```

---

## Related pages

<CardGroup>
  <Card title="CLI reference" href="/cli-reference">
    How stdin scripts get the helpers namespace and when usage errors exit.
  </Card>
  <Card title="Connection and session states" href="/connection-and-session">
    ready / blocked / no-window / not-running and physical reconnect rules.
  </Card>
  <Card title="Coordinates, capture, and OCR" href="/coordinates-capture-ocr">
    Screen-point space, window vs pixels, and why coordinates must not be cached.
  </Card>
  <Card title="Input backends" href="/input-backends">
    Background SkyLight path vs classic mirror CGEvent path.
  </Card>
  <Card title="Scroll and collect lists" href="/scroll-lists">
    Movement detection, settle windows, and stop reasons in depth.
  </Card>
  <Card title="Navigate apps and type text" href="/navigate-and-type">
    home, Spotlight open_app, type_text keycode limits, tap_icon vs tap_text.
  </Card>
  <Card title="Extend agent helpers" href="/extend-agent-helpers">
    Editing agent_helpers.py and PH_AGENT_WORKSPACE override.
  </Card>
  <Card title="Environment variables" href="/environment-variables">
    PHONE_HARNESS_BACKGROUND, PH_AGENT_WORKSPACE, and capture temp dir.
  </Card>
</CardGroup>