# Input backends

> Default background backend (SkyLight event records, no focus steal) versus classic mirror backend (CGEvent HID + activate); selection via PHONE_HARNESS_BACKGROUND and automatic fallback.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `src/phone_harness/helpers.py`
- `src/phone_harness/background.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/run.py`
- `SKILL.md`

---

---
title: "Input backends"
description: "Default background backend (SkyLight event records, no focus steal) versus classic mirror backend (CGEvent HID + activate); selection via PHONE_HARNESS_BACKGROUND and automatic fallback."
---

`helpers.py` binds every script to one transport module at import time: either `background` (default) or `mirror` (classic). That binding decides how capture and input reach the iPhone Mirroring window (`com.apple.ScreenContinuity`). Higher-level helpers (`ocr`, `tap_text`, `swipe`, `scroll_screen`, `open_app`, …) call the bound module through a shared `mirror` symbol, so the public API stays the same across backends.

## Two backends

| Property | Background (`background.py`) | Classic (`mirror.py`) |
| --- | --- | --- |
| Default | Yes (`PHONE_HARNESS_BACKGROUND` defaults to on) | Only when env forces it, or when background import fails |
| Capture | `CGWindowListCreateImage` by window id | `screencapture -l` / region after activate |
| Mouse input | SkyLight `SLPSPostEventRecordTo` event records | `CGEventPost` at `kCGHIDEventTap` |
| Keyboard | Make-key record + `CGEventPostToPid` | `CGEventPost` HID after activate |
| Focus steal | No (`activate` is a no-op) | Yes (every input path calls `activate`) |
| Scroll lists | Fast vertical flick (wheel events do not land unfocused) | Real scroll-wheel `CGEvent`s |
| Temp capture default | `$TMPDIR/phone-harness/background.png` | `$TMPDIR/phone-harness/window.png` |

Both use **global screen points** for coordinates — the same space as `ocr()` centers and `CGEvent` locations. Helpers re-query window bounds per call; do not cache coordinates across captures.

## Selection and fallback

Selection runs once when `phone_harness.helpers` is imported (including every `phone-harness` stdin script).

```text
PHONE_HARNESS_BACKGROUND
        │
        ▼
  default "1" ──► truthy? ──no──► import .mirror
        │
       yes
        ▼
  try import .background
        │
   success ──► use background
        │
   Exception ──► import .mirror, force _BACKGROUND=False
```

<ParamField body="PHONE_HARNESS_BACKGROUND" type="string" default="1">
Controls transport selection. Default is on. Treated as **off** only when the lowercased value is exactly one of `0`, `false`, or `no`. Any other value (including empty-after-default) keeps background preferred.
</ParamField>

| Value | Effect |
| --- | --- |
| unset / `1` / `true` / other | Prefer `background` |
| `0` / `false` / `no` (any case) | Force `mirror` |
| Background import raises | Silent fallback to `mirror`; process continues |

There is no public API to switch backends mid-process. Change the env var and start a new `phone-harness` invocation.

### Force classic mirror

```bash
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
print(screen_info())  # frontmost may flip True when capture/activate runs
PY
```

Use classic when you want the iPhone Mirroring window raised for live observation, when background taps or capture fail on a given macOS build, or when diagnosing focus-related issues.

## Shared public surface

After selection, these names are re-exported from the active backend onto `helpers` (and therefore into the script namespace):

| Symbol | Role |
| --- | --- |
| `tap(x, y)` | Left click / touch at global points |
| `long_press(x, y, duration=0.8)` | Hold then release |
| `drag(x1, y1, x2, y2, duration=0.35, steps=14)` | Touch-drag / swipe |
| `press(combo)` | Key combo, e.g. `cmd+1`, `return` |
| `type_text(text, delay=0.03)` | US-layout HID keycodes |
| `activate()` | Raise app (classic) or no-op (background) |
| `find_window()` | `{x, y, w, h, id}` or `None` |

Session helpers (`connection_state`, `ensure_mirroring`, `screenshot`, gestures, scroll family) always go through the same bound module for `capture`, `ensure_window`, `running_app`, and `activate`.

`background` reuses `mirror.find_window` and `mirror.running_app` (and keycode tables for keyboard). Window discovery itself never activates.

## Background backend

Module: `src/phone_harness/background.py`.

### Capture (eyes)

- `capture(path=None, retries=2)` → `(path, window_bounds)`.
- Uses `Quartz.CGWindowListCreateImage` with `kCGWindowListOptionIncludingWindow` and the mirroring window id.
- Works when the window is not frontmost and when another app occludes it.
- On failure after retries: `RuntimeError("background capture failed: …")`.

### Mouse (hands)

- Builds a `0xf8`-byte SkyLight event record (yabai-style layout): length, CGSEventType (`1` down, `2` up, `6` dragged), global location, window-local location, window id.
- Resolves process PSN via `GetProcessForPID`, then `_SLPSSetFrontProcessWithOptions` + `SLPSPostEventRecordTo`.
- Frontmost app on the Mac does not change for mouse paths.
- Event types and packing stay inside the fixed buffer; bad values fail as no-ops rather than crashes.

### Scroll in background

Mac scroll-wheel events only reach iPhone Mirroring when the app is active. Background `scroll_wheel(dy, x, y, steps=6)` therefore performs a **fast vertical flick** (drag with short per-step delays) from ~72% → ~28% of window height when `dy < 0` (reveal content below / `scroll_screen` `"up"`), and the reverse when `dy > 0`. Higher-level `scroll()`, `scroll_screen`, `scroll_until`, and `scroll_collect` keep calling `mirror.scroll_wheel`; behavior changes by backend without API changes.

### Keyboard

- `press` / `type_text` make the mirroring window **key** with a blanked-location focus record, then post key events with `CGEventPostToPid` — still without raising the app as frontmost.
- Same US keycode constraints as classic: unknown characters raise `ValueError(f"cannot type {ch!r} via keycodes")`.
- Shared keycode / modifier maps live on `mirror` (`_KEYCODES`, `_MODIFIERS`, `_keycode_for`).

### `activate` and `ensure_window`

| Function | Background behavior |
| --- | --- |
| `activate()` | Immediate `None` (documented no-op) |
| `ensure_window(timeout=5.0)` | No poll loop; raises if app not running or no phone window |

`ensure_mirroring()` still calls `mirror.activate()` on the ready path; under background that call does nothing.

## Classic mirror backend

Module: `src/phone_harness/mirror.py`.

### Why focus is required

The mirroring window is a video stream: Accessibility does not expose in-window UI. Classic capture uses `screencapture`, and input uses HID `CGEventPost`. Both paths target the focused app; unfocused events are swallowed silently.

### Capture

1. Prefer `screencapture -x -o -l <window_id>`.
2. On failure, `activate()`, then region capture `-R x,y,w,h`.
3. After `retries + 1` attempts: `RuntimeError("window capture failed after …")`.

### Input

Every `tap`, `long_press`, `drag`, `scroll_wheel`, `press`, and `type_text` calls `_focus()` → `activate()` first:

- `activate()` uses `NSApplicationActivateIgnoringOtherApps` when the app is running but not frontmost, then sleeps ~0.5s.
- Mouse events: `CGEventCreateMouseEvent` + `CGEventPost(kCGHIDEventTap, …)`.
- Scroll: `CGEventCreateScrollWheelEvent` (pixel unit) at the target location.
- Keys: `CGEventCreateKeyboardEvent` with optional modifier flags.

Unicode string payloads on keyboard events are **not** used: iPhone Mirroring forwards raw HID keycodes only.

### `ensure_window`

If no window is found, classic calls `activate()` and polls until `timeout` (default 5s), then raises if still missing. That can bring the app frontmost even before a successful connection.

## Runtime path

```mermaid
flowchart TB
  subgraph cli ["phone-harness CLI"]
    RUN["run.main: exec stdin"]
  end
  subgraph helpers ["helpers.py"]
    SEL["PHONE_HARNESS_BACKGROUND + import"]
    API["tap / capture / ocr / scroll_* / open_app"]
  end
  subgraph backends ["Transport modules"]
    BG["background.py\nSkyLight + CGWindowListCreateImage"]
    MIR["mirror.py\nscreencapture + CGEvent HID"]
  end
  subgraph os ["macOS / iPhone Mirroring"]
    APP["ScreenContinuity window"]
  end
  RUN --> API
  SEL --> BG
  SEL --> MIR
  API --> SEL
  BG --> APP
  MIR --> APP
```

## Behavioral differences that matter

| Concern | Background | Classic |
| --- | --- | --- |
| User keeps working in another Mac app | Yes for mouse/capture/keyboard paths here | No — window is raised |
| `screen_info()["frontmost"]` during automation | Often `False` | Typically `True` after actions |
| Mid-run focus steal by another click | Mouse still targets process by pid | Events may be swallowed; re-activate |
| List scrolling feel | Momentum flick | Wheel deltas |
| `ensure_window` on missing window | Fail fast | Activate + poll |
| Private API risk | Depends on SkyLight symbols across macOS builds | Public Quartz / `screencapture` only |

<Warning>
If SkyLight symbols fail to load or the private path misbehaves on a given OS build, import falls back to classic automatically. Silent focus-steal after an upgrade is a signal that fallback engaged — force classic explicitly with `PHONE_HARNESS_BACKGROUND=0` to confirm, or inspect whether `background` imports cleanly in a Python REPL.
</Warning>

## Permissions and silent failures

Both backends still need:

- **Screen Recording** — capture path
- **Accessibility** — synthesized input

Classic additionally requires the mirroring window frontmost for reliable HID delivery. Background is designed so mouse and capture work without frontmost, but Accessibility must still be granted to the terminal that runs `phone-harness`.

| Symptom | Likely cause |
| --- | --- |
| Taps do nothing (classic or after fallback) | Accessibility missing, or window not frontmost |
| Capture empty / fails | Screen Recording not effective until terminal restart |
| Background import fails every run | Private SkyLight load error → automatic classic fallback |
| Keyboard types nowhere | iOS text field not focused first (tap field, wait, then `type_text`) |

## Verify which path you are on

There is no exported `is_background()` flag. Practical checks:

```bash
# Prefer background (default)
phone-harness <<'PY'
info = screen_info()
print(info["frontmost"], info["window"])
# With background, capture need not raise iPhone Mirroring
PY

# Force classic and expect activation
PHONE_HARNESS_BACKGROUND=0 phone-harness <<'PY'
print(screen_info()["frontmost"])  # usually True after capture/activate
PY
```

Or probe import directly:

```bash
python3 -c "from phone_harness import helpers; print(helpers.mirror.__name__)"
# phone_harness.background  or  phone_harness.mirror
```

## Related pages

<CardGroup>
  <Card title="Environment variables" href="/environment-variables">
    `PHONE_HARNESS_BACKGROUND` defaults, falsey values, and fallback when SkyLight load fails.
  </Card>
  <Card title="Helpers API" href="/helpers-api">
    Public helper signatures that re-export backend primitives.
  </Card>
  <Card title="Coordinates, capture, and OCR" href="/coordinates-capture-ocr">
    Global screen points shared by both backends and OCR tap centers.
  </Card>
  <Card title="Connection and session" href="/connection-and-session">
    `connection_state` / `ensure_mirroring` gates on the same bound transport.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Silent taps, focus steal, blank capture, and recovery steps.
  </Card>
  <Card title="Doctor diagnostics" href="/doctor-diagnostics">
    Ordered permission and session checks for both capture and input.
  </Card>
</CardGroup>