# Navigate apps and type text

> home, app_switcher, open_app via Spotlight, press key combos, type_text US keycode constraints, and Home Screen tap_icon versus in-app tap_text.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `agent-workspace/agent_helpers.py`
- `SKILL.md`

---

---
title: "Navigate apps and type text"
description: "home, app_switcher, open_app via Spotlight, press key combos, type_text US keycode constraints, and Home Screen tap_icon versus in-app tap_text."
---

Navigation and typing in phone-harness are thin helpers over the active input backend (`src/phone_harness/background.py` by default, or `src/phone_harness/mirror.py` when background load fails or `PHONE_HARNESS_BACKGROUND` is falsey). Core shortcuts live in `src/phone_harness/helpers.py`; `press` and `type_text` are re-exported from the backend; Home Screen icon launching is the agent helper `tap_icon` in `agent-workspace/agent_helpers.py`.

## Prerequisites

- Session is `ready` (`ensure_mirroring()` / `connection_state()`).
- iPhone Mirroring shortcuts map as the Mac app defines them: **Cmd+1** Home, **Cmd+2** App Switcher, **Cmd+3** Spotlight.
- For typing: an iOS text field (or Spotlight search field) must already be focused; `type_text` only posts keycodes.

## Navigation helpers

| Helper | Action | Implementation |
| --- | --- | --- |
| `home()` | Home Screen | `press("cmd+1")`, then `sleep(0.8)` |
| `app_switcher()` | Multitasking switcher | `press("cmd+2")`, then `sleep(0.8)` |
| `open_app(name)` | Open app by name via Spotlight | `cmd+3` → type name → wait → `return` → `wait_stable()` |

These helpers do not verify the resulting UI. After navigation, call `wait_stable()` (already used by `open_app`) and re-read with `ocr()` or `screenshot()`.

### `home()`

```python
home()  # press("cmd+1"); sleep(0.8)
```

Use before Home Screen work (icon launch, page swipe). Does not confirm Home is visible.

### `app_switcher()`

```python
app_switcher()  # press("cmd+2"); sleep(0.8)
```

Opens the app switcher. Selecting an app still requires a separate `tap` / `tap_text` on the card.

### `open_app(name)`

Spotlight path for launching by name:

```text
press("cmd+3")
sleep(0.9)
type_text(name)
sleep(1.2)      # let Spotlight results populate
press("return")
wait_stable()
```

| Parameter | Type | Notes |
| --- | --- | --- |
| `name` | string | Typed with `type_text` (US keycode set only) |

```bash
phone-harness <<'PY'
ensure_mirroring()
open_app("Notes")
print([o["text"] for o in ocr()][:15])
PY
```

<Note>
`open_app` always commits the first Spotlight hit with Return. Use a distinctive `name` (for example `"Settings"` not a single letter). Characters outside the US keycode map raise before Return is pressed.
</Note>

Prefer `open_app` when the app is not on the current Home Screen page. Prefer `home()` + `tap_icon(label)` when the icon is visible and you want a direct Home Screen launch.

## `press(combo)`

Posts a single key or modifier chord through the active backend.

| Backend | Behavior |
| --- | --- |
| Classic (`mirror.py`) | `activate()` then `CGEventCreateKeyboardEvent` + `CGEventPost` at HID |
| Background (`background.py`) | Make window key via SkyLight event record, then `CGEventPostToPid` (no frontmost steal for the keystroke path) |

### Signature and grammar

```python
press(combo)  # e.g. press("return"), press("cmd+1"), press("cmd+3")
```

- Split on `+` (lowercased): last segment is the key; earlier segments are modifiers.
- Unknown key → `ValueError(f"unknown key {key!r}")`.
- Unknown modifier → `KeyError` (modifier not in `_MODIFIERS`).

### Known keys (`_KEYCODES`)

| Category | Keys |
| --- | --- |
| Control | `return` / `enter`, `tab`, `space`, `delete` / `backspace`, `escape` / `esc` |
| Arrows | `left`, `right`, `down`, `up` |
| Digits | `0`–`9` |
| Letters | `a`–`z` (lowercase key names) |

### Known modifiers (`_MODIFIERS`)

| Name | Flag |
| --- | --- |
| `cmd` | Command |
| `shift` | Shift |
| `alt` / `option` | Alternate |
| `ctrl` | Control |

Navigation chords used by core helpers:

| Combo | Meaning |
| --- | --- |
| `cmd+1` | Home Screen |
| `cmd+2` | App Switcher |
| `cmd+3` | Spotlight |

```python
press("return")
press("cmd+1")
press("escape")
```

## `type_text(text, delay=0.03)`

Types into the **already focused** iOS field via real HID keycodes (US layout). iPhone Mirroring forwards raw keycodes and ignores unicode payloads on keyboard events, so emoji and many non-US characters cannot be typed.

### Behavior

1. Split `text` on `\n`. Between lines, press Return.
2. For each character, resolve `(keycode, needs_shift)` via `_keycode_for`.
3. Missing keycode → `ValueError(f"cannot type {ch!r} via keycodes")`.
4. Per-character delay defaults to `0.03` seconds after each key up/down pair.

| Parameter | Default | Role |
| --- | --- | --- |
| `text` | required | String to type; `\n` becomes Return |
| `delay` | `0.03` | Pause after each character |

### Typable set (US)

| Class | Supported |
| --- | --- |
| Letters | `a`–`z`, `A`–`Z` (shift) |
| Digits | `0`–`9` |
| Shifted digit symbols | `! @ # $ % ^ & * ( )` |
| Punctuation | `. , / ; ' [ ] \ - = \`` and shifted forms `_ + : " < > ? ~ { } \|` |
| Space | `" "` |
| Newline | `\n` → Return |

Not supported: emoji, non-Latin scripts, combining marks, and any glyph without a US keycode entry.

```python
# Field must already be focused
tap_text("Title")
wait_stable()
type_text("hello from the harness")
type_text("line one\nline two")  # Return between lines
```

<Warning>
`type_text` does not open a field or wait for the software keyboard. Tap the field first, wait until the UI is stable, then type. Typing with no focused field is a silent no-op or lands in the wrong surface.
</Warning>

## Home Screen `tap_icon` versus in-app `tap_text`

| Surface | Helper | Target | Notes |
| --- | --- | --- | --- |
| Home Screen app icon | `tap_icon(label, index=0)` | Label OCR center, then **y − 35** | Label text is not the launch hit target |
| In-app button / row / control | `tap_text(query, index=0, exact=False)` | OCR box center | Label and control share the same center |

### Why labels fail on the Home Screen

OCR returns the caption under the icon. Tapping that center hits the label, not the icon; launch is a no-op. The agent helper offsets upward:

```python
# agent-workspace/agent_helpers.py
def tap_icon(label, index=0):
    hits = find_text(label)
    if not hits:
        raise RuntimeError(f"no Home-Screen label matching {label!r}")
    h = hits[index]
    tap(h["x"], h["y"] - 35)
    return h
```

| Parameter | Default | Role |
| --- | --- | --- |
| `label` | required | Case-insensitive substring match via `find_text` |
| `index` | `0` | Which match when several labels match |

Raises `RuntimeError` if no OCR hit matches.

`tap_icon` is loaded from `agent-workspace/agent_helpers.py` into the script namespace at helper import time (override workspace with `PH_AGENT_WORKSPACE`). Edit that file for task-specific variants; see [Extend agent helpers](/extend-agent-helpers).

### In-app: use `tap_text`

```python
tap_text("New Note")           # substring match
tap_text("Done", exact=True)   # exact string
```

On miss, `tap_text` raises with up to 30 currently visible OCR strings so the next step can replan.

## Choose a launch path

```text
Need an app open?
├── Name known, icon may be off-screen  →  open_app("AppName")
├── Icon visible on Home Screen         →  home(); tap_icon("AppName")
└── Already inside an app UI            →  tap_text(...) / tap(x, y)
```

```bash
phone-harness <<'PY'
ensure_mirroring()
home()
tap_icon("Weather")
wait_stable()
print([o["text"] for o in ocr()][:12])
PY
```

```bash
phone-harness <<'PY'
ensure_mirroring()
open_app("Notes")
tap_text("New Note")
type_text("hello from the harness")
wait_stable()
print([o["text"] for o in ocr()][:10])
PY
```

## Verify after navigation and typing

There is no accessibility tree inside the mirroring window. After `home`, `open_app`, `tap_icon`, or `type_text`:

1. `wait_stable()` (or rely on the settle already in `open_app`)
2. `ocr()` or `screenshot()` for ground truth
3. Retry or replan from what is actually visible

## Failure modes

| Symptom | Likely cause | Recovery |
| --- | --- | --- |
| `ValueError: cannot type '…' via keycodes` | Character outside US keycode map | Use only US typable characters; avoid emoji |
| `ValueError: unknown key '…'` | Bad `press` key name | Use keys from `_KEYCODES` |
| Home Screen tap does nothing | Used `tap_text` on icon label | Use `tap_icon` (~35 pt above label) |
| `RuntimeError: no Home-Screen label matching` | OCR miss / wrong page | `home()`, swipe pages, `screenshot()`, retry label |
| Typed text missing | Field not focused | Tap field, `wait_stable()`, then `type_text` |
| Spotlight opens wrong app | Ambiguous name / first result | More specific `name` |
| Silent no-op on classic backend | Window lost frontmost focus | Re-`activate()` / re-`ensure_mirroring()` |

## Related pages

<CardGroup>
  <Card title="See, act, and verify" href="/see-act-verify">
    OCR-first read loop, `tap` / `tap_text`, `wait_stable`, and capture-as-ground-truth after every action.
  </Card>
  <Card title="Helpers API" href="/helpers-api">
    Full signatures for navigation, timing, gestures, and re-exported backend primitives.
  </Card>
  <Card title="Extend agent helpers" href="/extend-agent-helpers">
    Edit `agent_helpers.py`, `PH_AGENT_WORKSPACE`, and the `tap_icon` Home Screen pattern.
  </Card>
  <Card title="Input backends" href="/input-backends">
    Background SkyLight path versus classic mirror HID + activate, and env selection.
  </Card>
  <Card title="Consent and limits" href="/consent-and-limits">
    Hard limits including keycode-only typing and when not to drive the phone.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Home Screen label misses, `type_text` field focus, blocked session, and silent input.
  </Card>
</CardGroup>