# Scroll and collect lists

> scroll_screen movement detection, scroll_until predicates, scroll_collect de-dup extraction, wheel versus flick behavior, settle windows, and stop reasons reached-end and max-scrolls.

- Repository: ShawnPana/phone-harness
- GitHub: https://github.com/ShawnPana/phone-harness
- Human docs: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e
- Complete Markdown: https://grok-wiki.com/public/docs/shawnpana-phone-harness-bf80173a2a2e/llms-full.txt

## Source Files

- `src/phone_harness/helpers.py`
- `src/phone_harness/mirror.py`
- `src/phone_harness/background.py`
- `src/phone_harness/ocr.py`
- `SKILL.md`

---

---
title: "Scroll and collect lists"
description: "scroll_screen movement detection, scroll_until predicates, scroll_collect de-dup extraction, wheel versus flick behavior, settle windows, and stop reasons reached-end and max-scrolls."
---

`scroll_screen`, `scroll_until`, and `scroll_collect` in `src/phone_harness/helpers.py` walk iOS lists through the iPhone Mirroring window by scrolling, waiting for a settle window, then judging end-of-list from **OCR text-set overlap** (whether the screen moved), not from whether your extractor found new rows. They call `mirror.scroll_wheel` on the active input backend (`background` by default, `mirror` when background is disabled or fails to load).

## Design rule: movement, not parser output

End-of-list is decided only by whether consecutive content OCR text sets still look like the same screen after a settle window. A dense list, a missed OCR line, or a slow lazy-load must not stop the walk early.

| Signal | Role |
|--------|------|
| Content OCR text set | Before/after comparison input (status bar and bottom chrome cropped out) |
| Jaccard overlap | `len(a ∩ b) / len(a ∪ b)` — ~1.0 same screen, low means it moved |
| `moved` | `overlap < moved_thresh` (default `0.6`) |
| Extractor / `done` | Collects or stops on a match; never ends the walk alone |

Empirical band used by the default threshold: real forward progress often lands under ~0.45 overlap; overscroll bounce at a boundary often sits above ~0.7 and would otherwise look like movement.

## Choose the right gesture

| Helper | Mechanism | Use for |
|--------|-----------|---------|
| `scroll_screen` / `scroll_until` / `scroll_collect` | `scroll_wheel` via the active backend | Long lists, feeds, Settings rows |
| `scroll(amount=300)` | One `scroll_wheel` at window center | Simple one-shot nudge |
| `swipe(direction, distance=0.4)` | Fast short `drag` (`duration=0.12`, `steps=6`) | Home Screen pages, carousels, momentum snaps |

A **slow** touch-drag barely moves an iOS list and often bounces back. List helpers intentionally avoid that path.

### Wheel (classic backend) versus flick (background backend)

Both backends expose the same `scroll_wheel(dy, x, y, steps=...)` name, but they do different work:

| Backend | `PHONE_HARNESS_BACKGROUND` | What `scroll_wheel` actually does |
|---------|----------------------------|-----------------------------------|
| `mirror.py` | `0` / `false` / `no`, or background import failure | Real `CGEventCreateScrollWheelEvent` pixel-wheel events after focusing the window |
| `background.py` | default `1` | **Fast vertical flick** (left-mouse drag), not a Mac wheel event |

Background reason (verified in code comments): Mac scroll-wheel events only reach iPhone Mirroring when the app is active, so background wheel posts produce ~0% movement. A slow drag also barely advances a list. A **fast** flick supplies release velocity and advances content (~29% frame change per flick in the author’s check).

Sign convention (shared by list helpers):

- `direction="up"` → negative `dy` → reveal content **below** (finger flicks up / content moves up)
- `direction="down"` → positive `dy` → reveal content **above**

Background flick geometry (fraction of window height):

- `dy < 0`: drag from `0.72h` → `0.28h`
- `dy ≥ 0`: drag from `0.28h` → `0.72h`
- Inter-step sleep: `0.006s` (velocity is the load-bearing part)

List helpers always call `scroll_wheel(..., steps=10)` with `dy = sign * int(window_height * amount)`.

## Content OCR crop

Before movement checks and collection, helpers OCR only the scrollable middle of the window via `_content_texts`:

| Parameter | Default | Purpose |
|-----------|---------|---------|
| `min_conf` | `0.4` | Drop low-confidence Vision boxes |
| `top_frac` | `0.06` | Exclude status bar (clock / battery) |
| `bottom_frac` | `0.92` | Exclude home indicator / bottom nav strip |

Boxes must satisfy `top < y < bot` in **global screen points**. Status-bar text changing every minute must not flip “settled” or “moved.”

## API surface

### `scroll_screen` — one step + settle + movement verdict

```python
scroll_screen(direction="up", amount=0.6, settle=2.5, moved_thresh=0.6)
```

<ParamField body="direction" type="str" default="up">
`"up"` or `"down"`. Any other value raises `ValueError`.
</ParamField>

<ParamField body="amount" type="float" default="0.6">
Fraction of window height passed into `scroll_wheel` as pixel magnitude. Prefer values `< 1.0` so consecutive screens overlap and rows are not skipped between captures.
</ParamField>

<ParamField body="settle" type="float" default="2.5">
Seconds allowed for the post-scroll settle loop (lazy-load spinner / new rows).
</ParamField>

<ParamField body="moved_thresh" type="float" default="0.6">
Jaccard threshold: `moved` is `True` when `overlap < moved_thresh`.
</ParamField>

**Sequence**

1. Snapshot `before` = content text set.
2. `scroll_wheel` at window center with `steps=10`.
3. Sleep `0.4s`.
4. Poll content OCR every `0.35s` until two consecutive identical text sets **or** the settle deadline.
5. Compare `before` vs settled `after`.

**Return shape**

| Field | Type | Meaning |
|-------|------|---------|
| `moved` | `bool` | Screen advanced past `moved_thresh` |
| `overlap` | `float` | Jaccard overlap, rounded to 3 decimals |
| `before` | `frozenset[str]` | Pre-scroll content texts |
| `after` | `frozenset[str]` | Settled post-scroll content texts |
| `boxes` | `list[dict]` | Settled content OCR boxes (ready for extract / `done`) |

```python
res = scroll_screen("up", amount=0.6)
print(res["moved"], res["overlap"], len(res["boxes"]))
```

### `scroll_until` — predicate stop or confirmed end

```python
scroll_until(done, direction="up", amount=0.6, max_scrolls=60, settle=2.5)
```

<ParamField body="done" type="callable" required>
`done(boxes) -> truthy | falsy`. Receives current content OCR boxes. Truthy return value is returned to the caller.
</ParamField>

<ParamField body="max_scrolls" type="int" default="60">
Hard cap on scroll steps after the initial pre-check.
</ParamField>

**Stop outcomes**

| Outcome | Return value |
|---------|----------------|
| `done(boxes)` truthy on current screen (including before any scroll) | That truthy value |
| Two consecutive non-moving scrolls (`stale >= 2`) after settle | `None` |
| `max_scrolls` exhausted without hit or confirmed end | `None` |

On a non-moving step that has not yet hit `stale >= 2`, the helper sleeps `0.8s`, calls `mirror.activate()`, and retries once so a stalled focus/lazy-load does not false-end the walk.

```python
hit = scroll_until(
    lambda boxes: next((b for b in boxes if "Wi-Fi" in b["text"]), None)
)
if hit:
    tap(hit["x"], hit["y"])
```

### `scroll_collect` — de-duped extraction to true end

```python
scroll_collect(
    extract=None,
    key=None,
    direction="up",
    amount=0.6,
    max_scrolls=400,
    end_after=3,
    settle=2.5,
    on_progress=None,
)
```

| Parameter | Default | Behavior |
|-----------|---------|----------|
| `extract` | strip non-empty `text` from each box | `extract(boxes) -> list[item]` for one screen |
| `key` | identity | Hashable de-dup key; first occurrence wins, order preserved |
| `direction` | `"up"` | Same sign convention as `scroll_screen` |
| `amount` | `0.6` | Keep `< 1.0` so screens overlap |
| `max_scrolls` | `400` | Hard cap |
| `end_after` | `3` | Consecutive non-moving scrolls required for `reached-end` |
| `settle` | `2.5` | Passed to each `scroll_screen` |
| `on_progress` | `None` | Optional `on_progress(i, total_items, new_count, moved, overlap)` |

**Algorithm**

1. Ingest the current screen (no scroll yet).
2. For `i` in `1..max_scrolls`: `scroll_screen` → ingest settled boxes → optional progress callback.
3. If `moved`: reset stale counter to `0`.
4. If not `moved`: increment stale; when `stale >= end_after`, stop with `reached-end` (after the same `0.8s` + `activate()` grace used on intermediate stalls).
5. If the loop finishes, stop with `max-scrolls`.

**Return shape**

```python
{"items": [...], "stop": "reached-end" | "max-scrolls", "scrolls": int}
```

| Field | Meaning |
|-------|---------|
| `items` | De-duped items in first-seen order |
| `stop` | Why the walk ended |
| `scrolls` | Number of scroll steps performed (`i` on early exit, or `max_scrolls`) |

## Stop reasons

```text
┌────────────────────┐
│  ingest screen 0   │
└─────────┬──────────┘
          ▼
┌────────────────────┐     moved=True      ┌──────────────┐
│ scroll_screen step │ ──────────────────► │ stale = 0    │──► next step
└─────────┬──────────┘                     └──────────────┘
          │ moved=False
          ▼
   stale += 1; sleep 0.8; activate()
          │
          ├── stale >= end_after (collect: 3) or >= 2 (until)
          │         → stop "reached-end" / return None
          │
          └── i hits max_scrolls
                    → stop "max-scrolls" / return None
```

| Reason | API | Meaning |
|--------|-----|---------|
| `reached-end` | `scroll_collect` | `end_after` consecutive non-moving scrolls after settle |
| `max-scrolls` | `scroll_collect` | Hit `max_scrolls` without that consecutive still streak |
| predicate hit | `scroll_until` | Returns `done(...)` truthy value |
| confirmed still / cap | `scroll_until` | Returns `None` (`stale >= 2` or `max_scrolls`) |

**Important:** zero *new extracted items* on a step is normal (overlap de-dup). That does **not** end the walk. Only `moved == False` for enough consecutive steps does.

## Examples

### Collect every visible text line

```bash
phone-harness <<'PY'
result = scroll_collect()
print(result["stop"], result["scrolls"], len(result["items"]))
print(result["items"][:20])
PY
```

### Structured rows with a stable key

```bash
phone-harness <<'PY'
def extract(boxes):
    # Example: keep lines that look like settings rows
    return [b["text"].strip() for b in boxes if b["text"].strip()]

result = scroll_collect(
    extract=extract,
    key=lambda s: s.lower(),
    amount=0.55,
    end_after=3,
    on_progress=lambda i, n, new, moved, ov:
        print(f"step={i} total={n} new={new} moved={moved} overlap={ov}"),
)
assert result["stop"] in ("reached-end", "max-scrolls")
print(result["stop"], len(result["items"]))
PY
```

### Stop when a label appears, then tap it

```bash
phone-harness <<'PY'
def done(boxes):
    for b in boxes:
        if "Airplane Mode" in b["text"]:
            return b
    return None

row = scroll_until(done, direction="up", amount=0.6, max_scrolls=40)
if row is None:
    raise SystemExit("label not found before end of list")
tap(row["x"], row["y"])
wait_stable()
print([o["text"] for o in ocr()][:15])
PY
```

### Single-step inspection

```bash
phone-harness <<'PY'
r = scroll_screen("up", amount=0.6, settle=2.5)
print({"moved": r["moved"], "overlap": r["overlap"], "n_boxes": len(r["boxes"])})
PY
```

## Defaults cheat sheet

| Constant / default | Value | Where |
|--------------------|-------|--------|
| Content OCR confidence | `0.4` | `_content_texts` |
| Top crop | `6%` of height | `_content_texts` |
| Bottom crop | below `92%` of height | `_content_texts` |
| Scroll amount | `0.6` window heights | list helpers |
| Settle window | `2.5s` | list helpers |
| Moved threshold | `0.6` Jaccard | `scroll_screen` |
| Post-scroll sleep | `0.4s` | `scroll_screen` |
| Settle poll interval | `0.35s` | `scroll_screen` |
| Non-move grace | `0.8s` + `activate()` | `scroll_until` / `scroll_collect` |
| End confirmation (`until`) | `2` consecutive stills | `scroll_until` |
| End confirmation (`collect`) | `3` consecutive stills (`end_after`) | `scroll_collect` |
| `scroll_until` cap | `60` | `max_scrolls` |
| `scroll_collect` cap | `400` | `max_scrolls` |
| Wheel/flick steps from lists | `10` | `scroll_screen` → `scroll_wheel` |

## Failure modes and recovery

| Symptom | Likely cause | What to do |
|---------|--------------|------------|
| `stop="max-scrolls"` with still-growing UI | List never went fully still (ads, live clocks inside content crop, infinite feed) | Raise `max_scrolls`, tighten `extract`/`key`, or use `scroll_until` with an explicit stop label |
| Early `reached-end` / `None` | Bounce misread as still, or settle too short for lazy-load | Increase `settle`; keep default `moved_thresh=0.6`; ensure session is `ready` |
| No rows advance, overlap stays high | Wrong surface (non-scrollable screen), blocked interstitial, or input not landing | `connection_state()` / `ensure_mirroring()`; check Accessibility; see input backends |
| Missing rows between screens | `amount` too large (no overlap) | Use `amount` in the ~0.5–0.7 range |
| Collect empty / sparse | Content is icon-only or OCR confidence below `0.4` | Custom `extract` after inspecting `scroll_screen()["boxes"]`; fall back to `screenshot()` for unlabeled chrome |
| `swipe` used on a long list | Slow/short drag path is for pages, not lists | Use `scroll_collect` / `scroll_screen` |
| Background session “scrolls” poorly | Expecting Mac wheel semantics | Background path is a **flick**; classic path is real wheel — both share the helper API |

<Warning>
Never end a list walk because `new == 0` on one screen. De-dup and dense OCR make empty ingest normal. Trust `moved` / stop reasons only.
</Warning>

## Backend selection (affects transport only)

List helper signatures and return shapes are backend-agnostic. Transport selection:

- Default: `PHONE_HARNESS_BACKGROUND=1` → `background.scroll_wheel` (fast flick)
- Force classic: `PHONE_HARNESS_BACKGROUND=0|false|no` → `mirror.scroll_wheel` (HID wheel + focus)
- Import failure of the background module falls back to classic automatically

Keyboard helpers may still activate the window on the background backend; mouse scroll/flick does not require focus steal on that path.

## Related pages

<CardGroup>
  <Card title="See, act, and verify" href="/see-act-verify">
    OCR-first reading, wait_stable, and capture-as-ground-truth after each action.
  </Card>
  <Card title="Input backends" href="/input-backends">
    Background SkyLight events versus classic HID wheel, and PHONE_HARNESS_BACKGROUND.
  </Card>
  <Card title="Coordinates, capture, and OCR" href="/coordinates-capture-ocr">
    Screen-point boxes, confidence, and why content crop uses global coordinates.
  </Card>
  <Card title="Helpers API" href="/helpers-api">
    Full helper signatures, defaults, and return shapes including the scroll family.
  </Card>
  <Card title="Connection and session states" href="/connection-and-session">
    ready / blocked gates before any scroll walk.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    Silent input, blocked mirroring, and OCR miss recovery.
  </Card>
</CardGroup>