# Trend monitoring

> SQLite --store persistence, watchlist.py scheduling and delivery, briefing.py digests, and recommended baseline cadence.

- Repository: mvanhorn/last30days-skill
- GitHub: https://github.com/mvanhorn/last30days-skill
- Human docs: https://grok-wiki.com/public/docs/mvanhorn-last30days-skill-50b5421a8cca
- Complete Markdown: https://grok-wiki.com/public/docs/mvanhorn-last30days-skill-50b5421a8cca/llms-full.txt

## Source Files

- `CONFIGURATION.md`
- `skills/last30days/scripts/store.py`
- `skills/last30days/scripts/watchlist.py`
- `skills/last30days/scripts/briefing.py`
- `tests/test_watchlist_commands.py`
- `skills/last30days/scripts/last30days.py`

---

---
title: "Trend monitoring"
description: "SQLite --store persistence, watchlist.py scheduling and delivery, briefing.py digests, and recommended baseline cadence."
---

Trend monitoring turns one-off `/last30days` research into a local time series: ranked findings land in SQLite at `~/.local/share/last30days/research.db`, `watchlist.py` drives recurring engine runs and optional webhooks, and `briefing.py` emits structured digest JSON for agent-side synthesis. Markdown snapshots under `LAST30DAYS_MEMORY_DIR` still save on every engine run; the database is the cross-run substrate for deltas, budgets, and digests.

## Snapshot mode vs accumulation

| Mode | Trigger | Primary artifact | Cross-run memory |
|------|---------|------------------|------------------|
| Snapshot (default) | `/last30days "<topic>"` or engine without persistence | `<slug>-raw.md` (or suffix) under `LAST30DAYS_MEMORY_DIR` | None — same topic + suffix overwrites or date-stamps per save rules |
| Accumulation | `--store`, `LAST30DAYS_STORE=1`, or watchlist `run-one` / `run-all` | `research.db` + optional brief JSON | URL-deduped `findings`, per-run `finding_sightings`, run history |

<Note>
Watchlist runs do **not** pass `--store` to the engine. They spawn `last30days.py` with `--emit=json`, then persist up to 25 findings via `store.findings_from_report()` and `store.store_findings()` inside `watchlist.py`.
</Note>

## Architecture

```mermaid
flowchart TB
  subgraph user["Operator / scheduler"]
    Cron["cron / Task Scheduler / CI"]
    Slash["/last30days or direct CLI"]
  end

  subgraph engine["skills/last30days/scripts/last30days.py"]
    Pipeline["v3 pipeline"]
  end

  subgraph persist["~/.local/share/last30days/"]
    DB["research.db — store.py"]
    Briefs["briefs/*.json — briefing.py"]
  end

  subgraph ops["Operational scripts"]
    WL["watchlist.py"]
    BR["briefing.py"]
  end

  subgraph delivery["Optional notifications"]
    Slack["Slack incoming webhook"]
    Hook["Generic HTTPS webhook"]
  end

  Slash --> Pipeline
  Cron --> WL
  WL -->|"subprocess: --emit=json --quick --lookback-days 90"| Pipeline
  Pipeline -->|"--store or LAST30DAYS_STORE"| DB
  WL --> DB
  WL -->|"new findings only"| Slack
  WL --> Hook
  BR --> DB
  BR --> Briefs
  Pipeline --> Mem["LAST30DAYS_MEMORY_DIR markdown"]
```

## SQLite store (`store.py`)

Default path: `~/.local/share/last30days/research.db` (overridable in tests via `store._db_override`). Initialization uses WAL mode, `PRAGMA foreign_keys=ON`, and lightweight versioned migrations.

### Schema (v1 + migration v2)

```mermaid
erDiagram
    topics ||--o{ research_runs : "topic_id"
    topics ||--o{ findings : "topic_id"
    research_runs ||--o{ findings : "run_id"
    findings ||--o{ finding_sightings : "finding_id"
    research_runs ||--o{ finding_sightings : "run_id"

    topics {
        int id PK
        text name UK
        text search_queries
        text schedule
        int enabled
    }

    research_runs {
        int id PK
        int topic_id FK
        text run_date
        text status
        real token_cost
        int findings_new
        int findings_updated
    }

    findings {
        int id PK
        text source_url UK
        int sighting_count
        real engagement_score
    }

    finding_sightings {
        int id PK
        int run_id FK
        int finding_id FK
    }
```

| Table / object | Role |
|----------------|------|
| `topics` | Watchlist entries; `name` is unique; optional `search_queries` JSON |
| `research_runs` | Per-execution metadata, cost, `findings_new` / `findings_updated` |
| `findings` | One row per `source_url` (UNIQUE); re-sightings bump `sighting_count` and `last_seen` |
| `finding_sightings` | Per-run ledger for `compute_topic_delta()` (new / continued / dropped URLs) |
| `findings_fts` | FTS5 (porter + unicode61) over content, summary, title, author |
| `settings` | Global knobs: budget, delivery, briefing defaults |

Default `settings` rows on first init:

| Key | Default |
|-----|---------|
| `daily_budget` | `5.00` |
| `delivery_channel` | empty |
| `delivery_mode` | `announce` |
| `briefing_format` | `concise` |
| `default_schedule` | `0 8 * * *` |

### Dedup and persistence rules

- Only findings with a `source_url` (or `url` in the input dict) are inserted or updated; URL-less items are skipped silently.
- Re-sighting the same URL updates engagement to `max(new, existing)`, increments `sighting_count`, and refreshes `last_seen`.
- `findings_from_report()` prefers `ranked_candidates`; supplements Hacker News and Polymarket from `items_by_source` when ranked; if ranking is empty, supplements all sources from raw items.

### Store query CLI

Run from the scripts directory or with the full repo path:

```bash
python3 skills/last30days/scripts/store.py stats
python3 skills/last30days/scripts/store.py query "your topic" --since 7d
python3 skills/last30days/scripts/store.py search "keyword" --limit 20
python3 skills/last30days/scripts/store.py trending --days 7
```

## Engine persistence: `--store` and `LAST30DAYS_STORE`

<ParamField body="--store" type="flag">
When set (or when env/config enables storage), after a successful pipeline run the engine calls `persist_report()`: upsert topic, record run, convert report to findings, `store_findings()`, mark run completed. Stderr prints `[last30days] Stored N new, M updated findings`.
</ParamField>

<ParamField body="LAST30DAYS_STORE" type="env">
Read from `os.environ` and loaded config (`.env` is not always mirrored into `os.environ`). Truthy values: `1`, `true`, `yes` (case-insensitive). Additive with `--store` — either enables persistence.
</ParamField>

<CodeGroup>
```bash title="Per-run flag"
python3 skills/last30days/scripts/last30days.py "british airways middle east" \
  --emit=compact --days=30 --store
```

```bash title="Always-on (.env)"
LAST30DAYS_STORE=1
```
</CodeGroup>

Manual `--store` runs still write markdown to `LAST30DAYS_MEMORY_DIR` when the save path is active; SQLite is additional, not a replacement.

## Watchlist (`watchlist.py`)

Manages scheduled topics and headless research loops. The `schedule` field on each topic is **metadata only** — you must invoke `run-one` or `run-all` from an external scheduler (cron, launchd, GitHub Actions, etc.).

### Commands

| Command | Purpose |
|---------|---------|
| `add <topic>` | Register topic; optional `--schedule`, `--weekly`, `--queries "q1,q2"` |
| `remove <topic>` | Delete topic and its findings / runs |
| `list` | JSON list of topics plus `budget_used` / `budget_limit` |
| `delta <topic>` | Compare latest two **completed** runs (new / continued / dropped URLs) |
| `run-one <topic>` | Single headless engine invocation + DB persist |
| `run-all` | All `enabled` topics until daily budget exceeded |
| `config delivery <url>` | Set `delivery_channel` (Slack or HTTPS) |
| `config budget <amount>` | Set `daily_budget` |

Schedule defaults:

| Flag | Stored cron | Description |
|------|-------------|-------------|
| (default) | `0 8 * * *` | Daily 8:00 |
| `--weekly` | `0 8 * * 1` | Mondays 8:00 |
| `--schedule "..."` | Custom | Stored as-is on the topic row |

### What `run-one` / `run-all` execute

Each enabled topic triggers:

```text
python3 last30days.py <search_term> --emit=json --quick --lookback-days 90
```

- `search_term` = first entry in `search_queries` JSON, else topic `name`
- Subprocess timeout: **300 seconds**
- On success: parse JSON → `findings_from_report(..., limit=25)` → `store_findings()`
- Failed runs set `research_runs.status = failed` with truncated stderr

`run-all` skips remaining topics when `store.get_daily_cost()` ≥ `daily_budget` (default **$5.00** per calendar day, UTC date on `run_date`).

### Webhook delivery

Fires only when `delivery_channel` is set **and** `counts["new"] > 0` after a successful run. Delivery failures are logged to stderr and do not fail the research run.

| Channel pattern | Payload |
|-----------------|---------|
| URL contains `hooks.slack.com` | `{"text": "<message>"}` |
| `https://...` otherwise | JSON: `message`, `source: "last30days"`, `timestamp` |

`delivery_mode` (DB default `announce`) controls message shape: `announce` (emoji + counts), `silent` (no emoji), or other (generic “complete” line). The `watchlist config` subcommand sets **delivery URL** and **budget** only — change `delivery_mode` via direct `store.set_setting()` if needed.

<Steps>
<Step title="Register a topic">
```bash
python3 skills/last30days/scripts/watchlist.py add "british airways middle east" --weekly
```
</Step>
<Step title="Configure delivery and budget (optional)">
```bash
python3 skills/last30days/scripts/watchlist.py config delivery "https://hooks.slack.com/services/..."
python3 skills/last30days/scripts/watchlist.py config budget 5.00
```
</Step>
<Step title="Schedule recurring execution">
Point cron (or equivalent) at:
```bash
python3 skills/last30days/scripts/watchlist.py run-all
```
Or one topic: `run-one "british airways middle east"`.
</Step>
<Step title="Inspect run-to-run change">
```bash
python3 skills/last30days/scripts/watchlist.py delta "british airways middle east"
```
Requires at least two completed runs; otherwise status is `insufficient_history`.
</Step>
</Steps>

<Info>
Headless watchlist runs use the engine’s internal planner path (`--emit=json` without `--plan`). Host-model reasoning keys are not required for cron-style execution; configure source API keys and `.env` the same as interactive runs.
</Info>

## Briefings (`briefing.py`)

Collects structured JSON for synthesis — the script does not render final prose; the skill/agent turns the payload into a readable briefing.

| Command | Output |
|---------|--------|
| `generate` | Daily digest data; default window: findings with `first_seen` since yesterday |
| `generate --weekly` | Week-over-week counts and engagement % change per topic |
| `generate --since YYYY-MM-DD` | Daily window override |
| `show [--date DATE]` | Load saved JSON from archive |

Archive directory: `~/.local/share/last30days/briefs/` as `{YYYY-MM-DD}.json` or `{YYYY-MM-DD}-weekly.json`.

Daily payload highlights:

- Per-topic `findings`, `new_count`, `last_run`, `last_status`
- `stale: true` when last completed run is **> 36 hours** ago
- `top_finding` per topic and cross-topic `top_finding` for TL;DR
- `cost.daily` vs `cost.budget`
- `failed_topics` list

Weekly payload adds `this_week_count`, `last_week_count`, `engagement_change_pct`, and top five findings per topic.

```bash
python3 skills/last30days/scripts/briefing.py generate
python3 skills/last30days/scripts/briefing.py generate --weekly
python3 skills/last30days/scripts/briefing.py show --date 2026-06-01
```

## Recommended baseline cadence

| Step | Cadence | Action |
|------|---------|--------|
| Baseline corpus | One-time per topic | `/last30days "<topic>"` with `--days=30` and `--store` (or `LAST30DAYS_STORE=1`) |
| Watchlist enrollment | One-time per topic | `watchlist.py add "<topic>"` (`--weekly` if Monday cadence fits) |
| Recurring research | Daily or weekly (your scheduler) | `watchlist.py run-all` or per-topic `run-one` |
| Digest | Weekly (typical) | `briefing.py generate --weekly`; use daily `generate` for operational triage |

<Warning>
The `schedule` column does not trigger runs by itself. Missing cron means topics go stale (`briefing.py` flags > 36h).
</Warning>

## Operational checks

| Symptom | Check |
|---------|-------|
| Empty brief | No enabled topics — `watchlist.py list`; add with `add` |
| Delta unavailable | Fewer than two completed runs for topic |
| Webhook never fires | `delivery_channel` unset, or zero **new** findings (updates alone do not notify) |
| `run-all` skips topics | Daily budget exhausted — `list` shows `budget_used` vs `budget_limit` |
| Thin watchlist results | Engine uses `--quick` and 90-day lookback; not full default-depth profile |

## Related pages

<CardGroup>
<Card title="Configuration reference" href="/configuration-reference">
Environment variables, `.env` lookup, and trend-monitoring toggles including `LAST30DAYS_STORE`.
</Card>
<Card title="CLI reference" href="/cli-reference">
`last30days.py` flags (`--store`, `--days`, `--emit`) and direct CLI vs slash-command constraints.
</Card>
<Card title="Research pipeline" href="/research-pipeline">
What `--quick` changes inside the v3 pipeline that watchlist runs hardcode.
</Card>
<Card title="Output contract" href="/output-contract">
Markdown save paths under `LAST30DAYS_MEMORY_DIR` alongside SQLite accumulation.
</Card>
<Card title="Quickstart" href="/quickstart">
First successful `/last30days` run before enabling `--store` or watchlist.
</Card>
</CardGroup>