# Data sources

> Per-platform retrieval modules, keyless vs keyed sources, --search alias map, and engagement scoring inputs by source.

- Repository: mvanhorn/last30days-skill
- GitHub: https://github.com/mvanhorn/last30days-skill
- Human docs: https://grok-wiki.com/public/docs/mvanhorn-last30days-skill-50b5421a8cca
- Complete Markdown: https://grok-wiki.com/public/docs/mvanhorn-last30days-skill-50b5421a8cca/llms-full.txt

## Source Files

- `skills/last30days/scripts/lib/pipeline.py`
- `CONFIGURATION.md`
- `skills/last30days/scripts/lib/reddit_public.py`
- `skills/last30days/scripts/lib/bird_x.py`
- `skills/last30days/scripts/lib/github.py`
- `skills/last30days/scripts/lib/polymarket.py`
- `skills/last30days/scripts/lib/youtube_yt.py`

---

---
title: "Data sources"
description: "Per-platform retrieval modules, keyless vs keyed sources, --search alias map, and engagement scoring inputs by source."
---

Each research run fans out through `skills/last30days/scripts/lib/pipeline.py`, which resolves which platforms are callable (`available_sources`), normalizes CLI `--search` tokens, dispatches per-source retrieval in `_retrieve_stream`, and passes normalized `SourceItem` records through `signals.annotate_stream` for engagement-aware local ranking before fusion and LLM rerank.

## Source inventory

Canonical source names match `pipeline.MOCK_AVAILABLE_SOURCES` and `last30days.parse_search_flag` validation. Retrieval modules live under `skills/last30days/scripts/lib/`; `_retrieve_stream` is the single dispatch table.

| Source | Module | Auth / prerequisites | Default availability |
|--------|--------|----------------------|----------------------|
| `reddit` | `reddit_public.py` → `reddit_keyless.py`; backup `reddit.py` (ScrapeCreators) | Keyless tiers first; `SCRAPECREATORS_API_KEY` for paid backup | Always |
| `hackernews` | `hackernews.py` | None (Algolia) | Always |
| `polymarket` | `polymarket.py` | None (Gamma API) | Always |
| `github` | `github.py` | `GITHUB_TOKEN` or `gh` CLI | When token/CLI present |
| `youtube` | `youtube_yt.py` | `yt-dlp` on PATH; optional `SCRAPECREATORS_API_KEY` fallback + comment enrichment | When `yt-dlp` or SC YouTube path available |
| `grounding` | `grounding.py` | One of `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, `PARALLEL_API_KEY` (or forced via `--web-backend`) | When a web-search key is set |
| `x` | `bird_x.py`, `xai_x.py`, or `xurl_x.py` | Bird: `AUTH_TOKEN`+`CT0`+Node; xAI: `XAI_API_KEY`; xurl: authenticated CLI | When `env.get_x_source()` resolves a backend |
| `tiktok` | `tiktok.py` | `SCRAPECREATORS_API_KEY` or `APIFY_API_TOKEN` | When SC/Apify key set (see opt-out below) |
| `instagram` | `instagram.py` | `SCRAPECREATORS_API_KEY` | When SC key set |
| `threads` | `threads.py` | `SCRAPECREATORS_API_KEY` | When SC key set |
| `bluesky` | `bluesky.py` | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | When credentials set |
| `truthsocial` | `truthsocial.py` | `TRUTHSOCIAL_TOKEN` | When token set |
| `digg` | `digg.py` | `digg-pp-cli` on PATH | When CLI installed |
| `perplexity` | `perplexity.py` | `OPENROUTER_API_KEY` + `INCLUDE_SOURCES` contains `perplexity` (or explicit `--search=perplexity`) | Opt-in |
| `pinterest` | `pinterest.py` | `SCRAPECREATORS_API_KEY` | Only when `--search` (or planner) explicitly requests `pinterest` |
| `xiaohongshu` | `xiaohongshu_api.py` | Reachable `XIAOHONGSHU_API_BASE` with logged-in session | Only when explicitly requested and health probe passes |
| `xquik` | `xquik.py` | `XQUIK_API_KEY` | When key set |

<Note>
`available_sources()` is the runtime gate. User-facing docs sometimes mention `INCLUDE_SOURCES=tiktok`; in code, TikTok and Instagram are added whenever `SCRAPECREATORS_API_KEY` is set. Use `EXCLUDE_SOURCES=tiktok,instagram` to deny them without removing the key.
</Note>

## Keyless vs keyed retrieval

**Keyless (no API key in config):** Reddit (public/keyless pipeline), Hacker News, Polymarket, and GitHub when only the unauthenticated search API is used (higher rate limits with `GITHUB_TOKEN` or `gh`).

**Keyed or tool-backed:** Everything else depends on env vars, CLI tools, or session cookies as in the table above.

### Reddit tiering

Reddit is the most layered source:

1. **Tier 0** — legacy `reddit.com/search.json` via `reddit_public` (often 403; cheap one-shot).
2. **Tier 1** — RSS discovery (`reddit_rss.py`) inside `reddit_keyless.py`.
3. **Tier 2** — Shreddit comment enrichment (`reddit_shreddit.py`).
4. **Backup** — `reddit.search_and_enrich` via ScrapeCreators when keyless tiers return empty and `SCRAPECREATORS_API_KEY` is set.

Pipeline passes `raw_topic` (not only the planner sub-query) into Reddit/YouTube/TikTok/Instagram so query expansion stays anchored to the user topic.

### X backend selection

`env.get_x_source()` picks the active X path: `LAST30DAYS_X_BACKEND` pin, else `xai` if `XAI_API_KEY`, else Bird when explicit `AUTH_TOKEN`/`CT0` and vendored `bird-search.mjs`+Node, else `xurl`. Browser cookie probing is disabled in normal runs (`BIRD_DISABLE_BROWSER_COOKIES=1`).

### YouTube dual path

`_retrieve_stream` tries `youtube_yt.search_and_transcribe` (yt-dlp) first, then `search_youtube_sc` when SC is configured. Optional `youtube_comments` in `INCLUDE_SOURCES` triggers `enrich_with_comments` on top-N videos by engagement.

### Web grounding (`grounding`)

Registered as source `grounding`. CLI alias `web` maps to it. Backends: Brave, Exa, Serper, Parallel — selected from keys or `--web-backend`. Planner may append `grounding` to every sub-query when available; `--web-backend none` removes it.

## `--search` alias map

The `--search` flag accepts a comma-separated list. Tokens are lowercased, aliased, deduped, and validated against `MOCK_AVAILABLE_SOURCES`.

| CLI token | Canonical source |
|-----------|------------------|
| `hn` | `hackernews` |
| `bsky` | `bluesky` |
| `truth` | `truthsocial` |
| `web` | `grounding` |
| `xhs` | `xiaohongshu` |
| `xquik` | `xquik` |
| *(any other valid name)* | unchanged (e.g. `reddit`, `youtube`) |

```python
# pipeline.SEARCH_ALIAS (representative)
SEARCH_ALIAS = {
    "hn": "hackernews",
    "bsky": "bluesky",
    "truth": "truthsocial",
    "web": "grounding",
    "xhs": "xiaohongshu",
    "xquik": "xquik",
}
```

After normalization, `run()` intersects the list with `available_sources()`. Unknown tokens exit with `Unknown search source`. An empty list exits with `--search requires at least one source.`

<Warning>
Slash-command invocations do not pass shell flags. Restricting sources belongs in engine flags the hosting model translates, or in direct CLI: `python3 skills/last30days/scripts/last30days.py "topic" --search=reddit,youtube`.
</Warning>

## INCLUDE_SOURCES, EXCLUDE_SOURCES, and per-run opt-in

| Mechanism | Effect |
|-----------|--------|
| `EXCLUDE_SOURCES` | Removed from `available_sources()` (comma-separated, case-insensitive). Primary way to suppress TikTok/Instagram while keeping `SCRAPECREATORS_API_KEY`. |
| `INCLUDE_SOURCES=perplexity` | Enables Perplexity Sonar when `OPENROUTER_API_KEY` is set (unless `--search=perplexity` forces it). |
| `INCLUDE_SOURCES=youtube_comments` | Top-video comment enrichment via ScrapeCreators (requires SC key). |
| `INCLUDE_SOURCES=tiktok_comments` | Top-post TikTok comment enrichment (requires SC key). |
| Explicit `--search=pinterest` / `xiaohongshu` | Required for those sources even when keys/health checks pass. |

`quality_nudge.py` treats a **non-empty** `INCLUDE_SOURCES` as an allowlist when judging silent Instagram failures (zero items may be expected if `instagram` was omitted intentionally).

## Retrieval dispatch flow

```mermaid
flowchart TB
  subgraph cli [CLI / harness]
    L30[last30days.py]
    SF["--search comma list"]
  end
  subgraph pipe [pipeline.py]
    NS[normalize_requested_sources]
    AV[available_sources]
    PL[planner.plan_query]
    RS["_retrieve_stream per subquery × source"]
    NORM["_normalize_score_dedupe"]
  end
  subgraph mods [lib/*.py source modules]
    R[reddit_public / reddit]
    X[bird_x / xai_x / xurl_x]
    YT[youtube_yt]
    PM[polymarket]
    GH[github]
    GR[grounding]
    OTH[tiktok instagram hn bluesky ...]
  end
  L30 --> SF --> NS
  NS --> AV --> PL --> RS
  RS --> R & X & YT & PM & GH & GR & OTH
  RS --> NORM
```

Parallel fan-out uses a `ThreadPoolExecutor` sized from sub-query × source count. `MAX_SOURCE_FETCHES` caps **X** at two fetches per run to limit duplicate API cost.

## Normalization contract

`normalize.normalize_source_items()` maps each source’s raw dicts into `schema.SourceItem` with a shared `engagement` dict. Per-source normalizers are registered by source name (`reddit`, `x`, `youtube`, `polymarket`, etc.). Comment threads from Reddit, YouTube, and TikTok are remapped to `{score, excerpt}` in metadata `top_comments` for downstream scoring and rendering.

## Engagement scoring

Engagement is computed in two stages: **raw fields** from retrieval modules, then **pipeline scores** in `signals.py`.

### Stage 1 — Raw `engagement` dict by source

| Source | Typical `engagement` keys (from retrieval / normalize) |
|--------|--------------------------------------------------------|
| `reddit` | `score`, `num_comments`, `upvote_ratio`; top comment via metadata |
| `x` / `xquik` | `likes`, `reposts`, `replies`, `quotes` |
| `youtube` | `views`, `likes`, `comments` |
| `tiktok` / `instagram` | `views`, `likes`, `comments` (short-form video normalizer) |
| `hackernews` | `points`, `comments` |
| `bluesky` / `truthsocial` / `threads` | `likes`, `reposts`, `replies`, `quotes` (microblog shape) |
| `polymarket` | `volume` (from `volume1mo` or `volume24hr`), `liquidity` |
| `digg` | `postCount`, `uniqueAuthors`, `rank_score` |
| `pinterest` | `saves`, `comments` |
| `github` | `reactions`, `comments` (issue/PR search); profile modes use `reactions` as star-like signals |
| `xiaohongshu` | `likes`, `comments`, `favorites` |
| `grounding` / `perplexity` | Usually empty — ranking leans on text relevance and freshness |

### Stage 2 — `engagement_raw` and `local_rank_score`

`signals.engagement_raw()` applies per-source formulas (all use `log1p` on numeric inputs):

| Source | Formula class |
|--------|----------------|
| `reddit` | Custom: 50% score + 35% comments + 5% upvote_ratio + 10% top comment |
| `youtube` | Custom: 45% views + 32% likes + 13% comments + 10% top comment |
| `tiktok` | Custom: 45% views + 27% likes + 18% comments + 10% top comment |
| `x` | Weighted: likes 0.55, reposts 0.25, replies 0.15, quotes 0.05 |
| `instagram` | Weighted: views 0.50, likes 0.30, comments 0.20 |
| `hackernews` | Weighted: points 0.55, comments 0.45 |
| `bluesky` | Weighted: likes 0.40, reposts 0.30, replies 0.20, quotes 0.10 |
| `truthsocial` | Weighted: likes 0.45, reposts 0.30, replies 0.25 |
| `polymarket` | Weighted: volume 0.60, liquidity 0.40 |
| `digg` | Weighted: postCount 0.40, uniqueAuthors 0.30, rank_score 0.30 |
| Others with sparse metrics | `_generic_engagement`: mean of log1p across all numeric engagement values |

Scores are **min–max normalized to 0–100** within each stream batch (`signals.normalize`), stored on `SourceItem.engagement_score`.

**Local rank** (pre-fusion sort within a stream):

```
local_rank_score = 0.65 × local_relevance
                 + 0.25 × (freshness / 100)
                 + 0.10 × (engagement_score / 100)
```

`local_relevance` uses token overlap (`relevance.token_overlap_relevance`) with source-specific floors (e.g. YouTube views > 100k → min 0.3; GitHub `project-mode` label → min 0.8).

**Source quality** (`SOURCE_QUALITY` in `signals.py`) is a separate editorial multiplier used later in fusion weighting — not the same as engagement fields.

### Pruning rules tied to engagement

- `prune_low_relevance`: social sources with zero `engagement_score` need ~1.5× the relevance floor to survive.
- TikTok/Instagram: items with **&lt; 1000 views** drop when other sources contributed items (unless that source is the only one in the batch).

## Planner capabilities (source roles)

`planner.SOURCE_CAPABILITIES` tags each source with roles (`discussion`, `social`, `video`, `web`, `market`, …). The planner (LLM or deterministic fallback) picks sub-queries and source sets from **available** sources and query intent. `INTENT_SOURCE_EXCLUSIONS` drops `polymarket` for `concept` and `how_to` intents.

## Verification

<Steps>
<Step title="List resolved sources">

```bash
python3 skills/last30days/scripts/last30days.py --diagnose
```

Confirms keys, X backend, native web backend, and `available_sources` for the current config.

</Step>
<Step title="Run a source-scoped smoke test">

```bash
python3 skills/last30days/scripts/last30days.py "test topic" --quick --search=grounding,hackernews --emit=compact
```

Uses aliases (`hn` → `hackernews`, `web` → `grounding`) the same way as full names.

</Step>
</Steps>

## Related pages

<CardGroup>
<Card title="Configure sources" href="/configure-sources">
Credential layers, INCLUDE_SOURCES, setup wizard, and --diagnose availability checks.
</Card>
<Card title="Research pipeline" href="/research-pipeline">
Planner sub-queries, parallel fan-out, fusion, dedupe, and depth profiles after retrieval.
</Card>
<Card title="CLI reference" href="/cli-reference">
--search, targeting flags, and direct CLI vs slash-command constraints.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
Environment variables and web-backend priority that gate grounding and reasoning.
</Card>
</CardGroup>