# Configure sources

> Credential layers (.env paths, keychain), per-source API keys, INCLUDE_SOURCES, setup wizard auto-actions, and --diagnose availability checks.

- Repository: mvanhorn/last30days-skill
- GitHub: https://github.com/mvanhorn/last30days-skill
- Human docs: https://grok-wiki.com/public/docs/mvanhorn-last30days-skill-50b5421a8cca
- Complete Markdown: https://grok-wiki.com/public/docs/mvanhorn-last30days-skill-50b5421a8cca/llms-full.txt

## Source Files

- `CONFIGURATION.md`
- `skills/last30days/scripts/lib/env.py`
- `skills/last30days/scripts/lib/setup_wizard.py`
- `skills/last30days/scripts/lib/providers.py`
- `skills/last30days/scripts/setup-keychain.sh`
- `skills/last30days/scripts/last30days.py`

---

---
title: "Configure sources"
description: "Credential layers (.env paths, keychain), per-source API keys, INCLUDE_SOURCES, setup wizard auto-actions, and --diagnose availability checks."
---

The last30days engine loads credentials through `lib/env.py::get_config()`, merges project and global `.env` files with optional macOS Keychain entries, and computes which retrieval backends are runnable via `lib/pipeline.py::available_sources()` and `diagnose()`. Slash-command hosts read the same keys when they invoke `scripts/last30days.py`; direct CLI users can verify wiring with `--diagnose` before spending a full research run.

## Credential resolution order

Configuration is assembled once per engine invocation. Effective value for each key follows this precedence (highest wins):

| Priority | Source | Path or mechanism |
| --- | --- | --- |
| 1 | Process environment | `export KEY=value` or harness-injected env |
| 2 | Project-scoped file | `.claude/last30days.env` (walks up from `cwd`) |
| 3 | Global file | `~/.config/last30days/.env` (override dir with `LAST30DAYS_CONFIG_DIR`) |
| 4 | macOS Keychain | Generic password service `last30days-<KEY>` (Darwin only) |

<Note>
`LAST30DAYS_CONFIG_DIR=""` disables global file loading entirely (clean/no-config mode). `LAST30DAYS_CONFIG_DIR=/path` points the global file at `/path/.env`.
</Note>

```text
  os.environ ──────────────┐
                           ▼
  .claude/last30days.env ──┼──► merged config dict
                           │         │
  ~/.config/last30days/.env┘         │ _CONFIG_SOURCE label
                                     ▼
  Keychain (last30days-*) ──► lowest file priority
```

POSIX hosts warn on stderr when a loaded secrets file is group- or world-readable; target mode `600`.

### macOS Keychain

`scripts/setup-keychain.sh` stores generic passwords for the keys listed in `lib/env.py::KEYCHAIN_KEYS` (kept in sync via `tests/test_env_keychain.py`). Each item uses service name `last30days-<KEY>` for the current user.

<CodeGroup>
```bash title="Interactive store all keys"
./skills/last30days/scripts/setup-keychain.sh
```

```bash title="List stored items"
./skills/last30days/scripts/setup-keychain.sh --list
```

```bash title="Store one key"
security add-generic-password -a "$USER" -s last30days-XAI_API_KEY -w "xai-..."
```
</CodeGroup>

Keychain values never override `.env` or process env; they only fill gaps.

## Environment file locations

| File | When used |
| --- | --- |
| `.claude/last30days.env` | Present in `cwd` or any parent up to `$HOME` |
| `$LAST30DAYS_CONFIG_DIR/.env` | Custom global path |
| `~/.config/last30days/.env` | Default global fallback |

Per-client workflows typically place keys and `INCLUDE_SOURCES` in `.claude/last30days.env`, then `cd` into that project so the engine picks up the file without extra flags.

## Per-source credentials

### Always-on (no API key)

| Source | Requirement |
| --- | --- |
| Reddit | Public Algolia API |
| Hacker News | Free Algolia API |
| Polymarket | Free Gamma API |

### CLI-backed (no skill API key)

| Source | Requirement |
| --- | --- |
| GitHub | `gh` CLI installed and authenticated |
| YouTube | `yt-dlp` on `PATH`, or `SCRAPECREATORS_API_KEY` as SC fallback |
| Digg | `digg-pp-cli` on `PATH` |

### Keyed social and enrichment

| Source | Key(s) | Engine gate |
| --- | --- | --- |
| X / Twitter | `XAI_API_KEY`, or `AUTH_TOKEN` + `CT0`, or `FROM_BROWSER`, or `xurl` OAuth CLI | `env.get_x_source()` |
| TikTok | `SCRAPECREATORS_API_KEY` or `APIFY_API_TOKEN` | On when SC/Apify key set; opt out with `EXCLUDE_SOURCES` |
| Instagram | `SCRAPECREATORS_API_KEY` | Same as TikTok |
| Threads | `SCRAPECREATORS_API_KEY` | On with SC key; opt out with `EXCLUDE_SOURCES=threads` |
| Pinterest | `SCRAPECREATORS_API_KEY` | Only when `--search=pinterest` (or planner requests `pinterest`) |
| Bluesky | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` | `env.is_bluesky_available()` |
| Truth Social | `TRUTHSOCIAL_TOKEN` or browser cookie via `FROM_BROWSER` | `env.is_truthsocial_available()` |
| Xiaohongshu | `XIAOHONGSHU_API_BASE` (HTTP API health + login probe) | `--search=xiaohongshu` when reachable |
| Xquik | `XQUIK_API_KEY` | `env.is_xquik_available()` |

### Web grounding and deep research

| Capability | Key(s) | Notes |
| --- | --- | --- |
| Native web search (Step 2 / `--auto-resolve`) | `BRAVE_API_KEY`, `EXA_API_KEY`, `SERPER_API_KEY`, or `PARALLEL_API_KEY` | Auto priority: Brave → Exa → Serper → Parallel; override with `--web-backend` |
| Perplexity Sonar | `OPENROUTER_API_KEY` + `INCLUDE_SOURCES=perplexity` or `--search=perplexity` | Also auto-added when `--deep-research` runs |
| Deep Research (~$0.90/query) | `OPENROUTER_API_KEY` | `--deep-research` flag |

### Reasoning (headless runs only)

When the host model is not doing plan/rerank, `lib/providers.py::resolve_runtime()` auto-picks: Gemini (`GOOGLE_API_KEY` / `GEMINI_API_KEY` / `GOOGLE_GENAI_API_KEY`) → OpenAI (`OPENAI_API_KEY`) → xAI → OpenRouter → local deterministic fallback. Pin with `LAST30DAYS_REASONING_PROVIDER`.

<Warning>
Do not commit real API keys, cookies, or `.env` contents. Tests use obvious dummy values only.
</Warning>

### Legacy and rotation notes

- `SCRAPE_CREATORS_API_KEY` (underscore spelling) is accepted as an alias when the canonical `SCRAPECREATORS_API_KEY` is unset.
- Comma-separated `SCRAPECREATORS_API_KEY` values rotate per run via `random.choice`.
- Browser cookie extraction (`FROM_BROWSER`): default tries Firefox and Safari silently; `FROM_BROWSER=auto` adds Chrome; `FROM_BROWSER=off` disables extraction.

## INCLUDE_SOURCES and EXCLUDE_SOURCES

Both are comma-separated, case-insensitive lists in `.env` or process env.

### EXCLUDE_SOURCES (runtime denylist)

`pipeline.available_sources()` removes any listed source after computing availability. This is the supported way to disable TikTok, Instagram, or Threads while keeping `SCRAPECREATORS_API_KEY` set.

```bash
EXCLUDE_SOURCES=tiktok,instagram,threads
```

### INCLUDE_SOURCES (opt-in extensions)

Empty `INCLUDE_SOURCES` means no allowlist filter in the pipeline. Non-empty values gate specific add-ons:

| Token | Effect |
| --- | --- |
| `perplexity` | Enables Perplexity Sonar when `OPENROUTER_API_KEY` is set |
| `youtube_comments` | Comment enrichment on YouTube items (requires SC key) |
| `tiktok_comments` | Comment enrichment on TikTok items (requires SC key) |
| `threads`, `pinterest` | Mentioned in quality nudges; Threads is already default-on with SC key; Pinterest still needs `--search=pinterest` |

`--deep-research` temporarily appends `perplexity` to `INCLUDE_SOURCES` for that run.

<Info>
Quality scoring (`lib/quality_nudge.py`) treats a non-empty `INCLUDE_SOURCES` as an intentional allowlist when detecting “silent” Instagram failures. That does not change which sources the pipeline fetches unless combined with `EXCLUDE_SOURCES` or `--search`.
</Info>

### Per-run source selection (`--search`)

`--search` (alias map: `hn`, `bsky`, `truth`, `web`, `xhs`, `xquik`) intersects the availability list. Explicit `--search=perplexity` or `--search=threads` can enable those sources without editing `INCLUDE_SOURCES` (see `tests/test_cli_v3.py`).

## Example `.env` skeleton

```bash
# Reasoning (headless / cron)
GOOGLE_API_KEY=<your-gemini-key>

# Web grounding
BRAVE_API_KEY=<your-brave-key>

# ScrapeCreators family (TikTok, Instagram, Threads default-on)
SCRAPECREATORS_API_KEY=<your-sc-key>
EXCLUDE_SOURCES=threads          # optional denylist

# Opt-in enrichments
INCLUDE_SOURCES=youtube_comments,perplexity

# X (pick one path)
XAI_API_KEY=<your-xai-key>
# FROM_BROWSER=firefox

# Bluesky
BSKY_HANDLE=<handle>.bsky.social
BSKY_APP_PASSWORD=xxxx-xxxx-xxxx-xxxx
BSKY_SEARCH_HOST=api.bsky.app

SETUP_COMPLETE=true
```

After editing: `chmod 600` on the secrets file.

## First-run setup wizard

Setup spans the SKILL.md Step 0 UX (host model reads `nux-wizard.md` when referenced) and engine-side automation in `lib/setup_wizard.py`.

### Detection

- **SKILL path:** missing `~/.config/last30days/.env`, or file without `SETUP_COMPLETE=true` → run wizard; otherwise skip silently.
- **Engine path:** `setup_wizard.is_first_run(config)` when `SETUP_COMPLETE` is unset.

### CLI auto-setup (`setup` subcommand)

From the skill scripts directory (or with full path):

```bash
python3 skills/last30days/scripts/last30days.py setup
```

<Steps>
<Step title="Auto actions">
`run_auto_setup()` probes browsers in `auto` mode for X and Truth Social cookies (`COOKIE_DOMAINS`), checks `yt-dlp`, and runs `brew install yt-dlp` when Homebrew exists.
</Step>
<Step title="Persist flags">
`write_setup_config()` appends `SETUP_COMPLETE=true` and `FROM_BROWSER=<browser>` to the global `.env` without overwriting existing keys.
</Step>
<Step title="Optional auth flows">
`setup --github` — PAT via `gh auth token`, else GitHub device flow through ScrapeCreators (returns JSON API key). `setup --device-auth` — device flow only. `setup --openclaw` — JSON probe of CLI + key presence (no cookies).
</Step>
</Steps>

Status text for auto-setup is returned by `get_setup_status_text()` on stderr.

## Verify with `--diagnose`

`--diagnose` loads config, calls `pipeline.diagnose()`, prints JSON, and exits without running research. No topic required.

```bash
python3 skills/last30days/scripts/last30days.py --diagnose
```

<ParamField body="--search" type="string">
Optional comma-separated list; narrows `available_sources` the same way as a real run (e.g. `--diagnose --search=threads,pinterest`).
</ParamField>

### Response shape

| Field | Meaning |
| --- | --- |
| `providers` | `google`, `openai`, `xai`, `openrouter` booleans |
| `local_mode` | `true` when no reasoning API keys resolve |
| `reasoning_provider` | `LAST30DAYS_REASONING_PROVIDER` or `auto` |
| `x_backend` | Active X backend: `bird`, `xai`, `xurl`, or `null` |
| `bird_installed` / `bird_authenticated` / `bird_username` | Bird CLI state |
| `native_web_backend` | `brave`, `exa`, `serper`, `parallel`, or `null` |
| `has_scrapecreators` | SC key detected |
| `has_github` | `GITHUB_TOKEN` or `gh` on PATH |
| `available_sources` | Sources that would run for this config (and `--search` filter) |

During normal runs, `ui.show_diagnostic_banner()` and post-run promos consume the same `diag` object when core sources are missing.

### Quick interpretation

| Symptom in `available_sources` | Typical fix |
| --- | --- |
| No `x` | Add `XAI_API_KEY`, Bird cookies, or install/authenticate `xurl` |
| No `youtube` | `brew install yt-dlp` or set SC key |
| No `grounding` | Add a web search API key or use a host with native WebSearch |
| No `tiktok` / `instagram` | Set `SCRAPECREATORS_API_KEY`; check `EXCLUDE_SOURCES` |
| No `perplexity` | `OPENROUTER_API_KEY` + `INCLUDE_SOURCES=perplexity` or `--search=perplexity` |
| No `bluesky` | `BSKY_HANDLE` + `BSKY_APP_PASSWORD` |

## Configuration vs skill contract

- **Engine truth:** `get_config()`, `available_sources()`, `diagnose()` in `scripts/lib/`.
- **Slash-command truth:** `SKILL.md` Step 0 (wizard), intent parsing, and `ACTIVE_SOURCES_LIST` messaging must match keys you actually set.
- **User reference:** `CONFIGURATION.md` mirrors knobs for operators; if it diverges from code, code wins.

New env vars or CLI flags that affect sources should update `CONFIGURATION.md`, `SKILL.md`, and this page together.

## Related pages

<CardGroup>
<Card title="Configuration reference" href="/configuration-reference">
Full env var catalog, web-backend priority, output paths, and trend-monitoring toggles.
</Card>
<Card title="Data sources" href="/data-sources">
Per-platform retrieval modules, scoring inputs, and `--search` alias map.
</Card>
<Card title="Quickstart" href="/quickstart">
First `/last30days` run, wizard signals, and initial `--diagnose` check.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Missing-source recovery, thin results, and documented failure modes.
</Card>
<Card title="Per-client setup" href="/per-client-setup">
Project-scoped `.claude/last30days.env` and save-dir isolation patterns.
</Card>
</CardGroup>