# Troubleshooting

> LFS and artifact-path failures, viewer_id must be specified, TEST_USER_IDS empty feeds, Thunder RESOURCE_EXHAUSTED, missing retrieval_sequence, and unpublished Grox modules.

- Repository: xai-org/x-algorithm
- GitHub: https://github.com/xai-org/x-algorithm
- Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c
- Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt

## Source Files

- `phoenix/run_pipeline.py`
- `phoenix/README.md`
- `home-mixer/server.rs`
- `home-mixer/scored_posts_server.rs`
- `thunder/thunder_service.rs`
- `home-mixer/sources/phoenix_source.rs`
- `grox/main.py`
- `candidate-pipeline/scorer.rs`

---

---
title: "Troubleshooting"
description: "LFS and artifact-path failures, viewer_id must be specified, TEST_USER_IDS empty feeds, Thunder RESOURCE_EXHAUSTED, missing retrieval_sequence, and unpublished Grox modules."
---

Local Phoenix inference fails on a Git LFS pointer or a wrong `--artifacts_dir`. Home Mixer and Thunder fail later, on `viewer_id`, silent `TEST_USER_IDS` empties, `RESOURCE_EXHAUSTED`, or a missing `retrieval_sequence`. Grox and the Rust workspace do not start from this checkout: required packages and crates are unpublished.

<Warning>
This tree has no `Cargo.toml`, no `grox` install metadata, and no `home-mixer/params` or `thunder/args` modules. The only packaged, locally executable surface is `phoenix/` after the LFS archive is materialized. See [Runtime boundaries](/runtime-boundaries).
</Warning>

## Symptom index

| Symptom | Surface | Typical cause |
|---|---|---|
| `oss-phoenix-artifacts.zip` is ~135 bytes of ASCII (`version https://git-lfs.github.com/spec/v1`) | `phoenix/artifacts/` | Git LFS not installed or not pulled |
| `unzip: End-of-central-directory signature not found` | `unzip artifacts/oss-phoenix-artifacts.zip` | Unzipping the LFS pointer instead of the ~2.9 GB object |
| `FileNotFoundError: .../retrieval/config.json` (or `ranker/`, `sports_corpus.npz`, `example_sequence.json`) | `run_pipeline.py` | Archive not extracted, or `--artifacts_dir` still the default `./artifacts` |
| `INVALID_ARGUMENT: viewer_id must be specified` | `QueryBuilder.build` | `ScoredPostsQuery.viewer_id` is `0` or omitted |
| `INVALID_ARGUMENT: query must be specified` | `ForYouFeedService` | `ForYouFeedQuery.query` is missing |
| `200` with `scored_posts: []` or `items: []` and no pipeline work | `ScoredPostsServer` / `ForYouFeedServer` | `query.user_id` is in unpublished `params::TEST_USER_IDS` |
| `RESOURCE_EXHAUSTED: Server at capacity, please retry` | `ThunderServiceImpl.get_in_network_posts` | Request semaphore is full; `try_acquire` rejects immediately |
| `PhoenixSource: missing retrieval_sequence` (also `PhoenixMOESource`, `PhoenixTopicsSource`) | Phoenix candidate sources | `RetrievalSequenceQueryHydrator` failed or was skipped; `retrieval_sequence` stays `None` |
| `ThunderSource: no available channel` / `ThunderSource: ...` | `ThunderSource` | No Thunder channel, or the gRPC status (including `RESOURCE_EXHAUSTED`) is mapped into a source `Err` |
| `ModuleNotFoundError: grox.config` / `grox.service` / `monitor.logging` | `python grox/main.py` | Unpublished Grox packages |

```mermaid
flowchart TD
  req["ScoredPostsQuery / ForYouFeedQuery"] --> vid{"viewer_id == 0?"}
  vid -->|yes| e400["INVALID_ARGUMENT: viewer_id must be specified"]
  vid -->|no| test{"user_id in TEST_USER_IDS?"}
  test -->|yes| empty["OK empty scored_posts / items"]
  test -->|no| hyd["RetrievalSequenceQueryHydrator"]
  hyd -->|"UAS fetch fails"| none["retrieval_sequence stays None"]
  hyd -->|ok| seq["sequence populated"]
  none --> phx["Phoenix*Source: missing retrieval_sequence"]
  phx --> drop["source Err dropped by fetch_candidates"]
  seq --> src["ThunderSource + enabled Phoenix sources"]
  src --> cap{"Thunder semaphore full?"}
  cap -->|yes| rex["RESOURCE_EXHAUSTED: Server at capacity, please retry"]
  rex --> drop
  cap -->|no| posts["candidates collected"]
  drop --> maybe["empty or Thunder-only feed"]
```

`CandidatePipeline.execute` does not fail the RPC when a hydrator or source returns `Err`. Failed query hydrators are skipped; failed sources are dropped via `flatten`. An empty feed can therefore be a success response.

## Phoenix artifacts and Git LFS

`phoenix/artifacts/oss-phoenix-artifacts.zip` is tracked by Git LFS (`.gitattributes` applies `filter=lfs` to `*.zip` and `*.npz`). A complete object is **2903518802** bytes (about 2.70 GiB). The published pointer is:

```text
version https://git-lfs.github.com/spec/v1
oid sha256:fbc6017d00588754e22e0c7eb2f786a008a74d309c03c8085fa2fad418a83dac
size 2903518802
```

`run_pipeline.py` opens these paths under `--artifacts_dir` with no extra existence check:

| Relative path | Role |
|---|---|
| `retrieval/config.json` | Retrieval hash + transformer config |
| `retrieval/model_params.npz` | Retrieval weights |
| `retrieval/embedding_tables.npz` | Retrieval hash embeddings |
| `ranker/config.json` | Ranker hash + transformer config |
| `ranker/model_params.npz` | Ranker weights |
| `ranker/embedding_tables.npz` | Ranker hash embeddings |
| `sports_corpus.npz` | Default corpus (`--corpus_file` override) |
| `example_sequence.json` | Default user sequence (`--sequence_file` override) |

<ParamField body="--artifacts_dir" type="path" default="./artifacts">
Directory that must already contain `retrieval/` and `ranker/` as siblings, plus the corpus and sequence files unless you override those flags. The README extract layout is `artifacts/oss-phoenix-artifacts`, not `./artifacts`.
</ParamField>

<ParamField body="--sequence_file" type="path" default="artifacts_dir/example_sequence.json">
User action sequence JSON. Missing file raises `FileNotFoundError`.
</ParamField>

<ParamField body="--corpus_file" type="path" default="artifacts_dir/sports_corpus.npz">
Precomputed candidate representations. Missing file raises `FileNotFoundError`.
</ParamField>

<AccordionGroup>
<Accordion title="Pointer zip: unzip reports it is not a zipfile">
`file artifacts/oss-phoenix-artifacts.zip` prints `ASCII text` when LFS did not materialize the object. `unzip` then fails with `End-of-central-directory signature not found`.

<Steps>
<Step title="Confirm the pointer">
Check size. A pointer is about 135 bytes. A real archive is 2903518802 bytes.
</Step>
<Step title="Install and pull LFS">
```bash
git lfs install
git lfs pull
```
Then re-check: `git lfs ls-files` should list `phoenix/artifacts/oss-phoenix-artifacts.zip`, and `file` should report a Zip archive, not ASCII.
</Step>
<Step title="Extract into the documented layout">
```bash
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
This creates `artifacts/oss-phoenix-artifacts/` with `retrieval/`, `ranker/`, `sports_corpus.npz`, and `example_sequence.json`.
</Step>
</Steps>
</Accordion>

<Accordion title="FileNotFoundError on retrieval/config.json after unzip">
`run_pipeline.py` defaults `--artifacts_dir` to `./artifacts`. The extract step writes `artifacts/oss-phoenix-artifacts/`. Running from `phoenix/` without the flag looks for `phoenix/artifacts/retrieval/config.json`, which does not exist.

<CodeGroup>
```bash title="Correct — extracted layout"
cd phoenix
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```

```bash title="Wrong — default path"
cd phoenix
uv run run_pipeline.py
# FileNotFoundError: artifacts/retrieval/config.json
```
</CodeGroup>
</Accordion>

<Accordion title="BadZipFile or numpy load of an LFS pointer">
If you point `--corpus_file` or a model NPZ at a leftover LFS pointer (ASCII starting with `version https://git-lfs.github.com/spec/v1`), `numpy.load` / zip extraction fails. Pull LFS, then use files from the extracted `oss-phoenix-artifacts/` tree, not the `.zip` path itself.
</Accordion>
</AccordionGroup>

Success signal for the local pipeline is the ranked table (`PIPELINE RESULTS — User …`) printed by `run_pipeline.py`. Phoenix unit tests do not need the archive:

```bash
cd phoenix
uv sync
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```

`run_ranker.py` and `run_retrieval.py` initialize random demo weights. They do not load `oss-phoenix-artifacts` and cannot diagnose LFS or extract-path failures.

## viewer_id must be specified

`QueryBuilder.build` rejects a proto `ScoredPostsQuery` whose `viewer_id` is `0` (protobuf default for an omitted `uint64`) before Gizmoduck, feature switches, or the pipeline run:

```text
INVALID_ARGUMENT: viewer_id must be specified
```

This gate is shared by:

| RPC | Extra request check |
|---|---|
| `ScoredPostsService.GetScoredPosts` | None beyond `viewer_id` |
| `ScoredPostsService.GetDebugScoredPosts` | Uses `DebugScoredPostsQuery.query` (default empty proto if unset) — still requires non-zero `viewer_id` |
| `ForYouFeedService.GetForYouFeed` | `ForYouFeedQuery.query` must be present or the status is `query must be specified` |
| `ForYouFeedService.GetForYouFeedUrt` | Same nested `query` requirement |

<RequestExample>
```text title="Rejected scored-posts request"
viewer_id: 0
# or viewer_id omitted
```
</RequestExample>

<ResponseExample>
```text title="QueryBuilder.build"
INVALID_ARGUMENT: viewer_id must be specified
```
</ResponseExample>

A non-zero `viewer_id` is copied onto `ScoredPostsQuery.user_id`. Gizmoduck `get_viewer_data` is then called with a **200 ms** timeout. Timeout or client error becomes `ViewerData::default()` and does **not** fail the request. `in_network_only` becomes true only when the proto flag is set or `allow_for_you_recommendations == Some(false)`.

Home Mixer process flags (`home-mixer/main.rs`) do not supply a viewer. They only bind the server:

<ParamField body="--grpc_port" type="u16" default="50051">gRPC listen port.</ParamField>
<ParamField body="--metrics_port" type="u16" default="9090">Metrics port.</ParamField>
<ParamField body="--shard_coordinate" type="i16" default="-1">Negative disables shard coordinates.</ParamField>
<ParamField body="--datacenter" type="string" default="atla">Feature-switch custom string `datacenter`.</ParamField>

`XServiceBuilder` still needs unpublished `params::STRINGCENTER_BUNDLE_PATH`, `params::FS_PATH`, decider paths, and mTLS from the environment. A missing `viewer_id` is a request bug, not a process-flag bug.

## TEST_USER_IDS empty feeds

If `params::TEST_USER_IDS` contains `query.user_id`, both product servers return an empty success and skip `CandidatePipeline.execute`:

| Server | Return value |
|---|---|
| `ScoredPostsServer.run_pipeline` | `PipelineOutput { scored_posts: vec![], pipeline_result: PipelineResult::empty() }` |
| `ForYouFeedServer.get_for_you_feed` | `ForYouFeedOutput { items: vec![] }` |

`PipelineResult::empty()` is documented in the pipeline crate as the short-circuit for test users. `GetForYouFeedUrt` still serializes that empty item list into a URT timeline.

<Warning>
This is not an error status. Logs such as `Scored Posts response - … N posts` never appear. Treat a silent empty feed for a known test account as the `TEST_USER_IDS` gate, not as Phoenix or Thunder outage.
</Warning>

`params::TEST_USER_IDS` and `params::TRACE_USER_IDS` live in the unpublished `home-mixer/params` module. `TRACE_USER_IDS` only calls `b3_info.force_sample()`; it does not empty the feed.

## Thunder RESOURCE_EXHAUSTED

`ThunderServiceImpl` constructs `Semaphore::new(max_concurrent_requests)` from unpublished `thunder/args`. `GetInNetworkPosts` uses `try_acquire`, not a blocking wait:

| Semaphore result | Metrics | gRPC status |
|---|---|---|
| permit acquired | `IN_FLIGHT_REQUESTS.inc()` (decremented on drop) | request proceeds |
| at capacity | `REJECTED_REQUESTS.inc()` | `RESOURCE_EXHAUSTED` / `Server at capacity, please retry` |

Home Mixer `ThunderSource` always sends `debug: false`. It maps any tonic error, including this one, to `ThunderSource: {status}`. `fetch_candidates` then drops that `Err`, so the scored-posts RPC can still return `OK` with only out-of-network (or zero) candidates.

<Info>
Strato following-list fallback in Thunder runs only when `following_user_ids` is empty **and** `req.debug` is true. The Home Mixer client never sets `debug`, so an empty `user_features.followed_user_ids` (failed `FollowedUserIdsQueryHydrator`, or no follows) yields an empty in-network set without calling Strato.
</Info>

Related Thunder source strings:

- `ThunderSource: no available channel` — `ThunderClient.get_random_channel` returned `None` for the resolved `ThunderCluster`.
- `Failed to fetch following list: …` — debug-only Strato path.
- `Failed to process posts: …` — `spawn_blocking` join error after PostStore lookup.

Raise `max_concurrent_requests` on the Thunder process, reduce caller concurrency, or retry. There is no in-process queue.

## Missing retrieval_sequence

`ScoredPostsQuery::new` sets `retrieval_sequence` and `columnar_retrieval_sequence` to `None`. `RetrievalSequenceQueryHydrator` is the writer: it calls `UserActionAggregationClient.fetch_aggregated_sequence` and, on success, copies `result.sequence` and `result.columnar_bytes`.

`hydrate_query` runs enabled hydrators in parallel and **only applies `Ok` results**. A failed UAS call is logged (`Aggregation service call failed: …`) and left unset.

These sources then hard-fail if the field is still `None`:

| Source | Error string | Enable predicate (must all hold) |
|---|---|---|
| `PhoenixSource` | `PhoenixSource: missing retrieval_sequence` | Not a non-bulk topic request; new-user topic retrieval not forcing topics-only; `!in_network_only`; `!has_cached_posts` |
| `PhoenixMOESource` | `PhoenixMOESource: missing retrieval_sequence` | `EnablePhoenixMOESource`; same topic / cache / in-network guards |
| `PhoenixTopicsSource` | `PhoenixTopicsSource: missing retrieval_sequence` | Topic request (non-bulk) or new-user topic IDs; `!in_network_only`; `!has_cached_posts` |

Because `fetch_candidates` flattens `Result`, those errors remove Phoenix candidates instead of failing `GetScoredPosts`. Thunder (and cache / TweetMixer when enabled) can still populate the feed.

<Check>
If the request is `in_network_only` or `has_cached_posts`, Phoenix sources are disabled and this string never appears. Diagnose an empty out-of-network slice by checking UAS hydration logs, then `query.retrieval_sequence`, then source enable predicates.
</Check>

`ScoringSequenceQueryHydrator` is a sibling hydrator for ranking, not retrieval. A missing scoring sequence is a later scorer concern and is not the `missing retrieval_sequence` string.

## Unpublished Grox modules

`grox/main.py` cannot start in this checkout. `serve()` imports modules that are not in the tree:

```python
from grox.engine import Engine
from grox.service import GrpcServer
from grox.dispatcher import Dispatcher
from grox.config.config import grox_config
```

Present top-level packages: `classifiers`, `data_loaders` (partial), `embedder`, `engine.py`, `dispatcher.py`, `generators`, `lib`, `plans`, `schedules`, `summarizer`, `tasks`.

Missing imports that fail first:

| Import | Used by |
|---|---|
| `grox.config.config` (`grox_config`, `ModelName`, `KafkaTopicName`, …) | `main.py`, `engine.py`, `dispatcher.py`, classifiers, loaders, summarizers |
| `grox.service` (`GrpcServer`) | `main.py` |
| `grox.data_loaders.data_types` | classifiers, loaders, tasks |
| `grox.data_loaders.media_processor` / `media_loader` / `media_description_loader` | `engine.py`, classifiers |
| `grox.data_loaders.mappers.post_mapper` | `strato_loader.py` |
| `grox.lm.*` (`convo`, `post`, `post_v5`, `thread`, `user`) | classifiers, summarizer |
| `grox.prompts.template` | classifiers |
| `grox.classifiers.content.classifier_data_collection` | classifier graph |
| `monitor.logging` / `monitor.metrics` | `schedules/init.py`, `engine.py`, `dispatcher.py` |

There is no Grox `pyproject.toml` or requirements file. External names such as `setproctitle` and `tenacity` are also imported from published files but not pinned here.

Treat `python grox/main.py` `ModuleNotFoundError` as expected for this snapshot. The published tree is the plan/task graph, not a runnable service.

## Unpublished Rust workspace

The same boundary applies to Home Mixer, Thunder, and the candidate-pipeline crate:

| Missing in-tree module | Imported as |
|---|---|
| `home-mixer/params.rs` | `crate::params` / `params::TEST_USER_IDS`, feature params, `STRINGCENTER_BUNDLE_PATH` |
| `home-mixer/clients/` | Gizmoduck, UAS, TES, Kafka, … |
| `home-mixer/util.rs` | URT helpers |
| `thunder/args.rs` | `max_concurrent_requests`, Kafka and port flags |
| `thunder/config.rs` | `MAX_INPUT_LIST_SIZE`, `MAX_POSTS_TO_RETURN`, `MAX_VIDEOS_TO_RETURN` |
| `thunder/metrics.rs`, `o2.rs`, `schema.rs`, `strato_client.rs` | service internals |

Published files also `use` unpublished crates (`xai_home_mixer_proto`, `xai_thunder_proto`, `xai_x_service_builder`, `xai_http_server`, `xai_candidate_pipeline::component_library`, …). There is no workspace `Cargo.toml`. `cargo build` is not a supported local action.

## Adjacent pipeline errors

These are not the six named failures, but they produce empty or unscored feeds after the request is accepted.

<AccordionGroup>
<Accordion title="Scorer or hydrator length_mismatch">
`Scorer::run` and `Hydrator::run` require the output vector length to match the input. On mismatch they log `Skipped: length_mismatch expected=N got=M` and replace the batch with `Err` entries. `update_all` then leaves those candidates unchanged (no scores copied). The RPC still returns whatever the selector keeps.
</Accordion>

<Accordion title="Empty Thunder following list with debug false">
`FollowedUserIdsQueryHydrator` writes `user_features.followed_user_ids`. If that hydrator fails, Thunder is called with an empty following list and `debug: false`, so it does not consult Strato. Combined with a missing `retrieval_sequence`, both major sources contribute nothing.
</Accordion>
</AccordionGroup>

## Next

<CardGroup>
<Card title="Installation" href="/installation">
Python 3.11+, uv or pip, Git LFS, and the extract layout required before inference.
</Card>
<Card title="Quickstart" href="/quickstart">
Extract `oss-phoenix-artifacts` and recognize the ranked table as success.
</Card>
<Card title="Runtime boundaries" href="/runtime-boundaries">
What this checkout can execute versus Home Mixer, Thunder, and Grox snapshots.
</Card>
<Card title="Run the inference pipeline" href="/run-inference-pipeline">
`--artifacts_dir`, corpus, and sequence flags after LFS is healthy.
</Card>
<Card title="Assemble a Home Mixer request" href="/assemble-home-mixer-request">
`QueryBuilder.viewer_id` validation and CLI flags.
</Card>
<Card title="ScoredPostsQuery and gRPC" href="/scored-posts-query">
`TEST_USER_IDS` empty responses and proto mapping.
</Card>
<Card title="Thunder GetInNetworkPosts" href="/thunder-in-network-posts">
Semaphore capacity, Kafka ingest, and Strato fallback.
</Card>
<Card title="Execute Grox content plans" href="/execute-grox-plans">
Engine / Dispatcher / GrpcServer startup that depends on unpublished modules.
</Card>
<Card title="Test Phoenix" href="/test-phoenix">
`uv run pytest` targets that do not need the artifact archive.
</Card>
</CardGroup>