# X For You Algorithm Documentation
> Technical reference for the X For You recommendation stack: Phoenix retrieval and ranking, Home Mixer orchestration, Thunder in-network storage, Grox content understanding, and the Candidate Pipeline framework.
## Context Links
- [Agent index](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms.txt)
- [Human interactive docs](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c)
- [GitHub repository](https://github.com/xai-org/x-algorithm)
## Repository Metadata
- Repository: xai-org/x-algorithm
- Generated: 2026-08-13T19:56:12.136Z
- Updated: 2026-08-13T20:05:51.617Z
- Runtime: Grok CLI
- Format: Documentation
- Pages: 25
## Page Index
- 01. [Overview](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/01-overview.md) - Public surfaces of the For You stack, who can run what from this checkout, and the first docs routes to follow.
- 02. [Installation](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/02-installation.md) - Python 3.11+, uv or pip, Git LFS for the Phoenix artifact archive, and the extract layout required before inference.
- 03. [Quickstart](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/03-quickstart.md) - Extract oss-phoenix-artifacts, run run_pipeline.py, and recognize the ranked table as the success signal.
- 04. [Runtime boundaries](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/04-runtime-boundaries.md) - What this checkout can execute locally versus production Home Mixer, Thunder, and Grox snapshots that depend on unpublished crates.
- 05. [For You request lifecycle](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/05-for-you-request-lifecycle.md) - CandidatePipeline.execute stages from query hydration through side effects, and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline.
- 06. [In-network and out-of-network](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/06-in-network-and-out-of-network.md) - Thunder followed-author lookup versus Phoenix retrieval, enable predicates, and how in_network_only changes sourcing and served_type.
- 07. [Candidate isolation](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/07-candidate-isolation.md) - make_recsys_attn_mask rules: candidates attend to user and history plus self, never to other candidates, so scores stay batch-independent.
- 08. [Hash embeddings](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/08-hash-embeddings.md) - HashConfig, linear-congruential ID hashing, pad-offset unified tables, and how RecsysBatch hashes are looked up before the transformer.
- 09. [Multi-action scoring](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/09-multi-action-scoring.md) - Per-action logits, demo weighted sums in run_pipeline.py, and production WeightedScorer versus RankingScorer weight tables.
- 10. [Run the inference pipeline](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/10-run-the-inference-pipeline.md) - Load retrieval and ranker checkpoints, encode example_sequence.json, retrieve from sports_corpus.npz, and rank with the published action weights.
- 11. [Customize a user sequence](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/11-customize-a-user-sequence.md) - example_sequence.json fields, ActionName indices, history padding to history_seq_len, and --top_k_retrieval / --top_k_display knobs.
- 12. [Assemble a Home Mixer request](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/12-assemble-a-home-mixer-request.md) - HomeMixerServer CLI flags, QueryBuilder.viewer_id validation, ScoredPostsQuery construction, and For You versus ScoredPosts entry points.
- 13. [Blend ads into the feed](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/13-blend-ads-into-the-feed.md) - AdsSource intake, SafeGapAdsBlender versus PartitionOrganicAdsBlender, AdsBlenderType selection, and prompt / who-to-follow / push-to-home insertion.
- 14. [Execute Grox content plans](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/14-execute-grox-content-plans.md) - Engine, Dispatcher, and GrpcServer startup, PlanMaster fan-out, TaskEligibility gating, and merged TaskResult fields.
- 15. [Add a pipeline component](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/15-add-a-pipeline-component.md) - Source, Hydrator, Filter, Scorer, Selector, QueryHydrator, and SideEffect contracts, including length-match and no-drop rules.
- 16. [Phoenix model configuration](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/16-phoenix-model-configuration.md) - PhoenixModelConfig and PhoenixRetrievalModelConfig fields, TransformerConfig, RecsysBatch, RecsysEmbeddings, and published config.json keys.
- 17. [Action indices](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/17-action-indices.md) - ActionName indices used in example_sequence.json, run_pipeline.py IDX_* constants, runners.ACTIONS labels, and scorer weight keys.
- 18. [ScoredPostsQuery and gRPC](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/18-scoredpostsquery-and-grpc.md) - ScoredPostsQuery fields, QueryBuilder defaults, TEST_USER_IDS empty responses, and ScoredPost / ForYouFeed / URT response mapping.
- 19. [Candidate sources](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/19-candidate-sources.md) - PhoenixCandidatePipeline and ForYouCandidatePipeline sources, enable predicates, cluster resolution, and served_type assignment.
- 20. [Filters and hydrators](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/20-filters-and-hydrators.md) - Pre-score and post-selection filters, query hydrators that fill ScoredPostsQuery, and candidate hydrators that must preserve order and length.
- 21. [Scorers and weights](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/21-scorers-and-weights.md) - PhoenixScorer cluster and egress fallback, WeightedScorer and RankingScorer formulas, AuthorDiversityScorer decay, OON and VM rankers, TopKScoreSelector.
- 22. [Thunder GetInNetworkPosts](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/22-thunder-getinnetworkposts.md) - InNetworkPostsService RPC fields, PostStore timelines and retention, Kafka ingest, semaphore capacity, and Strato following-list fallback.
- 23. [Grox plans and tasks](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/23-grox-plans-and-tasks.md) - PlanMaster.ALL_PLANS, TaskEligibility values, DAG TASK_DEPENDENCIES, ContentClassifier, and TaskResult merge rules.
- 24. [Test Phoenix](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/24-test-phoenix.md) - uv run pytest targets, attention-mask and retrieval assertions, and the success criteria encoded in the Phoenix test modules.
- 25. [Troubleshooting](https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/25-troubleshooting.md) - LFS and artifact-path failures, viewer_id must be specified, TEST_USER_IDS empty feeds, Thunder RESOURCE_EXHAUSTED, missing retrieval_sequence, and unpublished Grox modules.
## Source File Index
- `.gitattributes`
- `candidate-pipeline/candidate_pipeline.rs`
- `candidate-pipeline/filter.rs`
- `candidate-pipeline/hydrator.rs`
- `candidate-pipeline/lib.rs`
- `candidate-pipeline/query_hydrator.rs`
- `candidate-pipeline/scorer.rs`
- `candidate-pipeline/selector.rs`
- `candidate-pipeline/side_effect.rs`
- `candidate-pipeline/source.rs`
- `grox/classifiers/content/classifier.py`
- `grox/dispatcher.py`
- `grox/engine.py`
- `grox/main.py`
- `grox/plans/plan_master.py`
- `grox/plans/plan_post_safety.py`
- `grox/plans/plan.py`
- `grox/schedules/types.py`
- `grox/tasks/task.py`
- `home-mixer/ads/mod.rs`
- `home-mixer/ads/partition_organic_blender.rs`
- `home-mixer/ads/safe_gap_blender.rs`
- `home-mixer/ads/util.rs`
- `home-mixer/candidate_hydrators/ads_brand_safety_hydrator.rs`
- `home-mixer/candidate_hydrators/core_data_candidate_hydrator.rs`
- `home-mixer/candidate_hydrators/mod.rs`
- `home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`
- `home-mixer/filters/age_filter.rs`
- `home-mixer/filters/mod.rs`
- `home-mixer/filters/vf_filter.rs`
- `home-mixer/for_you_server.rs`
- `home-mixer/lib.rs`
- `home-mixer/main.rs`
- `home-mixer/models/candidate.rs`
- `home-mixer/models/query.rs`
- `home-mixer/models/user_features.rs`
- `home-mixer/query_hydrators/mod.rs`
- `home-mixer/query_hydrators/retrieval_sequence_query_hydrator.rs`
- `home-mixer/query_hydrators/scoring_sequence_query_hydrator.rs`
- `home-mixer/scored_posts_server.rs`
- `home-mixer/scorers/author_diversity_scorer.rs`
- `home-mixer/scorers/mod.rs`
- `home-mixer/scorers/oon_scorer.rs`
- `home-mixer/scorers/phoenix_scorer.rs`
- `home-mixer/scorers/ranking_scorer.rs`
- `home-mixer/scorers/vm_ranker.rs`
- `home-mixer/scorers/weighted_scorer.rs`
- `home-mixer/selectors/blender_selector.rs`
- `home-mixer/selectors/top_k_score_selector.rs`
- `home-mixer/server.rs`
- `home-mixer/sources/ads_source.rs`
- `home-mixer/sources/cached_posts_source.rs`
- `home-mixer/sources/mod.rs`
- `home-mixer/sources/phoenix_moe_source.rs`
- `home-mixer/sources/phoenix_source.rs`
- `home-mixer/sources/phoenix_topics_source.rs`
- `home-mixer/sources/thunder_source.rs`
- `LICENSE`
- `phoenix/artifacts/oss-phoenix-artifacts.zip`
- `phoenix/grok.py`
- `phoenix/pyproject.toml`
- `phoenix/README.md`
- `phoenix/recsys_model.py`
- `phoenix/recsys_retrieval_model.py`
- `phoenix/run_pipeline.py`
- `phoenix/run_ranker.py`
- `phoenix/run_retrieval.py`
- `phoenix/runners.py`
- `phoenix/test_recsys_model.py`
- `phoenix/test_recsys_retrieval_model.py`
- `phoenix/uv.lock`
- `README.md`
- `thunder/kafka_utils.rs`
- `thunder/kafka/tweet_events_listener.rs`
- `thunder/lib.rs`
- `thunder/main.rs`
- `thunder/posts/post_store.rs`
- `thunder/thunder_service.rs`
---
## 01. Overview
> Public surfaces of the For You stack, who can run what from this checkout, and the first docs routes to follow.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/01-overview.md
- Generated: 2026-08-13T19:41:58.227Z
### Source Files
- `README.md`
- `phoenix/README.md`
- `phoenix/run_pipeline.py`
- `home-mixer/lib.rs`
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`
- `candidate-pipeline/candidate_pipeline.rs`
- `thunder/thunder_service.rs`
---
title: "Overview"
description: "Public surfaces of the For You stack, who can run what from this checkout, and the first docs routes to follow."
---
This repository is a source snapshot of X's For You recommendation stack: Home Mixer orchestrates feed assembly, Thunder serves in-network posts, Phoenix retrieves and ranks candidates, Grox runs async content-understanding plans, and `candidate-pipeline` defines the shared `CandidatePipeline` trait. The only executable surface in this checkout is the Phoenix JAX inference tree under `phoenix/`. Home Mixer, Thunder, Grox, and the pipeline crate are published as production source without a Cargo/Python workspace that can start those servers.
Do not expect `cargo run` or `python grox/main.py` to start production services from this tree. There is no `Cargo.toml`, and Home Mixer, Thunder, and Grox import unpublished crates and modules (`xai_*`, `grox.service`, `grox.config`, Thunder `args`/`config`/`strato_client`).
## Checkout layout
:::files
x-algorithm/
├── phoenix/ # Runnable JAX retrieval + ranking
│ ├── run_pipeline.py # End-to-end entry (retrieval → rank)
│ ├── run_retrieval.py # Standalone retrieval toy runner
│ ├── run_ranker.py # Standalone ranking toy runner
│ ├── recsys_model.py # PhoenixModelConfig + ranker
│ ├── recsys_retrieval_model.py
│ ├── artifacts/oss-phoenix-artifacts.zip # Git LFS, ~3 GB
│ └── test_recsys_*.py
├── home-mixer/ # For You + Scored Posts gRPC snapshot
│ ├── main.rs # HomeMixerServer CLI
│ ├── server.rs # QueryBuilder + service registration
│ └── candidate_pipeline/ # ForYouCandidatePipeline, PhoenixCandidatePipeline
├── thunder/ # InNetworkPostsService snapshot
│ ├── main.rs
│ └── thunder_service.rs # GetInNetworkPosts
├── grox/ # Content-understanding engine snapshot
│ ├── main.py # Engine + Dispatcher + GrpcServer
│ └── plans/plan_master.py
└── candidate-pipeline/ # Source / Hydrator / Filter / Scorer traits
:::
## Who can run what
| Surface | Path | Runnable here? | What this checkout actually provides |
|---|---|---|---|
| Phoenix inference | `phoenix/run_pipeline.py` | Yes, after LFS extract | Loads published checkpoints + `sports_corpus.npz`, prints a ranked table |
| Phoenix unit tests | `phoenix/test_recsys_*.py` | Yes | Attention-mask and retrieval assertions via `uv run pytest` |
| Phoenix toy runners | `run_ranker.py`, `run_retrieval.py` | Partial | Hardcoded 128-dim toy configs; they do **not** load the LFS archive |
| Home Mixer gRPC | `home-mixer/main.rs` | No | Source for `HomeMixerServer`, `QueryBuilder`, both feed services |
| Thunder gRPC | `thunder/main.rs` | No | Source for `ThunderServiceImpl::get_in_network_posts` |
| Grox server | `grox/main.py` | No | Source for `Engine` / `Dispatcher` / `PlanMaster`; `grox.service` and `grox.config` are unpublished |
| Pipeline framework | `candidate-pipeline/` | No (library snapshot) | Trait contracts only; Home Mixer imports extra unpublished `xai_candidate_pipeline::component_library` |
`phoenix/pyproject.toml` requires Python `>=3.11` and pins `jax==0.8.1` plus `dm-haiku`. Install with `uv sync` in `phoenix/`, or `pip install jax jaxlib dm-haiku numpy`. The zip is a Git LFS pointer until `git lfs pull`; extract to `phoenix/artifacts/oss-phoenix-artifacts/` before inference.
## Production request path
A live For You request does not call Phoenix Python. `HomeMixerServer` builds `PhoenixCandidatePipeline`, wraps it in `ScoredPostsServer`, then wraps that server as `ScoredPostsSource` inside `ForYouCandidatePipeline`.
```mermaid
flowchart TB
subgraph grpc [Home Mixer gRPC]
FY[ForYouFeedService]
SP[ScoredPostsService]
end
subgraph fyPipe [ForYouCandidatePipeline]
QH1[ServedHistory / PastRequestTimestamps]
SRC1[ScoredPostsSource]
SRC2[AdsSource / WhoToFollow / Prompts / PushToHome]
BLEND[BlenderSelector]
end
subgraph phxPipe [PhoenixCandidatePipeline]
QH2[Action sequences, follows, topics, bloom, IP]
TH[ThunderSource]
PX[PhoenixSource / Topics / MoE]
TM[TweetMixerSource]
CACHE[CachedPostsSource]
HYD[Candidate hydrators]
FIL[Pre-score filters]
SC[PhoenixScorer → RankingScorer → VMRanker]
TOPK[TopKScoreSelector]
PSF[VF + conversation dedup]
end
subgraph unpublished [Not startable from this checkout]
THSVC[Thunder InNetworkPostsService]
PXCLUS[Phoenix retrieval clusters]
GROX[Grox Engine / PlanMaster]
end
FY --> QH1 --> SRC1
QH1 --> SRC2 --> BLEND
SRC1 --> SP
SP --> QH2 --> TH & PX & TM & CACHE
TH & PX & TM & CACHE --> HYD --> FIL --> SC --> TOPK --> PSF
TH -.-> THSVC
PX -.-> PXCLUS
```
`ForYouCandidatePipeline` has empty hydrator, filter, and scorer slices. Organic ranking happens inside `PhoenixCandidatePipeline`; For You only hydrates served-history context, fans out extra item types, and blends with `BlenderSelector`.
`CandidatePipeline::execute` is the shared stage order:
1. Query hydrators, then dependent query hydrators
2. Sources (parallel)
3. Candidate hydrators (length and order must match; drops are illegal)
4. Pre-score filters
5. Scorers
6. Selector
7. Post-selection hydrators and filters
8. Truncate to `result_size`
9. `finalize`, then fire-and-forget side effects
`PipelineResult::empty()` short-circuits `TEST_USER_IDS` without running those stages.
## Public gRPC and CLI surfaces
These identifiers exist in source. Binding them requires the unpublished proto crates and production clients.
### Home Mixer
`HomeMixerServer` registers two services with gzip/zstd and `params::MAX_GRPC_MESSAGE_SIZE`:
| Service | RPC | Input | Output |
|---|---|---|---|
| `ScoredPostsService` | `GetScoredPosts` | `ScoredPostsQuery` | `ScoredPostsResponse` (`ScoredPost` list) |
| `ScoredPostsService` | `GetDebugScoredPosts` | `DebugScoredPostsQuery` | scored posts + `debug_json` |
| `ForYouFeedService` | `GetForYouFeed` | `ForYouFeedQuery` (must include `query`) | `ForYouFeedResponse` (`FeedItem` list) |
| `ForYouFeedService` | `GetForYouFeedUrt` | `ForYouFeedQuery` | `ForYouFeedUrtResponse` (serialized URT) |
`QueryBuilder::build` rejects `viewer_id == 0` with `INVALID_ARGUMENT: viewer_id must be specified`. `in_network_only` is true when the proto flag is set **or** Gizmoduck `allow_for_you_recommendations == Some(false)`. Both `ScoredPostsServer::run_pipeline` and `ForYouFeedServer::get_for_you_feed` return empty payloads when `params::TEST_USER_IDS` contains the viewer.
`home-mixer/main.rs` CLI:
gRPC listen port.Metrics port.Shard ordinal. Negative disables `ShardCoordinate`.Shard ring size when `shard_coordinate >= 0`.Passed into prod clients and feature-switch recipient.OpenTelemetry endpoint.
### Thunder
`ThunderServiceImpl` implements `InNetworkPostsService.GetInNetworkPosts`. Request fields used in this snapshot: `user_id`, `following_user_ids`, `exclude_tweet_ids`, `max_results`, `is_video_request`, `debug`. An empty following list is fetched from Strato only when `debug` is set. At semaphore capacity the RPC returns `RESOURCE_EXHAUSTED` (`"Server at capacity, please retry"`). `thunder/lib.rs` also declares `args`, `config`, `metrics`, `o2`, `schema`, and `strato_client`, which are not present in the tree.
### Grox
`grox/main.py` starts `Engine`, `Dispatcher`, and `GrpcServer` from a shared schedule context. `Engine` fans every `TaskPayload` through `PlanMaster.ALL_PLANS` (banger, post safety, spam, embeddings, reply ranking, PTOS). Startup imports `grox.service` and `grox.config.config`, which are not in this checkout.
## Local Phoenix inference
The published path is retrieve-then-rank against exported artifacts, not Home Mixer.
`phoenix/artifacts/oss-phoenix-artifacts.zip` is a Git LFS object (`.gitattributes` tracks `*.zip` and `*.npz`). After `git lfs pull`:
```bash
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
That must produce `artifacts/oss-phoenix-artifacts/{retrieval,ranker,sports_corpus.npz,example_sequence.json}`. The script default `--artifacts_dir ./artifacts` does **not** match the extract layout; pass the nested directory.
```bash uv
cd phoenix
uv sync
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
```bash pip
cd phoenix
pip install jax jaxlib dm-haiku numpy
python run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
Success is a `PIPELINE RESULTS` table, not a gRPC response. Columns: `Rank`, `Score`, `Ret`, `Fav`, `Reply`, `RT`, `Dwell`, `VQV`, `Topics`, Post URL (`https://x.com/a/status/{post_id}`).
Directory that contains `retrieval/`, `ranker/`, and the corpus/sequence files.User history JSON. Default: `{artifacts_dir}/example_sequence.json`.Corpus NPZ. Default: `{artifacts_dir}/sports_corpus.npz`.Dot-product cutoff against the corpus.Rows printed after ranking.
Local ranking uses a demo weighted sum, not Home Mixer's `RankingScorer` feature-switch table:
```text
score = 1.0·P(fav) + 0.5·P(reply) + 0.3·P(rt) + 0.2·P(dwell)
```
Action indices in that sum: `IDX_FAV=1`, `IDX_REPLY=4`, `IDX_QUOTE=5`, `IDX_RT=6`, `IDX_DWELL=11`, `IDX_VQV=13`. Production `PhoenixCandidatePipeline` scorers are `PhoenixScorer` (cluster + egress fallback), `RankingScorer`, then `VMRanker`. `WeightedScorer`, `AuthorDiversityScorer`, and `OONScorer` exist as files but are not in `home-mixer/scorers/mod.rs` and are not wired into `build_with_clients`.
The Phoenix README documents the published mini checkpoint as 128-dim, 4 layers, 4 heads, `history_seq_len=127`, `candidate_seq_len=64`, 1M user/item/author vocabs, 2 hashes per entity, 19 actions. Live weights come from each `config.json` after extract. The sports corpus is a frozen ~537K-post Sports-topic slice, not the production ANN index.
Tests (no LFS required):
```bash
cd phoenix
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```
## Candidate sources in this snapshot
**`PhoenixCandidatePipeline`** (`PostCandidate`): `ThunderSource`, `TweetMixerSource`, `PhoenixSource`, `PhoenixTopicsSource`, `PhoenixMOESource`, `CachedPostsSource`. Thunder is in-network (followed-author `GetInNetworkPosts`). Phoenix sources are out-of-network retrieval.
**`ForYouCandidatePipeline`** (`FeedItem`): `ScoredPostsSource` (in-process `ScoredPostsServer::run_pipeline`), `AdsSource`, `WhoToFollowSource`, `PromptsSource`, `PushToHomeSource`.
## Design constraints that leak into every later page
- **Candidate isolation.** Ranker attention lets each candidate see user + history + self, never other candidates. Scores are batch-independent.
- **Hash embeddings.** User/item/author IDs go through linear-congruential hashes into pad-offset unified tables (`pad=65`) before the transformer.
- **Hydrator contract.** Returned vectors must keep input length and order. A length mismatch is skipped and replaced with errors; use a `Filter` to drop.
- **Apache 2.0.** See `LICENSE`. Transformer blocks are adapted from the Grok-1 open-source release.
## Next
Python 3.11+, uv or pip, Git LFS, and the extract layout required before inference.
Extract `oss-phoenix-artifacts`, run `run_pipeline.py`, and recognize the ranked table.
What this checkout can execute versus Home Mixer, Thunder, and Grox snapshots.
`CandidatePipeline.execute` stages and how For You wraps Phoenix.
---
## 02. Installation
> Python 3.11+, uv or pip, Git LFS for the Phoenix artifact archive, and the extract layout required before inference.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/02-installation.md
- Generated: 2026-08-13T19:41:30.337Z
### Source Files
- `phoenix/pyproject.toml`
- `phoenix/README.md`
- `.gitattributes`
- `phoenix/uv.lock`
- `phoenix/run_pipeline.py`
- `phoenix/artifacts/oss-phoenix-artifacts.zip`
---
title: "Installation"
description: "Python 3.11+, uv or pip, Git LFS for the Phoenix artifact archive, and the extract layout required before inference."
---
The only complete local install surface in this checkout is the Phoenix Python project under `phoenix/`. `phoenix/pyproject.toml` requires Python 3.11+, names the project `grok-1` `0.1.0`, and pins `jax==0.8.1` plus `dm-haiku>=0.0.13`, `numpy>=1.26.4`, and `pyright>=1.1.408`. `phoenix/uv.lock` resolves those packages for `sys_platform == 'darwin'` and `sys_platform == 'linux'` only. End-to-end inference additionally needs the Git LFS object `phoenix/artifacts/oss-phoenix-artifacts.zip` (2,903,518,802 bytes) extracted so `run_pipeline.py` can open `retrieval/config.json`, `ranker/config.json`, the matching `*.npz` checkpoints, `sports_corpus.npz`, and `example_sequence.json`.
A clone that skipped Git LFS leaves `phoenix/artifacts/oss-phoenix-artifacts.zip` as a 135-byte pointer (`version https://git-lfs.github.com/spec/v1`). `unzip` and `run_pipeline.py` both fail until `git lfs pull` replaces that pointer with the real archive.
## What this checkout installs
| Surface | Installable from this tree? | What you get |
|---|---|---|
| `phoenix/` | Yes | JAX/Haiku retrieval + ranking, `run_pipeline.py`, unit tests |
| `phoenix/run_ranker.py`, `phoenix/run_retrieval.py` | Yes, after `uv sync` | Random-init demos; no checkpoint archive required |
| `phoenix/run_pipeline.py` | Yes, after LFS + extract | Frozen mini Phoenix checkpoint on `sports_corpus.npz` |
| `home-mixer/`, `thunder/`, `candidate-pipeline/` | No | Rust snapshots with no `Cargo.toml`; they import unpublished `xai_*` crates |
| `grox/` | No | Python snapshot with no `pyproject.toml`; `grox.main` imports missing `grox.service` and `grox.config` |
`[tool.uv] environments` in `phoenix/pyproject.toml` lists only Darwin and Linux. The locked `jaxlib==0.8.1` wheels are CPU builds for `macosx_11_0_arm64`, `manylinux_2_27_aarch64`, and `manylinux_2_27_x86_64`. There is no CUDA extra, no Windows environment, and no `macosx` x86_64 `jaxlib` wheel in the lockfile.
## Prerequisites
| Requirement | Constraint |
|---|---|
| Python | `>=3.11` (`requires-python` in `phoenix/pyproject.toml` and `phoenix/uv.lock`) |
| OS | macOS (Apple Silicon for the locked JAX stack) or Linux (`aarch64` or `x86_64`) |
| Package manager | [uv](https://docs.astral.sh/uv/getting-started/installation/) (preferred; uses `uv.lock`) or `pip` |
| Git LFS | Tracks `*.zip` and `*.npz` via `.gitattributes` |
| Disk | ~2.90 GB for the LFS zip; plan ~6 GB if you keep both the zip and the extract |
| Working directory | All Phoenix commands below run from `phoenix/` |
Tests do **not** need the artifact archive. They construct dummy `RecsysBatch` / embedding tensors in process.
## Clone and fetch LFS artifacts
```bash
git lfs install
git clone https://github.com/xai-org/x-algorithm.git
cd x-algorithm
```
`.gitattributes` marks `*.zip` and `*.npz` as LFS. The published object is `phoenix/artifacts/oss-phoenix-artifacts.zip`. Individual `*.npz` checkpoints live inside that zip, not as sibling LFS files.
```bash
git lfs pull
# or, scoped:
git lfs pull --include="phoenix/artifacts/oss-phoenix-artifacts.zip"
```
Verify the object, not the pointer:
```bash
wc -c phoenix/artifacts/oss-phoenix-artifacts.zip
# expect 2903518802
file phoenix/artifacts/oss-phoenix-artifacts.zip
# expect a Zip archive, not "ASCII text"
```
A pointer file starts with:
```text
version https://git-lfs.github.com/spec/v1
oid sha256:fbc6017d00588754e22e0c7eb2f786a008a74d309c03c8085fa2fad418a83dac
size 2903518802
```
## Python environment
Install from `phoenix/`. The lockfile package name is `grok-1`; `uv sync` creates the project environment there.
```bash
cd phoenix
uv sync
```
`uv sync` installs the locked runtime set and the default `dev` group (`pytest`). Runtime pins from `phoenix/uv.lock`:
| Package | Locked version | Role |
|---|---|---|
| `jax` | `0.8.1` | Exact pin in `pyproject.toml` |
| `jaxlib` | `0.8.1` | Transitive CPU wheels |
| `dm-haiku` | `0.0.16` | `>=0.0.13` |
| `numpy` | `2.4.1` | `>=1.26.4` |
| `pyright` | `1.1.408` | Runtime dependency, not only a type-check extra |
| `pytest` | `9.0.2` | `dev` group |
Skip tests with `uv sync --no-dev` if you only need inference.
```bash
cd phoenix
python3.11 -m venv .venv
source .venv/bin/activate
pip install 'jax==0.8.1' jaxlib 'dm-haiku>=0.0.13' 'numpy>=1.26.4'
pip install pytest # only if you will run tests
```
The Phoenix README's unpinned `pip install jax jaxlib dm-haiku numpy` can resolve a newer JAX than `0.8.1`. Match the `pyproject.toml` pin. `pyright` is listed as a project dependency but is not required to import the model modules or run `run_pipeline.py`.
Import check (no artifacts required):
```bash
cd phoenix
uv run python -c "import jax, haiku as hk, numpy as np; print(jax.__version__, np.__version__)"
```
Expect `0.8.1` for JAX when using the lockfile.
## Extract the Phoenix archive
`run_pipeline.py` does not read the zip. It opens files under `--artifacts_dir`.
```bash
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
That command creates `artifacts/oss-phoenix-artifacts/`. Point `--artifacts_dir` at **that** directory, not at `artifacts/`.
`run_pipeline.py` defaults `--artifacts_dir` to `./artifacts`. After `unzip … -d artifacts/`, `./artifacts/retrieval/config.json` does not exist. The documented invocation is `--artifacts_dir artifacts/oss-phoenix-artifacts`.
### Expected extract layout
`run_pipeline.py` requires this tree as the value of `--artifacts_dir`:
:::files
oss-phoenix-artifacts/
retrieval/
model_params.npz # retrieval transformer + candidate tower
embedding_tables.npz # user / item / author hash tables
config.json # architecture + hash_params
ranker/
model_params.npz # ranking transformer + action head
embedding_tables.npz # user / item / author hash tables
config.json # architecture + hash_params
sports_corpus.npz # post_ids, candidate_representations, author_ids, optional topics
example_sequence.json # user_id + history[]
:::
Phoenix documents those embedding tables at about 1.4 GB each and the model param files at about 3 MB each. `run_pipeline.py` reconstructs a pad-offset unified table (`pad=65` plus `user_vocab_size` + `item_vocab_size` + `author_vocab_size`) from `user_embeddings`, `item_embeddings`, and `author_embeddings` inside each `embedding_tables.npz`.
Architecture values are **not** hardcoded for the published checkpoint. `build_model_config()` reads `emb_size`, `history_seq_len`, `candidate_seq_len`, `num_heads`, `num_layers`, `num_actions`, vocab sizes, and `hash_params` from each `config.json`.
## Verify the install
```bash
cd phoenix
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```
`test_recsys_model.py` covers `make_recsys_attn_mask` and related helpers. `test_recsys_retrieval_model.py` is a `unittest` module that pytest collects; it builds tiny random models (`emb_size=64`, `num_layers=1`) via `create_example_batch` / `create_example_corpus`. Neither test opens `artifacts/`.
```bash
test -f artifacts/oss-phoenix-artifacts/retrieval/config.json \
&& test -f artifacts/oss-phoenix-artifacts/ranker/config.json \
&& test -f artifacts/oss-phoenix-artifacts/sports_corpus.npz \
&& test -f artifacts/oss-phoenix-artifacts/example_sequence.json \
&& echo ok
```
```bash
cd phoenix
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
Success is a `PIPELINE RESULTS — User ` table with columns `Rank`, `Score`, `Ret`, `Fav`, `Reply`, `RT`, `Dwell`, `VQV`, `Topics`, and a post URL, plus a final `Weighted score range: [low, high]` line. Defaults retrieve 200 corpus rows (`--top_k_retrieval`) and print 30 (`--top_k_display`).
Random-init scripts that do **not** need the archive, once the venv exists:
```bash
cd phoenix
uv run run_retrieval.py
uv run run_ranker.py
```
Those scripts hardcode a tiny demo transformer (`emb_size=128`, `num_layers=2`, `history_seq_len=32`, `candidate_seq_len=8`) and call `initialize()` on synthetic batches.
## `run_pipeline.py` paths and flags
Directory that contains `retrieval/`, `ranker/`, `sports_corpus.npz`, and `example_sequence.json`. After the documented unzip, pass `artifacts/oss-phoenix-artifacts`.
User history JSON. Default: `/example_sequence.json`. Must include `user_id` and `history[]` with `post_id`, `author_id`, and `actions`.
Corpus NPZ. Default: `/sports_corpus.npz`. Required keys: `post_ids`, `candidate_representations`, `author_ids`. Optional: `topics`.
How many corpus rows to keep after the retrieval dot product, capped at corpus length.
How many ranked rows to print.
```bash
cd phoenix
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file artifacts/oss-phoenix-artifacts/example_sequence.json \
--corpus_file artifacts/oss-phoenix-artifacts/sports_corpus.npz \
--top_k_retrieval 200 \
--top_k_display 30
```
```text
========================================================================================================================
PIPELINE RESULTS — User
History: 3 items | Corpus: posts
Retrieved top 200 → Ranked by engagement model
========================================================================================================================
Rank Score Ret Fav Reply RT Dwell VQV Topics Post URL
------------------------------------------------------------------------------------------------------------------------
1 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 … https://x.com/a/status/
…
Weighted score range: [low, high]
========================================================================================================================
```
`load_model_params()` reads each `model_params.npz` into a Haiku params dict keyed by `/`-separated module paths. `load_embedding_table()` returns the raw NPZ mapping.
## What does not install
Home Mixer (`home-mixer/main.rs`) starts through `XServiceBuilder` and unpublished crates (`xai_home_mixer`, `xai_home_mixer_proto`, `xai_x_rpc`, `xai_x_service_builder`, `xai_dark_traffic`, `xai_stringcenter`). Thunder (`thunder/main.rs`) needs `xai_http_server` plus Kafka/Strato wiring. There is no `Cargo.toml` or `rust-toolchain` in this tree.
Grox (`grox/main.py`) calls `Engine`, `Dispatcher`, and `GrpcServer`, but `grox.service` and `grox.config` are not in the snapshot, and there is no Grox package manifest. Treat those directories as source reference, not a second install path.
## Common failures
| Symptom | Cause | Fix |
|---|---|---|
| `unzip` reports not a zip / empty archive | LFS pointer still on disk (135 bytes, ASCII) | `git lfs install && git lfs pull`, then re-check `wc -c` |
| `FileNotFoundError` for `retrieval/config.json` or `ranker/config.json` | `--artifacts_dir` is `./artifacts` after `unzip -d artifacts/` | Pass `--artifacts_dir artifacts/oss-phoenix-artifacts` |
| `FileNotFoundError` for `sports_corpus.npz` / `example_sequence.json` | Incomplete extract, or overrides point elsewhere | Confirm the four required leaves listed above |
| `uv sync` refuses the platform | Windows, or a Darwin/Linux combo outside `[tool.uv] environments` | Use macOS arm64 or Linux; do not expect a locked Windows env |
| `jaxlib` wheel missing on Intel macOS | Lockfile ships `macosx_11_0_arm64` only for Darwin | Use Apple Silicon or Linux |
| Tests pass, pipeline missing | Tests never load LFS objects | Extract the archive before `run_pipeline.py` |
| Want Home Mixer / Thunder / Grox locally | No crate/package manifests; unpublished modules | See runtime boundaries; this checkout cannot build those services |
`phoenix/.gitignore` only ignores `__pycache__/`. Extracted `artifacts/oss-phoenix-artifacts/` is untracked local data sitting next to the LFS zip.
## Next
Extract the archive, run `run_pipeline.py`, and treat the ranked table as the success signal.
How retrieval and ranking load checkpoints, encode `example_sequence.json`, and score `sports_corpus.npz`.
`uv run pytest` targets and the attention-mask / retrieval assertions they encode.
What this checkout can execute versus Home Mixer, Thunder, and Grox snapshots.
LFS pointer failures, wrong `--artifacts_dir`, and other setup errors in more detail.
---
## 03. Quickstart
> Extract oss-phoenix-artifacts, run run_pipeline.py, and recognize the ranked table as the success signal.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/03-quickstart.md
- Generated: 2026-08-13T19:41:41.482Z
### Source Files
- `phoenix/README.md`
- `phoenix/run_pipeline.py`
- `phoenix/pyproject.toml`
- `.gitattributes`
- `phoenix/runners.py`
---
title: "Quickstart"
description: "Extract oss-phoenix-artifacts, run run_pipeline.py, and recognize the ranked table as the success signal."
---
`phoenix/run_pipeline.py` is the local end-to-end entry point in this checkout: it loads the published retrieval and ranker checkpoints, encodes `example_sequence.json`, retrieves from `sports_corpus.npz`, scores the top-K posts, and prints a ranked engagement table. That table is the success signal. Home Mixer, Thunder, and Grox are not part of this run.
This page assumes a Python 3.11+ environment on Darwin or Linux. The `phoenix/uv.lock` environments list only `sys_platform == 'darwin'` and `sys_platform == 'linux'`. For LFS, `uv`/`pip`, and the extract layout in more detail, see [Installation](/installation).
## What this run executes
`run_pipeline.py` composes the two Phoenix stages that production Home Mixer also uses, against a frozen sports corpus rather than live Thunder or cluster retrieval:
```text
example_sequence.json retrieval/ sports_corpus.npz
user_id + history + config.json post_ids
model_params.npz author_ids
embedding_tables.npz candidate_representations
│ │
▼ │
user representation ── dot product ──────┘
│
▼
top-K post / author IDs
│
▼
ranker/ ──► sigmoid(logits) ──► demo weighted Score
│
▼
PIPELINE RESULTS table
```
`run_ranker.py` and `run_retrieval.py` are leftover dummy-data scripts. They initialize random batches and do not load `oss-phoenix-artifacts`. Do not treat their output as a successful artifact run.
## Prerequisites
- Git LFS installed, and `phoenix/artifacts/oss-phoenix-artifacts.zip` materialized (not a 135-byte pointer).
- Working directory `phoenix/`.
- Python `>=3.11` with either `uv` or `pip`.
- Declared runtime deps from `phoenix/pyproject.toml`: `jax==0.8.1`, `dm-haiku>=0.0.13`, `numpy>=1.26.4`.
Confirm the archive is the real object before unzipping. The Git LFS pointer records size `2903518802` bytes (~2.70 GiB):
```bash
wc -c artifacts/oss-phoenix-artifacts.zip
# expected: 2903518802
```
If the file is ~135 bytes and starts with `version https://git-lfs.github.com/spec/v1`, run `git lfs pull` from the repository root. `.gitattributes` routes `*.zip` and `*.npz` through LFS.
## Extract the artifacts
```bash
cd phoenix
```
```bash
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
The zip’s top-level directory is `oss-phoenix-artifacts/`, so this creates `artifacts/oss-phoenix-artifacts/`. That nested directory is the `--artifacts_dir` you pass to the script.
`run_pipeline.py` opens these paths under `--artifacts_dir` (defaults shown in the next section). Missing any of them raises `FileNotFoundError` or `np.load` failure immediately.
:::files
phoenix/artifacts/oss-phoenix-artifacts/
retrieval/
config.json
model_params.npz
embedding_tables.npz
ranker/
config.json
model_params.npz
embedding_tables.npz
sports_corpus.npz
example_sequence.json
:::
The script default `--artifacts_dir` is `./artifacts`, not `./artifacts/oss-phoenix-artifacts`. After the documented unzip, you must pass `--artifacts_dir artifacts/oss-phoenix-artifacts`. Pointing at `./artifacts` looks for `artifacts/retrieval/config.json`, which does not exist.
The Phoenix README documents the packaged sample as a frozen mini checkpoint plus a Sports-topic corpus and a three-post user history (NFL, NBA, NHL). `run_pipeline.py` does not hardcode those sizes; it reads `emb_size`, `num_actions`, `history_seq_len`, `candidate_seq_len`, hash parameters, and transformer fields from each `config.json`.
## Install and run
```bash
cd phoenix
uv sync
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
```bash
cd phoenix
pip install "jax==0.8.1" "dm-haiku>=0.0.13" "numpy>=1.26.4"
python run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
JAX is pinned to `0.8.1` in `pyproject.toml`. The first run compiles Haiku transforms for retrieval and ranking; later runs in the same process reuse those transforms.
### CLI flags
Directory that contains `retrieval/`, `ranker/`, and (unless overridden) `example_sequence.json` and `sports_corpus.npz`. After the documented unzip, pass `artifacts/oss-phoenix-artifacts`.
User action history JSON. Default: `/example_sequence.json`.
Corpus NPZ. Default: `/sports_corpus.npz`.
How many corpus posts to keep after the user-representation dot product. Capped at corpus length.
How many ranked rows to print. Capped at the retrieval K.
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file artifacts/oss-phoenix-artifacts/example_sequence.json \
--corpus_file artifacts/oss-phoenix-artifacts/sports_corpus.npz \
--top_k_retrieval 200 \
--top_k_display 30
```
## Success signal
A successful run logs load/retrieve/rank progress, then prints a 120-column table whose header is `PIPELINE RESULTS — User {user_id}`. That header, the per-row `https://x.com/a/status/{post_id}` URLs, and the closing `Weighted score range:` line are the verification signal.
Progress logs (INFO) include:
| Log | Meaning |
| --- | --- |
| `Loading retrieval model...` | `retrieval/model_params.npz` + unified hash table |
| `Loading ranker model...` | `ranker/model_params.npz` + unified hash table |
| `Loading corpus...` then `N posts, repr shape ...` | `sports_corpus.npz` loaded |
| `Loading user sequence from ...` then `User {id}, {n} history items` | JSON parsed |
| `Running retrieval...` then `User repr norm=...` | user tower applied |
| `Retrieved {K} (score range: low - high)` | corpus dot-product top-K |
| `Ranking {K} candidates...` | ranker batches of `candidate_seq_len` |
```text
========================================================================================================================
PIPELINE RESULTS — User
History: items | Corpus: posts
Retrieved top → Ranked by engagement model
========================================================================================================================
Rank Score Ret Fav Reply RT Dwell VQV Topics Post URL
------------------------------------------------------------------------------------------------------------------------
1 https://x.com/a/status/
...
Weighted score range: [, ]
========================================================================================================================
```
The run succeeded when stdout contains `PIPELINE RESULTS — User`, `DISPLAY` ranked rows (`min(top_k_display, K)`), post URLs of the form `https://x.com/a/status/{id}`, and a `Weighted score range:` line. `uv run pytest` in `phoenix/` does **not** replace this signal: those tests use synthetic tensors and never open the artifact zip.
### Printed columns
| Column | Source |
| --- | --- |
| `Rank` | `1 .. DISPLAY` after `argsort(-weighted)` |
| `Score` | demo weighted sum (below) |
| `Ret` | retrieval dot-product score for that corpus row |
| `Fav` | `sigmoid(logits)[:, 1]` (`IDX_FAV`, `SERVER_TWEET_FAV`) |
| `Reply` | index `4` (`SERVER_TWEET_REPLY`) |
| `RT` | index `6` (`SERVER_TWEET_RETWEET`) |
| `Dwell` | index `11` (`CLIENT_TWEET_RECAP_DWELLED`) |
| `VQV` | index `13` (`CLIENT_TWEET_VIDEO_QUALITY_VIEW`) |
| `Topics` | `sports_corpus.npz` `topics` (empty string if the key is absent), truncated to 28 characters |
| `Post URL` | `https://x.com/a/status/{post_id}` |
`IDX_QUOTE = 5` is defined in `run_pipeline.py` but is not printed and is not part of the demo weighted sum.
## Inputs the script actually reads
### `example_sequence.json`
Required shape consumed by `run_pipeline.py`:
```json
{
"user_id": 123,
"history": [
{
"post_id": 1,
"author_id": 2,
"actions": { "1": 1.0, "11": 1.0 }
}
]
}
```
Hashed with the retrieval and ranker `hash_params` independently.
Copied into a zero-padded vector of length `history_seq_len`. Extra items beyond that length are dropped (`history[:hist_len]`).
Same padding and truncation as `post_id`.
String keys are `int`-parsed ActionName indices. Values are written into `history_actions[i, idx]` when `idx < num_actions`. Unknown or out-of-range indices are ignored.
Action indices used by the demo (`1`, `4`, `5`, `6`, `11`, `13`) follow the proto `ActionName` enum, not the 0-based order of `runners.ACTIONS`. Changing the sample history is covered on [Customize a user sequence](/customize-user-sequence).
### `sports_corpus.npz`
| Array | Role |
| --- | --- |
| `post_ids` | Candidate tweet IDs |
| `author_ids` | Author IDs aligned with `post_ids` |
| `candidate_representations` | Precomputed retrieval vectors; `scores = corpus_repr @ user_repr[0]` |
| `topics` | Optional; defaults to `""` per row if missing |
### Checkpoints
`runners.load_model_params` reads each `model_params.npz` as nested Haiku params (`key.split("/")` → module path + leaf). `load_embedding_table` expects `user_embeddings`, `item_embeddings`, and `author_embeddings`. `build_unified_emb_table` concatenates those tables behind a pad offset of `65`, then user, item, and author vocab blocks.
Retrieval sets `PhoenixRetrievalModelConfig.enable_linear_proj = True`. Ranking uses `PhoenixModelConfig` with `num_actions` and `post_age_granularity_mins` from `ranker/config.json` (granularity default `60` if omitted). Transformer `widening_factor` is hardcoded to `2.0` and `attn_output_multiplier` to `0.125`.
## Demo score versus production
The printed `Score` is a four-term demo combination, not Home Mixer’s `WeightedScorer`:
```python
weighted = (
all_probs[:, IDX_FAV] * 1.0
+ all_probs[:, IDX_REPLY] * 0.5
+ all_probs[:, IDX_RT] * 0.3
+ all_probs[:, IDX_DWELL] * 0.2
)
```
Production `WeightedScorer` sums many more Phoenix heads (including quote, click, VQV-when-eligible, share, follow, and negative-feedback terms) and then offsets/normalizes. VQV is displayed in the quickstart table but is **not** in this demo sum. See [Multi-action scoring](/multi-action-scoring) and [Scorers and weights](/scorers-and-weights).
Ranking runs in chunks of `candidate_seq_len` (from `ranker/config.json`), pads a short final chunk with zeros, then slices logits back to the real candidate count before concatenating.
## Common failures
`artifacts/oss-phoenix-artifacts.zip` is 135 bytes and `unzip` fails or extracts nothing useful. `.gitattributes` marks `*.zip` and `*.npz` as LFS. Install Git LFS, then `git lfs pull`. Expected size is `2903518802` bytes.
`--artifacts_dir` points at `phoenix/artifacts` (the default) instead of `phoenix/artifacts/oss-phoenix-artifacts`, or the unzip landed in a different directory. Confirm `retrieval/config.json`, `ranker/config.json`, `sports_corpus.npz`, and `example_sequence.json` all sit directly under the directory you pass.
`--corpus_file` and `--sequence_file` default to those names inside `--artifacts_dir`. If you flattened the zip or renamed files, pass the flags explicitly.
`build_unified_emb_table` requires `user_embeddings`, `item_embeddings`, and `author_embeddings`. Hashing requires `hash_params` plus `user_vocab_size`, `item_vocab_size`, and `author_vocab_size`. Use the published archive; do not substitute the dummy tensors from `run_ranker.py` / `run_retrieval.py`.
More LFS, `viewer_id`, Thunder, and Grox failures live on [Troubleshooting](/troubleshooting). This checkout cannot stand in for production Home Mixer; see [Runtime boundaries](/runtime-boundaries).
## Next
Python 3.11+, uv or pip, Git LFS, and the extract layout required before this command.
How checkpoints, `example_sequence.json`, and `sports_corpus.npz` are encoded, retrieved, and ranked.
`example_sequence.json` fields, ActionName indices, history padding, and the `--top_k_*` knobs.
`uv run pytest` targets for attention-mask and retrieval assertions (not the artifact table).
LFS pointer files, wrong `--artifacts_dir`, and other checkout failures.
What this checkout can execute locally versus unpublished Home Mixer, Thunder, and Grox surfaces.
---
## 04. Runtime boundaries
> What this checkout can execute locally versus production Home Mixer, Thunder, and Grox snapshots that depend on unpublished crates.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/04-runtime-boundaries.md
- Generated: 2026-08-13T19:41:42.682Z
### Source Files
- `README.md`
- `phoenix/pyproject.toml`
- `home-mixer/main.rs`
- `thunder/lib.rs`
- `grox/main.py`
- `candidate-pipeline/lib.rs`
- `LICENSE`
---
title: "Runtime boundaries"
description: "What this checkout can execute locally versus production Home Mixer, Thunder, and Grox snapshots that depend on unpublished crates."
---
This checkout is not a buildable For You stack. The only package with a lockfile, installable third-party dependencies, and an entry point that runs offline is `phoenix/` (`pyproject.toml` project name `grok-1`, Python `>=3.11`, `jax==0.8.1`). `home-mixer/`, `thunder/`, `candidate-pipeline/`, and `grox/` are production source snapshots: there is no `Cargo.toml` anywhere in the tree, Grox has no package manifest, and those trees import unpublished `xai_*` crates plus modules that are not present in this repository.
Do not expect `cargo run`, `python grox/main.py`, or a local Home Mixer / Thunder gRPC server to start from this clone. Those binaries need private crates, omitted modules (`crate::params`, `crate::clients`, `thunder::args`, `grox.config`, `grox.service`), and production backends (Kafka, Strato, Gizmoduck, Phoenix inference clusters).
## What runs vs what you can only read
| Surface | Tree | Local execution | What this checkout actually is |
|---|---|---|---|
| Phoenix inference | `phoenix/` | Yes, after Git LFS + unzip | JAX retrieval + ranker with published checkpoints |
| Phoenix unit tests | `phoenix/test_recsys_model.py`, `phoenix/test_recsys_retrieval_model.py` | Yes, no artifacts required | Attention-mask, RoPE, retrieval-tower, and runner assertions |
| Candidate pipeline traits | `candidate-pipeline/` | No | Trait snapshot of `xai_candidate_pipeline` without crate metadata or `component_library` |
| Home Mixer | `home-mixer/` | No | `HomeMixerServer` source; `XService` + proto servers, unpublished clients |
| Thunder | `thunder/` | No | `InNetworkPostsService` + `PostStore` source; missing `args` / `config` / `strato_client` |
| Grox | `grox/` | No | `Engine` / `Dispatcher` / `PlanMaster` source; missing `grox.config`, `grox.service`, `grox.lm` |
```mermaid
flowchart TB
subgraph local ["Locally executable"]
PX["phoenix/\nrun_pipeline.py / pytest"]
LFS["artifacts/oss-phoenix-artifacts.zip\nGit LFS, 2903518802 bytes"]
PX --> LFS
end
subgraph snapshots ["Source snapshots in this checkout"]
HM["home-mixer/\nHomeMixerServer"]
TH["thunder/\nThunderServiceImpl"]
GX["grox/\nEngine + Dispatcher + PlanMaster"]
CP["candidate-pipeline/\nCandidatePipeline traits"]
end
subgraph unpublished ["Not in this repository"]
CRATES["xai_* crates\nprotos, XServiceBuilder, Kafka, stats"]
HMOMIT["crate::params, crate::clients, crate::util"]
THOMIT["args, config, metrics, o2, schema, strato_client"]
GXOMIT["grox.config, grox.service, grox.lm,\ngrox.prompts, data_types, monitor"]
end
subgraph prod ["Production-only backends"]
KAFKA["Kafka ingest / side effects"]
STRATO["Strato following lists and stores"]
GIZ["Gizmoduck, TES, AdIndex, VM ranker"]
PHXCL["Phoenix prediction cluster / egress sidecar"]
end
HM --> CP
HM --> CRATES
HM --> HMOMIT
HM --> prod
TH --> CRATES
TH --> THOMIT
TH --> KAFKA
TH --> STRATO
GX --> GXOMIT
GX --> KAFKA
GX --> STRATO
```
## Phoenix: the local runtime
`phoenix/` is a self-contained JAX package. `uv.lock` pins Darwin and Linux. `pip` can install the same declared set: `jax`, `jaxlib`, `dm-haiku`, `numpy`. Dev extras add `pytest`.
```bash
cd phoenix
uv sync
```
```bash
cd phoenix
pip install jax jaxlib dm-haiku numpy
```
### Artifact gate
`.gitattributes` routes `*.zip` and `*.npz` through Git LFS. Until `git lfs pull`, `phoenix/artifacts/oss-phoenix-artifacts.zip` is a 135-byte pointer (`oid sha256:fbc6017d00588754e22e0c7eb2f786a008a74d309c03c8085fa2fad418a83dac`, size `2903518802`). `run_pipeline.py` will not load models from that stub.
Pull LFS objects, then extract next to the pointer:
```bash
git lfs pull
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
`run_pipeline.py` opens these paths relative to `--artifacts_dir` (default `./artifacts`):
:::files
artifacts/oss-phoenix-artifacts/
retrieval/
model_params.npz
embedding_tables.npz
config.json
ranker/
model_params.npz
embedding_tables.npz
config.json
sports_corpus.npz
example_sequence.json
:::
```bash
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
Success is a `PIPELINE RESULTS — User ` table with columns `Rank`, `Score`, `Ret`, `Fav`, `Reply`, `RT`, `Dwell`, `VQV`, `Topics`, and `https://x.com/a/status/` URLs, plus a `Weighted score range:` footer. Failure at this step is almost always an LFS pointer, a wrong `--artifacts_dir`, or a missing `retrieval/config.json`.
Directory that contains `retrieval/`, `ranker/`, and (unless overridden) the sequence and corpus files.
User history JSON. Default: `/example_sequence.json`.
Precomputed candidate NPZ (`post_ids`, `candidate_representations`, `author_ids`, optional `topics`). Default: `/sports_corpus.npz`.
Dot-product retrieval depth against the sports corpus.
How many ranked rows to print.
Local ranking is a demo weighted sum, not Home Mixer `WeightedScorer` / `RankingScorer` tables:
```text
score = P(fav)*1.0 + P(reply)*0.5 + P(rt)*0.3 + P(dwell)*0.2
```
`run_ranker.py` and `run_retrieval.py` still exist. They construct synthetic `create_example_batch` / `create_example_corpus` tensors and do **not** load the published archive. The production-shaped local path is `run_pipeline.py`.
### Tests that do not need checkpoints
```bash
cd phoenix
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```
These modules assert `make_recsys_attn_mask` isolation, right-anchored RoPE, post-age buckets, candidate-tower normalization, and retrieval `top_k`. They are the second local success signal after the ranked table.
The published checkpoint is a frozen mini Phoenix snapshot and a sports-only corpus (`sports_corpus.npz`, on the order of 537K posts). Production Phoenix is a larger, continuously trained model scored over a live global corpus through `PhoenixScorer` cluster / egress clients. Matching `config.json` keys is not the same as matching production capacity.
## Home Mixer: source snapshot, not a binary
`home-mixer/main.rs` defines a `HomeMixer gRPC Server` clap surface and boots `XServiceBuilder::new("home-mixer")`. That is production wiring, not something this clone can link.
| Flag | Default in source | Role |
|---|---|---|
| `--grpc_port` | `50051` | gRPC listen port |
| `--metrics_port` | `9090` | Metrics port |
| `--shard_coordinate` | `-1` | `< 0` means no `ShardCoordinate` |
| `--shard_total_size` | `500` | Used only when `shard_coordinate >= 0` |
| `--datacenter` | `atla` | Passed into `XServiceBuilder` and `PhoenixCandidatePipeline::prod` |
| `--otel_endpoint` | `""` | OpenTelemetry endpoint |
`main` also requires unpublished bootstrap files and env:
- `xai_stringcenter::init_from_file(params::STRINGCENTER_BUNDLE_PATH)`
- `.with_featureswitches(params::FS_PATH, true)`
- `.with_decider(params::decider_path(), None)`
- `.with_tls(TlsMode::server_mtls_from_env()?)`
- `RejectDarkTrafficLayer` / `dark_traffic_setup`
- `xai_profiling::profiling_router()`
`HomeMixerServer` implements `xai_x_service_builder::XService`. `build` constructs `ProdGizmoduckClient` (`"home-mixer.prod"`), `PhoenixCandidatePipeline::prod(shard_coordinate, datacenter)`, `ScoredPostsServer`, and `ForYouCandidatePipeline`. `register` attaches:
- `ScoredPostsServiceServer`
- `ForYouFeedServiceServer`
Gzip and Zstd compression are enabled. Message size limits come from `params::MAX_GRPC_MESSAGE_SIZE` (module not in tree).
### Omitted Home Mixer modules
`home-mixer/lib.rs` exports `ads`, `candidate_pipeline`, `models`, `scorers`, `server`. It does **not** declare the modules the rest of the crate uses:
| Missing path | Used for |
|---|---|
| `crate::params` | `TEST_USER_IDS`, `STRINGCENTER_BUNDLE_PATH`, `FS_PATH`, scorer weights, source enable flags, blender positions |
| `crate::clients` | Gizmoduck, TES, Strato, AdIndex, Kafka publishers, VM ranker, served history, S2S cert paths |
| `crate::util` | URT helpers (`for_you_server.rs` imports `crate::util::urt`) |
`PhoenixCandidatePipeline::prod` also reaches `xai_candidate_pipeline::component_library::clients` (Phoenix retrieval / prediction). That `component_library` tree is not in `candidate-pipeline/` here.
### Request gates that exist only in this source
`QueryBuilder::build` rejects `viewer_id == 0` with `Status::invalid_argument("viewer_id must be specified")`.
`ScoredPostsServer::run_pipeline` and `ForYouFeedServer::get_for_you_feed` short-circuit when `params::TEST_USER_IDS` contains the user: empty `scored_posts` / `items` and `PipelineResult::empty()`. You cannot evaluate that set locally because `params` is unpublished.
Phoenix sources (`PhoenixSource`, `PhoenixMOESource`, `PhoenixTopicsSource`) return `"missing retrieval_sequence"` when `ScoredPostsQuery.retrieval_sequence` is `None`. That is production query-hydration failure, not a local CLI error.
## Thunder: in-network store without a crate
`thunder/main.rs` parses `args::Args`, builds `PostStore` + `StratoClient` + `ThunderServiceImpl`, wraps the gRPC service in `xai_http_server::HttpServer`, optionally starts `xai_profiling::spawn_server(3000, ...)`, then calls `kafka_utils::start_kafka`.
`thunder/lib.rs` declares modules that are absent from the snapshot:
| Missing module | Callers |
|---|---|
| `args` | `main.rs`, `kafka_utils.rs` (`Args::parse`, Kafka/SASL fields) |
| `config` | `thunder_service.rs` (`MAX_INPUT_LIST_SIZE`, `MAX_POSTS_TO_RETURN`, `MAX_VIDEOS_TO_RETURN`) |
| `metrics` | `thunder_service.rs` histograms / in-flight gauges |
| `o2` | declared in `lib.rs` only |
| `schema` | declared in `lib.rs` only |
| `strato_client` | `StratoClient::new` / `fetch_following_list` |
Fields read from the missing `Args` type include `post_retention_seconds`, `request_timeout_ms`, `max_concurrent_requests`, `grpc_port`, `http_port`, `enable_profiling`, `kafka_num_threads`, `is_serving`, and Kafka/SASL settings (`security_protocol`, `sasl_*`, `kafka_group_id`, `auto_offset_reset`, `fetch_timeout_ms`, `skip_to_latest`, `in_network_events_consumer_dest`). Defaults are not in this checkout.
Kafka topic constants in `thunder/kafka_utils.rs` are empty strings (`TWEET_EVENT_TOPIC`, `TWEET_EVENT_DEST`, `IN_NETWORK_EVENTS_TOPIC`, `IN_NETWORK_EVENTS_DEST`). SASL password env lookups use `std::env::var("")`. Those are redacted production hooks, not a local broker config.
`GetInNetworkPosts` acquires a semaphore with `try_acquire`. At capacity it returns:
```text
Status::resource_exhausted("Server at capacity, please retry")
```
An empty `following_user_ids` list falls back to `StratoClient::fetch_following_list`. Neither the semaphore path nor Strato can be exercised without the missing modules and unpublished `xai_thunder_proto` / `xai_kafka` / `xai_wily` crates.
## Grox: plan engine without a service package
`grox/main.py` is a production process: `init_proc("main")`, then `Engine`, `Dispatcher`, and `GrpcServer` start, wait on SIGINT/SIGTERM, sleep 300 seconds, then stop. There is no `grox/pyproject.toml` or `requirements.txt`.
Immediate import failures from this tree:
```text
from grox.service import GrpcServer # no grox/service
from grox.config.config import grox_config # no grox/config
```
`Engine` and `Dispatcher` additionally import `monitor.logging`, `monitor.metrics`, `grox.data_loaders.media_processor`, and `grox.data_loaders.data_types`. Classifiers and embedders import `grox.lm.*` and `grox.prompts.template`.
Present in-tree (readable, not independently runnable):
- `PlanMaster.ALL_PLANS`: `PlanInitialBanger`, `PlanPostSafety`, `PlanSpamComment`, `PlanPostEmbeddingWithSummary`, `PlanPostEmbeddingWithSummaryForReply`, `PlanPostEmbeddingV5`, `PlanPostEmbeddingV5ForReply`, `PlanReplyRanking`, `PlanSafetyPtos`
- Per-plan `TASK_DEPENDENCIES` DAGs
- `Dispatcher` stream generators (post, safety, embedding v5, PTOS, recovery)
`python grox/main.py` fails at import before any plan runs.
## Candidate pipeline crate vs this folder
`candidate-pipeline/` is the public trait surface: `Source`, `Hydrator`, `Filter`, `Scorer`, `Selector`, `QueryHydrator`, `SideEffect`, and `CandidatePipeline::execute` (query hydrate → sources → hydrate → filter → score → select → post-select → side effects).
That folder still cannot compile here:
- No `Cargo.toml`
- `PipelineQuery` requires `xai_feature_switches::Params` and `xai_decider::Decider`
- Every stage uses `xai_stats_macro::receive_stats` and `xai_stats_receiver`
Home Mixer does not path-depend on this directory. It imports the unpublished crate `xai_candidate_pipeline`, including `component_library` clients and caches that are **not** in the snapshot (`MokaCache`, `PhoenixPredictionClient`, `PhoenixRetrievalClient`, `SocialGraphClientOps`, `StratoClient`). Treat `candidate-pipeline/` as documentation of the stage contracts, not as a drop-in crate.
## Unpublished `xai_*` crates referenced by the snapshots
These identifiers appear in Rust `use` paths and do not exist in this repository or on a public registry path in-tree:
| Crate | Typical consumer |
|---|---|
| `xai_home_mixer`, `xai_home_mixer_proto` | `home-mixer/main.rs`, servers, models |
| `xai_candidate_pipeline` | Home Mixer pipelines and components |
| `xai_x_service_builder`, `xai_x_rpc`, `xai_dark_traffic` | Home Mixer process bootstrap |
| `xai_stringcenter`, `xai_feature_switches`, `xai_decider` | Params / FS / decider |
| `xai_http_server`, `xai_thunder_proto`, `xai_kafka`, `xai_wily` | Thunder process + ingest |
| `xai_stats_macro`, `xai_stats_receiver`, `xai_profiling` | All Rust snapshots |
| `xai_recsys_proto`, `xai_visibility_filtering`, `xai_strato` | Scoring, VF, stores |
| `xai_safety_label_store`, `xai_manhattan`, `xai_redis_client` | Brand safety, cache, stores |
| `xai_pipeline_tracing`, `xai_urt_thrift`, `xai_x_thrift` | Tracing and response mapping |
| `xai_recsys_aggregation`, `xai_recsys_logging_thrift`, `xai_uas_thrift` | Sequence / logging |
| `xai_ads_injection_proto`, `xai_account_recommendations_mixer_proto`, `xai_prompts_thrift`, `xai_vm_ranker_proto` | Ads, WTF, prompts, VM ranker |
| `xai_twittercontext_proto`, `xai_geo_ip`, `xai_core_entities`, `xai_post_text` | Viewer / geo / entities |
Apache-2.0 in `LICENSE` covers the published files. It does not ship those crates.
## Production backends the snapshots assume
Even with the missing modules restored, the Rust and Grox snapshots are not offline:
| Backend | Snapshot use |
|---|---|
| Kafka | Thunder tweet / in-network ingest; Home Mixer side effects (`publish_seen_ids`, reranking, client events); Grox stream generators |
| Strato | Thunder following-list fallback; Home Mixer hydrators; Grox `TweetStratoLoader` |
| Gizmoduck | `ProdGizmoduckClient` in `HomeMixerServer::build` |
| Phoenix prediction / retrieval gRPC | `PhoenixScorer` cluster + egress; `PhoenixSource` / MoE / topics |
| Feature switches + decider + String Center | `XServiceBuilder` and every `query.params.get(...)` |
| mTLS / S2S certs | `TlsMode::server_mtls_from_env`, `S2S_CHAIN_PATH` / `S2S_CRT_PATH` / `S2S_KEY_PATH` |
| TES, AdIndex, VM ranker, served-history, Redis | Candidate hydration, ads, ranking, cache side effects |
Local Phoenix replaces that entire serving path with `sports_corpus.npz` dot products and in-process Haiku `apply`.
## Failure signals by surface
| Symptom | Meaning |
|---|---|
| `oss-phoenix-artifacts.zip` is ~135 bytes of `version https://git-lfs.github.com/spec/v1` | LFS pointer not pulled |
| `FileNotFoundError` for `retrieval/config.json` | Archive not extracted, or `--artifacts_dir` points at the zip parent without `oss-phoenix-artifacts/` |
| Ranked table prints | Local Phoenix pipeline succeeded |
| `uv run pytest …` green | Mask / retrieval unit tests succeeded; not a production parity check |
| No `Cargo.toml` / unresolved `xai_*` | Expected. Rust snapshots are not a workspace |
| `ModuleNotFoundError: grox.service` / `grox.config` / `monitor` | Expected. Grox process modules were not published |
| `viewer_id must be specified` | Home Mixer source behavior when `viewer_id == 0` |
| Empty feed for some user ids | `TEST_USER_IDS` short-circuit in source |
| `PhoenixSource: missing retrieval_sequence` | Query hydrator did not fill `ScoredPostsQuery` |
| `Server at capacity, please retry` | Thunder semaphore `RESOURCE_EXHAUSTED` |
If the goal is to exercise the published algorithm, stay in `phoenix/`: LFS pull, unzip, `uv run run_pipeline.py`, then `uv run pytest`. Use the Home Mixer, Thunder, and Grox trees as architecture references for how production composes those models, not as local services.
## License and reuse
Published files are Apache License 2.0 (`LICENSE`). You can run, modify, and redistribute the Phoenix package and the snapshot sources under that license. You cannot obtain the unpublished `xai_*` crates, proto crates, or omitted Grox/Home Mixer/Thunder modules from this repository.
## Next
Python 3.11+, uv or pip, Git LFS, and the extract layout required before inference.
Extract `oss-phoenix-artifacts` and recognize the ranked table as the success signal.
Load retrieval and ranker checkpoints, encode `example_sequence.json`, retrieve from `sports_corpus.npz`.
CLI flags, `QueryBuilder.viewer_id` validation, and For You versus ScoredPosts entry points in source.
Engine, Dispatcher, and `PlanMaster` fan-out as written — not as a local server.
LFS pointer failures, empty `TEST_USER_IDS` feeds, Thunder `RESOURCE_EXHAUSTED`, unpublished Grox modules.
---
## 05. For You request lifecycle
> CandidatePipeline.execute stages from query hydration through side effects, and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/05-for-you-request-lifecycle.md
- Generated: 2026-08-13T19:41:41.871Z
### Source Files
- `candidate-pipeline/candidate_pipeline.rs`
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`
- `home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`
- `home-mixer/server.rs`
- `home-mixer/scored_posts_server.rs`
- `home-mixer/for_you_server.rs`
---
title: "For You request lifecycle"
description: "CandidatePipeline.execute stages from query hydration through side effects, and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline."
---
`CandidatePipeline::execute` is the shared driver for every Home Mixer request. `HomeMixerServer` builds one `PhoenixCandidatePipeline` and one `ForYouCandidatePipeline`. Scored Posts RPCs call `PhoenixCandidatePipeline::execute` directly. For You RPCs call `ForYouCandidatePipeline::execute`, which re-enters Phoenix through `ScoredPostsSource` → `ScoredPostsServer::run_pipeline`.
Both pipelines implement `CandidatePipeline`. Phoenix’s candidate type is `PostCandidate`. For You’s candidate type is `FeedItem` (`Post`, `Ad`, `WhoToFollow`, `Prompt`, or `PushToHome`).
This checkout publishes the Home Mixer and candidate-pipeline sources. Production `params` constants (`RESULT_SIZE`, `FOR_YOU_MAX_RESULT_SIZE`, `TOP_K_CANDIDATES_TO_SELECT`, `TEST_USER_IDS`) and the unpublished client crates are not in this tree. Local inference uses `phoenix/run_pipeline.py`, not these gRPC servers.
## Request entry points
`HomeMixerServer::build` constructs a shared `PhoenixCandidatePipeline`, wraps it in `ScoredPostsServer`, then builds `ForYouCandidatePipeline` with that same `ScoredPostsServer`. Both gRPC services share one `QueryBuilder`.
:::endpoint POST /xai.home_mixer.ScoredPostsService/GetScoredPosts Ranked PostCandidate list
Runs `QueryBuilder::build` then `ScoredPostsServer::run_pipeline` → `PhoenixCandidatePipeline::execute`. Response is `ScoredPostsResponse.scored_posts`.
:::
:::endpoint POST /xai.home_mixer.ScoredPostsService/GetDebugScoredPosts Ranked posts plus pipeline dump
Forces B3 sampling, applies `feature_switch_overrides`, then returns `scored_posts` plus `debug_json` with query, retrieved, filtered, and selected candidates.
:::
:::endpoint POST /xai.home_mixer.ForYouFeedService/GetForYouFeed Blended FeedItem list
Requires `ForYouFeedQuery.query`. Runs `ForYouCandidatePipeline::execute`. Response is `ForYouFeedResponse.items`.
:::
:::endpoint POST /xai.home_mixer.ForYouFeedService/GetForYouFeedUrt URT timeline bytes
Same execute path as `GetForYouFeed`, then `urt::make_urt_timeline`. Decodes `cursor` into `is_bottom_request` / `is_top_request`. Copies `request_context` and `is_polling` onto `ScoredPostsQuery` after `QueryBuilder::build`.
:::
### QueryBuilder
Every RPC funnels through `QueryBuilder::build` before `execute`.
Must be non-zero. `0` returns `Status::invalid_argument("viewer_id must be specified")`.
Set if the proto flag is true **or** Gizmoduck `allow_for_you_recommendations == Some(false)`.
`ScoredPostsServer::run_pipeline` and `ForYouFeedServer::get_for_you_feed` return empty results without calling `execute`.
Forces B3 sampling on the request span.
Other `ScoredPostsQuery` fields set here: `seen_ids`, `served_ids`, `topic_ids`, `excluded_topic_ids`, `exclude_videos`, `is_bottom_request`, `is_preview`, `push_to_home_post_id`, feature-switch `params`, a per-user `Decider`, Gizmoduck roles / muted keywords / follower count / subscription / age, device status, `request_id`, and a new `prediction_id`. Gizmoduck viewer fetch times out at `200ms` and falls back to `ViewerData::default()`. `is_shadow_traffic` is `is_sampled(request_id, 0.5)`.
## Nested pipelines
For You does **not** embed `PhoenixCandidatePipeline` as a field. The wrap is a source call:
```text
ForYouFeedService / ForYouFeedUrt
│
▼
QueryBuilder.build → ScoredPostsQuery
│
▼
ForYouCandidatePipeline.execute
│ query hydrators (served history, timestamps)
│ sources (parallel)
│ ├─ ScoredPostsSource ──► ScoredPostsServer.run_pipeline
│ │ │
│ │ ▼
│ │ PhoenixCandidatePipeline.execute
│ │ │
│ │ ▼
│ │ Vec → FeedItem::Post
│ ├─ AdsSource
│ ├─ WhoToFollowSource
│ ├─ PromptsSource
│ └─ PushToHomeSource
│ BlenderSelector (ads / prompts / WTF / push-to-home)
│ side effects (spawned, non-blocking)
▼
ForYouFeedResponse.items | URT bytes
```
```mermaid
sequenceDiagram
participant Client
participant FY as ForYouFeedService
participant QB as QueryBuilder
participant ForYou as ForYouCandidatePipeline
participant SPS as ScoredPostsSource
participant SPSV as ScoredPostsServer
participant Phoenix as PhoenixCandidatePipeline
Client->>FY: GetForYouFeed(ForYouFeedQuery)
FY->>QB: build(ScoredPostsQuery)
alt viewer_id == 0
QB-->>FY: Status invalid_argument
else user_id in TEST_USER_IDS
FY-->>Client: empty items
else
FY->>ForYou: execute(query)
ForYou->>ForYou: hydrate_query
par parallel sources
ForYou->>SPS: source(query.clone())
SPS->>SPSV: run_pipeline
SPSV->>Phoenix: execute
Note over Phoenix: query hydrate → sources → hydrate → filter → score → select → post-select → side effects
Phoenix-->>SPSV: PipelineResult PostCandidate
SPSV-->>SPS: FeedItem::Post[]
ForYou->>ForYou: Ads / WTF / Prompts / PushToHome
end
ForYou->>ForYou: BlenderSelector
ForYou->>ForYou: spawn side effects
ForYou-->>FY: selected FeedItems
FY-->>Client: ForYouFeedResponse
end
```
Direct `GetScoredPosts` skips the For You layer and calls `PhoenixCandidatePipeline::execute` on the same `ScoredPostsQuery` type.
## CandidatePipeline.execute stages
`execute` is a default trait method. Neither Home Mixer pipeline overrides it. Stage order is fixed.
| Order | Stage | Parallelism | Drop / fail behavior |
| --- | --- | --- | --- |
| 1 | `hydrate_query` | Enabled `QueryHydrator`s in parallel | `Err` is logged and skipped; only `Ok` results call `update` |
| 2 | `hydrate_dependent_query` | Same, after stage 1 | Default is empty (`&[]`). Neither Home Mixer pipeline wires this |
| 3 | `fetch_candidates` | Enabled `Source`s in parallel | `Result` is flattened; a failed source contributes nothing |
| 4 | `hydrate` | Enabled candidate `Hydrator`s in parallel | Length must match input; mismatch skips that hydrator. Cannot drop candidates |
| 5 | `filter` | Enabled `Filter`s **sequentially** | Each filter partitions `kept` / `removed` |
| 6 | `score` | Enabled `Scorer`s **sequentially** | Length must match; mismatch skips that scorer. Cannot drop candidates |
| 7 | `select` | Single `Selector` | If `enable` is false, all candidates stay selected |
| 8 | `hydrate_post_selection` | Parallel hydrators | Same length-match / no-drop contract as stage 4 |
| 9 | `filter_post_selection` | Sequential filters | Removed candidates append to `filtered_candidates` |
| 10 | Truncate | `split_off(result_size())` | Surplus moves to `non_selected_candidates` |
| 11 | `finalize` | Sync hook | Default no-op. Neither pipeline overrides it |
| 12 | `run_side_effects` | `tokio::spawn` + `join_all` | Does not delay the returned `PipelineResult` |
Every component has `enable(&query)`. Disabled names are recorded on the tracing span as `disabled`.
### Stage contracts that change execute behavior
- **Query hydrators** return a fresh `Q` and copy only their fields in `update`. Failed hydrators leave those fields at `QueryBuilder` defaults (`retrieval_sequence: None`, empty follow lists, and so on).
- **Sources** return `Result, String>`. `PhoenixSource` fails with `"PhoenixSource: missing retrieval_sequence"` when stage 1 did not populate the sequence; that error is dropped by `flatten()`, so the request continues with other sources.
- **Hydrators and scorers** must return one result per input candidate, same order. `Hydrator::run` / `Scorer::run` replace a length mismatch with `Err("… length_mismatch …")` for every slot.
- **Filters** are the only stage allowed to drop candidates before selection.
- **Side effects** receive `SideEffectInput { query, selected_candidates, non_selected_candidates }`. `non_selected` is selector leftovers plus the post-truncate tail.
```rust
// candidate-pipeline/candidate_pipeline.rs — execute body
let hydrated_query = self.hydrate_query(query).await;
let hydrated_query = self.hydrate_dependent_query(hydrated_query).await;
let candidates = self.fetch_candidates(&hydrated_query).await;
let hydrated_candidates = self.hydrate(&hydrated_query, candidates).await;
let (kept_candidates, mut filtered_candidates) =
self.filter(&hydrated_query, hydrated_candidates.clone());
let scored_candidates = self.score(&hydrated_query, kept_candidates).await;
let SelectResult { selected: selected_candidates, non_selected: mut non_selected_candidates } =
self.select(&hydrated_query, scored_candidates);
// post-selection hydrate + filter, truncate to result_size(), finalize, spawn side effects
```
`PipelineResult.retrieved_candidates` is the **post-hydrate** set (`hydrated_candidates`), not the raw source union. `filtered_candidates` is pre-score removals plus post-selection removals.
## PhoenixCandidatePipeline
`CandidatePipeline`. `result_size()` is unpublished `params::RESULT_SIZE`. Selector is `TopKScoreSelector` with size `params::TOP_K_CANDIDATES_TO_SELECT` (truncate happens again at `RESULT_SIZE` after post-selection).
### Query hydrators (parallel)
`ScoringSequenceQueryHydrator`, `RetrievalSequenceQueryHydrator`, `BlockedUserIdsQueryHydrator`, `MutedUserIdsQueryHydrator`, `FollowedUserIdsQueryHydrator`, `SubscribedUserIdsQueryHydrator`, `CachedPostsQueryHydrator`, `MutualFollowQueryHydrator`, `UserDemographicsQueryHydrator`, `FollowedGrokTopicsQueryHydrator`, `FollowedStarterPacksQueryHydrator`, `InferredGrokTopicsQueryHydrator`, `ImpressionBloomFilterQueryHydrator`, `IpQueryHydrator`, `UserInferredGenderQueryHydrator`.
`ImpressedPostsQueryHydrator` is constructed in `build_with_clients` and bound to `_impressed_posts_hydrator` — it is **not** inserted into the hydrator list.
### Sources (parallel)
| Source | `enable` |
| --- | --- |
| `ThunderSource` | `!query.has_cached_posts` |
| `TweetMixerSource` | `!in_network_only && !has_cached_posts` |
| `PhoenixSource` | Not a small topic request; not new-user topic retrieval; `!in_network_only && !has_cached_posts` |
| `PhoenixTopicsSource` | Topic or new-user topic request; `!in_network_only && !has_cached_posts` |
| `PhoenixMOESource` | `EnablePhoenixMOESource` and same topic / `in_network_only` gates as Phoenix (no cache gate in `enable`) |
| `CachedPostsSource` | `query.has_cached_posts` |
When `CachedPostsQueryHydrator` sets `has_cached_posts`, Thunder / TweetMixer / Phoenix / Phoenix Topics disable and `CachedPostsSource` returns `query.cached_posts`. `PhoenixScorer` also disables on that flag, so cached candidates skip re-scoring.
### Pre-score hydrators and filters
Hydrators (parallel): `InNetworkCandidateHydrator`, `CoreDataCandidateHydrator`, `QuoteHydrator`, `VideoDurationCandidateHydrator`, `HasMediaHydrator`, `SubscriptionHydrator`, `GizmoduckCandidateHydrator`, `BlockedByHydrator`, `FilteredTopicsHydrator`, `LanguageCodeHydrator`.
Filters (sequential): `DropDuplicatesFilter`, `CoreDataHydrationFilter`, `AgeFilter(MAX_POST_AGE)`, `SelfTweetFilter`, `RetweetDeduplicationFilter`, `IneligibleSubscriptionFilter`, `PreviouslySeenPostsFilter`, `PreviouslySeenPostsBackupFilter`, `PreviouslyServedPostsFilter`, `MutedKeywordFilter`, `AuthorSocialgraphFilter`, `VideoFilter` (`exclude_videos` only), `TopicIdsFilter`, `NewUserTopicIdsFilter`.
### Scorers and selection
Wired sequential scorers: `PhoenixScorer` (`enable` = `!has_cached_posts`), `RankingScorer` (always on), `VMRanker` (`EnableVMRanker`).
`RankingScorer` writes the weighted multi-action score used by `TopKScoreSelector` (`candidate.score`, missing → `-inf`). Sibling modules `WeightedScorer`, `AuthorDiversityScorer`, and `OonScorer` exist in `home-mixer/scorers/` but are **not** in this pipeline’s `scorers` vec.
### Post-selection and side effects
Post-selection hydrators: `VFCandidateHydrator`, `AdsBrandSafetyHydrator`, `AdsBrandSafetyVfHydrator`, `TweetTypeMetricsHydrator`, `FollowingRepliedUsersHydrator`, `MutualFollowJaccardHydrator`.
Post-selection filters: `VFFilter`, `AncillaryVFFilter`, `DedupConversationFilter`.
Side effects (spawned): `PhoenixExperimentsSideEffect` (`is_shadow_traffic`), `RerankingKafkaSideEffect` (prod and 5% sample), `RedisPostCandidateCacheSideEffect` (prod and `!has_cached_posts`), `ScoredStatsSideEffect`, `MutualFollowStatsSideEffect` (`EnableMutualFollowJaccardHydration`), `PhoenixRequestCacheSideEffect` (`EnablePhoenixRequestCacheSideEffect`).
`CacheRequestInfoSideEffect` exists on disk and is **not** wired.
`ScoredPostsServer` maps `selected_candidates` to proto `ScoredPost` (`tweet_id`, `author_id`, `score`, `in_network`, `served_type`, ancestors, VF reason, tweet-type metrics, brand-safety verdict, safety labels, text). Surface logging uses `in_network_only` → `ranked_following`, non-empty `topic_ids` → `topics`, excluded topics → `for_you_with_snoozed_topics`, else `for_you`.
## ForYouCandidatePipeline
`CandidatePipeline`. Hydrators, filters, scorers, and both post-selection lists are empty slices — `execute` still walks those stages as no-ops. `result_size()` is unpublished `params::FOR_YOU_MAX_RESULT_SIZE`.
### Query hydrators
| Hydrator | `enable` | Writes |
| --- | --- | --- |
| `ServedHistoryQueryHydrator` | `EnableUrtMigrationComponents` | `served_history`, recent `served_ids`, `who_to_follow_eligible` (fatigue vs `WhoToFollowFatigueHours`) |
| `PastRequestTimestampsQueryHydrator` | `EnableUrtMigrationComponents` | `non_polling_timestamps` |
These run **before** `ScoredPostsSource` clones the query into Phoenix, so Phoenix filters that read `served_ids` see For You hydration when `EnableUrtMigrationComponents` is on.
### Sources (parallel)
| Source | `enable` | Output item |
| --- | --- | --- |
| `ScoredPostsSource` | always (default) | `FeedItem::Post` from Phoenix `run_pipeline` |
| `AdsSource` | `EnableAdsSource && !is_preview` | `FeedItem::Ad` |
| `WhoToFollowSource` | `EnableWhoToFollowModule && who_to_follow_eligible` | one `FeedItem::WhoToFollow` (max 3 users) |
| `PromptsSource` | `EnablePrompts` | `FeedItem::Prompt` |
| `PushToHomeSource` | `push_to_home_post_id.is_some()` | `FeedItem::PushToHome` |
### Selector
`BlenderSelector` overrides `select` (it does not use default score-sort). It partitions items, blends ads with `SafeGapAdsBlender` when `AdsBlenderType == "safe_gap"` else `PartitionOrganicAdsBlender`, inserts prompts at `PROMPTS_POSITION`, inserts one Who-to-Follow module at `WHO_TO_FOLLOW_POSITION`, and pins push-to-home at index `0`. `score()` is unused (`0.0`).
After `select`, `execute` still truncates the blended list to `FOR_YOU_MAX_RESULT_SIZE`.
### Side effects (spawned)
`AdsInjectionLoggingSideEffect`, `PublishSeenIdsToKafkaSideEffect`, `ServedCandidatesKafkaSideEffect` (prod + `is_shadow_traffic` + URT flag), `ClientEventsKafkaSideEffect`, `ForYouResponseStatsSideEffect`, `UpdatePastRequestTimestampsSideEffect` (prod, URT flag, not polling, not `BackgroundFetch`), `UpdateServedHistorySideEffect`, `TruncateServedHistorySideEffect`.
## PipelineResult and responses
Post-hydrate candidates. Phoenix: `PostCandidate`. For You: mixed `FeedItem`s from all enabled sources.
Union of pre-score and post-selection removals. Empty on For You (no filters).
Post-truncate, post-`finalize` list returned to the server.
Fully hydrated `ScoredPostsQuery` after both query-hydrator stages.
`GetDebugScoredPosts` serializes all four fields plus counts. `GetForYouFeed` returns only `selected_candidates`. `GetForYouFeedUrt` serializes those items into a URT timeline; cursor decode errors are logged and the cursor is ignored.
## Short-circuits and errors
| Condition | Effect |
| --- | --- |
| `viewer_id == 0` | RPC fails before `execute` |
| `user_id` in `TEST_USER_IDS` | Empty `ScoredPosts` / empty For You items; no `execute` |
| Missing `ForYouFeedQuery.query` | `Status::invalid_argument("query must be specified")` |
| Query hydrator `Err` | Field stays default; other hydrators still merge |
| Source `Err` | That source contributes zero candidates |
| `PhoenixSource: missing retrieval_sequence` | Phoenix retrieval dropped; Thunder and others can still fill the batch |
| Hydrator / scorer length mismatch | That component’s updates are skipped |
| URT cursor decode failure | Warning; request continues without cursor fields |
| Side-effect `Err` | Isolated inside the spawned task; response already returned |
Empty final Phoenix results increment `{PipelineName}.execute` with scope `requests/result_empty`.
## Related pages
CLI flags, QueryBuilder validation, and For You versus Scored Posts entry points.
Query fields, TEST_USER_IDS empty feeds, and ScoredPost / ForYouFeed / URT mapping.
Phoenix and For You sources, enable predicates, and served_type.
Pre-score vs post-selection filters and the length-match hydrator contract.
PhoenixScorer, RankingScorer, VMRanker, and TopKScoreSelector.
Source, Hydrator, Filter, Scorer, Selector, QueryHydrator, and SideEffect contracts.
AdsSource, blender types, and prompt / who-to-follow / push-to-home insertion.
viewer_id must be specified, TEST_USER_IDS, and missing retrieval_sequence.
---
## 06. In-network and out-of-network
> Thunder followed-author lookup versus Phoenix retrieval, enable predicates, and how in_network_only changes sourcing and served_type.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/06-in-network-and-out-of-network.md
- Generated: 2026-08-13T19:41:35.105Z
### Source Files
- `home-mixer/sources/thunder_source.rs`
- `home-mixer/sources/phoenix_source.rs`
- `thunder/posts/post_store.rs`
- `thunder/thunder_service.rs`
- `phoenix/recsys_retrieval_model.py`
- `home-mixer/scorers/oon_scorer.rs`
---
title: "In-network and out-of-network"
description: "Thunder followed-author lookup versus Phoenix retrieval, enable predicates, and how in_network_only changes sourcing and served_type."
---
`PhoenixCandidatePipeline` mixes two organic retrieval paths: `ThunderSource` looks up recent posts from the viewer's followed authors via Thunder `GetInNetworkPosts`, and `PhoenixSource` (plus topic / MoE variants) retrieves from a global Phoenix corpus. After fetch, `InNetworkCandidateHydrator` stamps `PostCandidate.in_network` from the following list. `ScoredPostsQuery.in_network_only` turns off every out-of-network source, retags Thunder posts as `RankedFollowing`, and switches Phoenix scoring to `HomeTimelineRankedFollowing`.
This checkout can run Phoenix retrieval locally (`phoenix/run_pipeline.py` against `sports_corpus.npz`). Home Mixer, Thunder, and the production Phoenix retrieval RPC depend on unpublished crates and services. See [Runtime boundaries](/runtime-boundaries).
## Two retrieval paths
```mermaid
flowchart TB
subgraph hm["Home Mixer — PhoenixCandidatePipeline"]
Q["ScoredPostsQuery in_network_only, followed_user_ids, retrieval_sequence, seen_ids"]
EN["Source.enable predicates"]
TH["ThunderSource"]
PH["PhoenixSource / PhoenixTopicsSource / PhoenixMOESource"]
TM["TweetMixerSource"]
HY["InNetworkCandidateHydrator"]
RS["PhoenixScorer + RankingScorer"]
Q --> EN
EN -->|"!has_cached_posts"| TH
EN -->|"!in_network_only && !has_cached_posts"| PH
EN -->|"!in_network_only && !has_cached_posts"| TM
TH --> HY
PH --> HY
TM --> HY
HY --> RS
end
subgraph thunder["Thunder — InNetworkPostsService"]
GPS["get_in_network_posts"]
PS["PostStore timelines original + secondary + video"]
GPS --> PS
end
subgraph phoenix["Phoenix retrieval"]
RET["retrieve(user sequence, corpus)"]
TOWER["user representation × corpus embeddings"]
TOPK["jax.lax.top_k"]
RET --> TOWER --> TOPK
end
TH -->|"GetInNetworkPostsRequest following_user_ids, exclude_tweet_ids"| GPS
PH -->|"retrieval_sequence required"| RET
```
`CandidatePipeline.fetch_candidates` filters `sources()` with `Source.enable`, then `join_all`s the enabled `source()` calls. Enabled sources concatenate; there is no source-level merge by tweet id at this stage.
`ForYouCandidatePipeline` does not call Thunder or Phoenix itself. It sources `FeedItem`s through `ScoredPostsSource`, which runs `PhoenixCandidatePipeline`.
## How `in_network_only` is set
`HomeMixerServer` builds `ScoredPostsQuery.in_network_only` as:
```text
proto_query.in_network_only || viewer_data.allow_for_you_recommendations == Some(false)
```
Either the client asks for following-only, or the viewer is not allowed For You recommendations. `viewer_id == 0` fails earlier with `invalid_argument: viewer_id must be specified`.
When true, Phoenix, Phoenix topics, Phoenix MoE, and Tweet Mixer do not run. Thunder still runs (unless a Redis cached-post hit replaces live sources). Thunder `served_type` becomes `RankedFollowing`. Phoenix scoring and request-cache product surface become `HomeTimelineRankedFollowing`.
`CachedPostsQueryHydrator` includes `in_network_only` in the Redis key (`cached_posts_key(user_id, topic_ids, in_network_only, exclude_videos)`), so following-only and For You caches do not collide. A hit is applied only when the decoded list has at least `500` posts (`MIN_CACHED_POSTS_THRESHOLD`); then `has_cached_posts` is true and live sources disable.
`FollowedGrokTopicsQueryHydrator` also skips filling `new_user_topic_ids` when `in_network_only` is true, so the new-user topic retrieval branch cannot turn on during following-only.
## Source enable predicates
`PhoenixCandidatePipeline` source order:
| Source | `enable` | `served_type` |
|---|---|---|
| `ThunderSource` | `!query.has_cached_posts` | `ForYouInNetwork` when `!in_network_only`, else `RankedFollowing` |
| `TweetMixerSource` | `!in_network_only && !has_cached_posts` | `ForYouTweetMixer` |
| `PhoenixSource` | not a topic request, or bulk topic (`topic_ids.len() > 6`); and not (`EnableNewUserTopicRetrieval` and `has_new_user_topic_ids`); and `!in_network_only && !has_cached_posts` | `ForYouPhoenixRetrieval` |
| `PhoenixTopicsSource` | `(is_topic_request && !is_bulk_topic_request)` or (`EnableNewUserTopicRetrieval` and `has_new_user_topic_ids`); and `!in_network_only && !has_cached_posts` | `ForYouPhoenixRetrieval` |
| `PhoenixMOESource` | `EnablePhoenixMOESource` and (not a topic request, or bulk topic); and `!in_network_only && !has_cached_posts` | `ForYouPhoenixRetrievalMoe` |
| `CachedPostsSource` | `query.has_cached_posts` | preserved from cache |
`ScoredPostsQuery.is_topic_request()` is `!topic_ids.is_empty()`. `is_bulk_topic_request()` is `topic_ids.len() > 6`. A bulk topic request keeps `PhoenixSource` / `PhoenixMOESource` and turns `PhoenixTopicsSource` off. A non-bulk topic request is the inverse.
`ThunderSource.enable` does **not** inspect `in_network_only`. Following-only still fetches Thunder. Out-of-network is gated by the other sources' predicates.
## Thunder: followed-author lookup
`FollowedUserIdsQueryHydrator` fills `query.user_features.followed_user_ids` from Social Graph before sources run. `ThunderSource` copies that list into `GetInNetworkPostsRequest.following_user_ids` (cast to `u64`), excludes `query.seen_ids`, and picks a Thunder cluster from `ThunderClusterId` plus decider override.
```rust ThunderSource request
GetInNetworkPostsRequest {
user_id: query.user_id,
following_user_ids: query.user_features.followed_user_ids, // as u64
max_results: query.params.get(ThunderMaxResults),
exclude_tweet_ids: query.seen_ids,
algorithm: query.params.get(ThunderAlgorithm),
debug: false,
is_video_request: false,
}
```
Thunder `get_in_network_posts` then:
1. Rejects immediately with `RESOURCE_EXHAUSTED` (`"Server at capacity, please retry"`) if the request semaphore cannot be acquired.
2. Caps `following_user_ids` and `exclude_tweet_ids` at `MAX_INPUT_LIST_SIZE`.
3. Fetches `PostStore.get_all_posts_by_users` (or `get_videos_by_users` when `is_video_request`).
4. Sorts by `created_at` descending (`score_recent`) and takes `max_results` (or `MAX_POSTS_TO_RETURN` / `MAX_VIDEOS_TO_RETURN` when `max_results == 0`).
`PostStore` keeps three per-author deques: original (non-reply, non-retweet), secondary (replies and retweets), and video. Originals are capped at `MAX_ORIGINAL_POSTS_PER_AUTHOR`; secondaries at `MAX_REPLY_POSTS_PER_AUTHOR`. Secondary lookup also filters reply threads so replies stay attached to followed conversations. Retweets of the requesting user are dropped.
`ThunderSource` always sends `debug: false`. Thunder's Strato following-list fallback runs only when `following_user_ids` is empty **and** `debug` is true, so the Home Mixer path never uses that fallback. `algorithm` is forwarded on the request but `ThunderServiceImpl` does not read it; ranking inside Thunder is recency only.
Each returned `LightPost` becomes a `PostCandidate` with `tweet_id`, `author_id`, reply / retweet ids, and ancestors (`in_reply_to_post_id`, then `conversation_id` if different). Reply rows are also written once into `query.in_network_replies` (`OnceLock>`) for `FollowingRepliedUsersHydrator`.
## Phoenix: out-of-network retrieval
`PhoenixSource` does not use the following list. It requires `query.retrieval_sequence` (from `RetrievalSequenceQueryHydrator`) and fails with `"PhoenixSource: missing retrieval_sequence"` if it is absent. The same missing-sequence error exists on `PhoenixTopicsSource` and `PhoenixMOESource`.
Production retrieve call (PhoenixSource):
`PhoenixRetrievalInferenceClusterId`, remapped to `PhoenixRetrievalNewUserInferenceClusterId` when `retrieval_sequence.metadata.length` is below `PhoenixRetrievalNewUserHistoryThreshold` (threshold `0` disables the remap). Decider keys `enable_phoenix_retrieval_lap7_to_fou` / `enable_phoenix_retrieval_fou_to_lap7` can swap `Experiment1Lap7` ↔ `Experiment1Fou`.
`PhoenixMaxResults` (MoE uses `PhoenixMOEMaxResults`).
`PhoenixSource` passes an empty exclude list and no topic filter. `PhoenixTopicsSource` passes expanded topic entity ids plus a topic filter mode.
Mapped fields: `tweet_id`, `author_id`, `in_reply_to_tweet_id`, `retweeted_tweet_id` (omitted when `0`), `served_type: ForYouPhoenixRetrieval`.
Locally, `PhoenixRetrievalModel.__call__` encodes a user representation from `RecsysBatch` + `RecsysEmbeddings`, scores `user_representation @ corpus_embeddings.T`, applies an optional `corpus_mask`, and returns `jax.lax.top_k`. That is corpus similarity, not followed-author lookup. The published demo corpus is `sports_corpus.npz` via `run_pipeline.py`.
## `in_network` versus `served_type`
These are independent fields on `PostCandidate`.
| Field | Who sets it | Meaning |
|---|---|---|
| `served_type` | The source, at fetch time | Provenance / product surface of the *source*. Thunder For You → `ForYouInNetwork`; Thunder following-only → `RankedFollowing`; Phoenix → `ForYouPhoenixRetrieval`; MoE → `ForYouPhoenixRetrievalMoe`; Tweet Mixer → `ForYouTweetMixer`. |
| `in_network` | `InNetworkCandidateHydrator`, after fetch | `author_id == viewer_id` **or** `author_id` is in `followed_user_ids`. |
`InNetworkCandidateHydrator.enable` is `!query.has_cached_posts`. Cached candidates keep the `in_network` value stored in Redis.
A Phoenix-retrieved post from an account the viewer already follows is still `served_type = ForYouPhoenixRetrieval` and becomes `in_network = Some(true)`. A Thunder post is `ForYouInNetwork` / `RankedFollowing` and is almost always in-network by construction (self posts also count as in-network).
`ScoredPostsServer` copies both onto proto `ScoredPost`:
```text
in_network: candidate.in_network.unwrap_or(false)
served_type: candidate.served_type.map(|t| t as i32).unwrap_or_default()
```
Request logs use a separate surface string: `ranked_following` when `in_network_only`, else `topics` if `topic_ids` is non-empty, else `for_you_with_snoozed_topics` if `excluded_topic_ids` is non-empty, else `for_you`.
## Scoring and filters that read `in_network`
`PhoenixScorer` still scores Thunder and Phoenix candidates together. The product surface on the prediction request is:
- `ProductSurface::HomeTimelineRankedFollowing` when `in_network_only`
- `ProductSurface::HomeTimelineRanking` otherwise
The same surface split appears on `PhoenixRequestCacheSideEffect`, `PhoenixExperimentsSideEffect`, and `RerankingKafkaSideEffect`. `CacheRequestInfoSideEffect` is enabled only when `APP_ENV == "prod"` **and** `!in_network_only`.
`RankingScorer` applies an out-of-network multiplier after author diversity:
```text
final = after_diversity * effective_oon_weight if in_network == Some(false)
final = after_diversity otherwise
```
`effective_oon_weight`:
| Condition | Weight |
|---|---|
| `topic_ids` non-empty | `TopicOonWeightFactor` |
| User snowflake age `< NewUserAgeThresholdSecs` **and** `followed_user_ids.len() >= NEW_USER_MIN_FOLLOWING` | `NEW_USER_OON_WEIGHT_FACTOR` |
| Else | `OonWeightFactor` |
Those identifiers live in the unpublished `params` crate; this checkout does not publish numeric defaults.
`NewUserTopicIdsFilter` keeps a candidate when `in_network == Some(true)` **or** the candidate's `filtered_topic_ids` intersect the expanded `new_user_topic_ids`.
`home-mixer/scorers/oon_scorer.rs` defines `OONScorer` (`base_score * OON_WEIGHT_FACTOR` when `in_network == Some(false)`), but `scorers/mod.rs` does not declare that module and `PhoenixCandidatePipeline` does not install it. Live OON downweight is `RankingScorer`.
## Following-only versus For You
```text
in_network_only = false (For You)
Thunder → ForYouInNetwork
TweetMixer / Phoenix / Topics / MoE as predicates allow
PhoenixScorer surface = HomeTimelineRanking
log surface = for_you | topics | for_you_with_snoozed_topics
in_network_only = true (Following / Ranked Following)
Thunder → RankedFollowing
Phoenix*, TweetMixer disabled
PhoenixScorer surface = HomeTimelineRankedFollowing
log surface = ranked_following
```
Hydrators, pre-score filters, `PhoenixScorer`, `RankingScorer`, `VMRanker`, and `TopKScoreSelector` still run on the remaining Thunder (or cached) candidates.
## Errors and verification
| Symptom | Cause |
|---|---|
| `"PhoenixSource: missing retrieval_sequence"` (or Topics / MoE equivalent) | `RetrievalSequenceQueryHydrator` did not fill the sequence; Phoenix sources cannot run. |
| `"ThunderSource: no available channel"` | `ThunderClient.get_random_channel` returned `None` for the resolved cluster. |
| gRPC `RESOURCE_EXHAUSTED` / `"Server at capacity, please retry"` | Thunder semaphore saturated. |
| `"Failed to fetch following list: ..."` | Thunder debug path only (`debug == true` and empty following list). Home Mixer never sends that. |
| Empty or following-only feed when For You was expected | `allow_for_you_recommendations == Some(false)` forced `in_network_only`. |
| Unexpected OON posts on a following request | `in_network_only` is false; check proto query and viewer data. |
| Local `run_pipeline.py` has no Thunder posts | Expected. The Python pipeline retrieves from the published corpus only. |
To confirm a production-shaped request, inspect `PipelineResult.retrieved_candidates`: Thunder rows carry `ForYouInNetwork` or `RankedFollowing`; Phoenix rows carry `ForYouPhoenixRetrieval`. After hydration, `in_network` should be `Some(true)` for followed authors regardless of `served_type`.
## Next
Full PhoenixCandidatePipeline and ForYouCandidatePipeline source list, cluster resolution, and served_type assignment.
RPC fields, PostStore timelines and retention, Kafka ingest, semaphore capacity, Strato fallback.
RankingScorer OON factor, WeightedScorer, AuthorDiversityScorer, VM ranker, TopKScoreSelector.
CandidatePipeline.execute stages and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline.
Query fields, QueryBuilder defaults, and ScoredPost mapping including in_network and served_type.
viewer_id, TEST_USER_IDS empty feeds, Thunder RESOURCE_EXHAUSTED, missing retrieval_sequence.
---
## 07. Candidate isolation
> make_recsys_attn_mask rules: candidates attend to user and history plus self, never to other candidates, so scores stay batch-independent.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/07-candidate-isolation.md
- Generated: 2026-08-13T19:40:55.751Z
### Source Files
- `phoenix/grok.py`
- `phoenix/recsys_model.py`
- `phoenix/test_recsys_model.py`
- `phoenix/README.md`
- `phoenix/recsys_retrieval_model.py`
---
title: "Candidate isolation"
description: "make_recsys_attn_mask rules: candidates attend to user and history plus self, never to other candidates, so scores stay batch-independent."
---
`make_recsys_attn_mask` in `phoenix/grok.py` is the ranking attention contract. `RecsysModel.build_inputs` concatenates one user token, the history window, and the candidate window; `Transformer.__call__` then multiplies that mask into the padding mask so each candidate can attend to user plus history and itself, and never to another candidate. Blocked keys are written to `-1e30` before the fp32 softmax, so a candidate's logits do not depend on which other candidates share the sequence.
Isolation is ranking-only. The retrieval user tower calls the same `Transformer` with `candidate_start_offset=None` and keeps a standard causal mask over `[user | history]`.
## Sequence layout
`RecsysModel.build_inputs` concatenates three blocks and returns `candidate_start_offset` as the first candidate index:
```text
[ user (1) | history (S) | candidates (C) ]
^ ^
1 candidate_start_offset = 1 + S
```
| Block | Source | Token count | Padding (`True` = valid) |
| --- | --- | --- | --- |
| User | `block_user_reduce` | 1 | `user_hashes[:, 0] != 0` |
| History | `block_history_reduce` | `history_seq_len` (`S`) | `history_post_hashes[:, :, 0] != 0` |
| Candidates | `block_candidate_reduce` | `candidate_seq_len` (`C`) | `candidate_post_hashes[:, :, 0] != 0` |
`candidate_start_offset` is `user_padding_mask.shape[1] + history_padding_mask.shape[1]`, not a config field. Hash `0` is reserved padding on user, history, and candidate first-hash slots.
`PhoenixModelConfig` history window. Live ranker length comes from published `config.json` via `build_model_config` in `run_pipeline.py`.
`PhoenixModelConfig` candidate window packed into one forward. Isolation is what makes packing safe.
## Mask rules
`make_recsys_attn_mask(seq_len, candidate_start_offset, dtype=jnp.float32)` returns `[1, 1, seq_len, seq_len]` with `1` = can attend and `0` = blocked.
| Query positions | Allowed keys | Blocked keys |
| --- | --- | --- |
| `0 .. offset-1` (user + history) | Causal: key `j` only if `j <= i` | Future user/history tokens; every candidate |
| `offset .. seq_len-1` (candidates) | All user + history keys, plus the query's own index | Every other candidate |
User and history use a **causal** lower triangle (`jnp.tril`), not bidirectional attention. Candidates attend to the entire prefix, including history tokens that later history tokens cannot yet see.
The `TestMakeRecsysAttnMask` fixture `[user, h1, h2, c1, c2, c3]` (`seq_len=6`, `offset=3`) is the contract:
```text
U h1 h2 c1 c2 c3
U 1 0 0 0 0 0
h1 1 1 0 0 0 0
h2 1 1 1 0 0 0
c1 1 1 1 1 0 0
c2 1 1 1 0 1 0
c3 1 1 1 0 0 1
```
Edge cases covered by the same tests: a single candidate is prefix-plus-self; `offset=1` (no history) is user-plus-self per candidate.
## Construction
The function does not start from a custom sparse pattern. It edits a full causal mask in three steps:
1. `causal_mask = jnp.tril(ones([1, 1, seq_len, seq_len]))`
2. Zero the candidate–candidate block: `[:, :, offset:, offset:] = 0`
3. Restore the candidate diagonal: `attn_mask[:, :, i, i] = 1` for `i` in `range(offset, seq_len)`
Step 2 is what removes later-candidate → earlier-candidate edges that a plain causal mask would allow.
Total length `1 + S + C`.
First candidate index. Must match the concatenated layout from `build_inputs`.
Mask dtype. `Transformer` passes `embeddings.dtype` (`PhoenixModelConfig.fprop_dtype` defaults to `jnp.bfloat16`).
## How the transformer applies it
```mermaid
flowchart TD
subgraph ranker ["RecsysModel.__call__"]
BI["build_inputs → embeddings, padding_mask, offset"]
ROPE{"right_anchored_rope?"}
POS["right_anchored_rope_positions"]
TF["Transformer(..., candidate_start_offset=offset)"]
CUT["out_embeddings[:, offset:, :]"]
HEAD["action logits + continuous sigmoid"]
BI --> ROPE
ROPE -->|true| POS --> TF
ROPE -->|false, default| TF
TF --> CUT --> HEAD
end
subgraph transformer ["Transformer.__call__"]
PAD["padding_mask[:, None, None, :]"]
OFF{"candidate_start_offset is not None?"}
ISO["make_recsys_attn_mask"]
CAU["jnp.tril causal"]
MUL["mask = padding * attn"]
MHA["MHA: where(mask, logits, -1e30) then softmax"]
PAD --> OFF
OFF -->|ranker| ISO --> MUL
OFF -->|retrieval user tower| CAU --> MUL
MUL --> MHA
end
TF --> PAD
```
The padding mask is a **key** mask: invalid (hash-`0`) positions cannot be attended to, even when the isolation mask would allow the index. Decoder layers do not consume `padding_mask` again after this multiply.
`MultiHeadAttention` keeps softmax in fp32, tanh-clips logits at 30, then replaces masked locations with `-1e30`. Isolation is attention-only; the pointwise FFN does not mix candidates.
## Ranker vs retrieval
| Caller | `candidate_start_offset` | Sequence | Effect |
| --- | --- | --- | --- |
| `RecsysModel.__call__` | `1 + S` | `[user \| history \| candidates]` | Isolation on |
| Retrieval user tower | `None` | `[user \| history]` | Standard causal only |
| Retrieval `CandidateTower` | n/a | No transformer over the candidate set | Candidates encoded independently by construction |
Omitting `candidate_start_offset` on a packed ranker sequence is a silent regression: later candidates regain causal access to earlier candidates.
## RoPE companion
When `PhoenixModelConfig.right_anchored_rope` is `True` (dataclass default is `False`), `right_anchored_rope_positions` assigns every candidate the same position `history_end = num_user_prefix_tokens + history_seq_len`. `RecsysModel` hard-codes `num_user_prefix_tokens=1`. Padded tokens get position `0`. Shared candidate positions keep rotary encodings from encoding batch order; they do not replace the attention mask.
## Outputs after isolation
Only candidate positions are unembedded:
| Field | Shape | Source |
| --- | --- | --- |
| `RecsysModelOutput.logits` | `[B, C, num_actions]` | `dot(candidate_embeddings, unembedding)` |
| `RecsysModelOutput.continuous_preds` | `[B, C, num_continuous]` | `sigmoid(dot(candidate_embeddings, continuous_head))` |
A candidate's vector is a function of user, unpadded history, and that candidate. Changing neighbors in the `C` window does not change those logits.
The repo states this keeps scores consistent across batch composition and therefore cacheable. Isolation itself does not implement a cache.
## Verify
From `phoenix/`:
```bash
uv run pytest test_recsys_model.py::TestMakeRecsysAttnMask -v
```
`TestMakeRecsysAttnMask` asserts shape `[1, 1, T, T]`, causal prefix, full prefix visibility from every candidate, self-attend, zero off-diagonal candidate block, `float32`/`float16` dtypes, and the two edge layouts above. `TestRightAnchoredRopePositions.test_candidates_share_position` covers the optional RoPE companion.
Do not treat user/history as fully visible to each other. The implemented prefix is causal. A bidirectional prefix would be a behavior change and would fail `test_user_history_has_causal_attention`.
## Contributor constraints
- Always pass `candidate_start_offset` from `build_inputs` into `Transformer`. `None` is only correct when the sequence has no candidate block.
- Keep `offset` aligned with `[user | history | candidates]`. Shifting the cut mixes history into the isolated block or candidates into the causal prefix.
- Do not add candidate–candidate edges for diversity or calibration inside this mask. Diversity and weighted scoring happen after logits (`WeightedScorer`, `AuthorDiversityScorer`).
- Isolation does not hide padded keys. Preserve hash-`0` padding so the key mask stays correct.
## Related pages
PhoenixModelConfig fields, TransformerConfig, RecsysBatch, and published config.json keys that set S and C.
How RecsysBatch hashes are looked up before the isolated transformer.
Per-action logits taken from isolated candidate embeddings.
Attention-mask and retrieval pytest targets.
Retrieve then rank with the published checkpoints.
---
## 08. Hash embeddings
> HashConfig, linear-congruential ID hashing, pad-offset unified tables, and how RecsysBatch hashes are looked up before the transformer.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/08-hash-embeddings.md
- Generated: 2026-08-13T19:41:41.096Z
### Source Files
- `phoenix/recsys_model.py`
- `phoenix/run_pipeline.py`
- `phoenix/runners.py`
- `phoenix/recsys_retrieval_model.py`
- `phoenix/README.md`
---
title: "Hash embeddings"
description: "HashConfig, linear-congruential ID hashing, pad-offset unified tables, and how RecsysBatch hashes are looked up before the transformer."
---
Phoenix never feeds raw `user_id`, `post_id`, or `author_id` values into the transformer. `run_pipeline.py` maps each ID through a training-matched linear-congruential hash, shifts the result into a pad-offset region of a unified table, and passes the looked-up vectors as `RecsysEmbeddings`. `PhoenixModel.build_inputs` and `PhoenixRetrievalModel.build_user_representation` only consume those pre-looked-up tensors plus the hash indices that mark padding.
Retrieval and ranking keep **separate** `config.json` `hash_params` and **separate** `embedding_tables.npz` files. Hash a given ID with the stage that owns the table you index. Mixing retrieval hashes into the ranker table (or the reverse) is a silent lookup error.
## Contract
The model API is two objects, not an ID list:
| Object | Owns | Does not own |
|---|---|---|
| `RecsysBatch` | Hash indices, history actions, product-surface IDs, optional timestamps / IP hashes | Embedding vectors |
| `RecsysEmbeddings` | Vectors already gathered from the hash tables | ID hashing |
Callers look up first, then call the Haiku module. `RecsysInferenceRunner.rank` and `RecsysRetrievalInferenceRunner.encode_user` both document this: hashes go in the batch, embeddings are already gathered.
```mermaid
flowchart LR
subgraph ids [Raw IDs]
U["user_id"]
P["post_id"]
A["author_id"]
end
subgraph hasher ["run_pipeline.build_hash_functions"]
LCG["_hash_ids LCG"]
OFF["pad-offset user / item / author"]
end
subgraph tables ["build_unified_emb_table"]
T["pad 65 + user + item + author"]
end
subgraph tensors [Model inputs]
B["RecsysBatch hashes"]
E["RecsysEmbeddings"]
end
subgraph reduce ["block_*_reduce"]
R["proj_mat_1 / 2 / 3"]
end
subgraph xf [Transformer]
X["user + history + candidates"]
end
U --> LCG
P --> LCG
A --> LCG
LCG --> OFF
OFF --> B
OFF --> T
T --> E
B --> R
E --> R
R --> X
```
## HashConfig
`HashConfig` is the width of the hash dimension. It is not the hash function itself. Published `config.json` supplies the counts; `PhoenixModelConfig` and `PhoenixRetrievalModelConfig` both default to a new `HashConfig()` when the field is omitted.
Hash functions per user. Shapes `user_hashes` as `[B, num_user_hashes]` and `user_embeddings` as `[B, num_user_hashes, D]`.
Hash functions per post. Shapes `history_post_hashes` / `candidate_post_hashes` as `[B, S|C, num_item_hashes]`.
Hash functions per author. Shapes `history_author_hashes` / `candidate_author_hashes` as `[B, S|C, num_author_hashes]`.
Optional IP hash width. When `user_ip_embeddings` is set and this value is greater than 0, `block_user_reduce` sums the IP vectors and adds them to the projected user embedding. The published pipeline leaves this at 0.
`PhoenixModelConfig.use_ip_address` exists as a config flag (`False` by default) and is not read by `build_inputs`. IP injection is gated only by `num_ip_hashes` and a non-`None` `user_ip_embeddings` tensor.
The published mini checkpoint uses 2 hashes per entity and 1,000,000 rows per split table (`user` / `item` / `author`). Demo scripts `run_ranker.py` and `run_retrieval.py` hard-code the same `HashConfig(2, 2, 2)` but skip published hashing entirely.
## Linear-congruential ID hashing
Published inference reconstructs the training hash in `_hash_ids`. Arithmetic is **numpy `int64`**, including wrapping overflow, so Python arbitrary-precision ints will not match.
For each ID and each hash function `j`:
```text
raw = (id * scales[j] + biases[j]) % modulus
hash = 0 if id == 0
hash = (raw % (num_buckets - 1)) + 1 otherwise
```
| Rule | Effect |
|---|---|
| `id == 0` | Output is 0 for every hash function. Unused history slots and missing IDs stay padding. |
| Nonzero ID | Output is in `[1, num_buckets - 1]`. Bucket 0 is reserved. |
| `num_buckets` | The matching vocab size: `user_vocab_size`, `item_vocab_size`, or `author_vocab_size`. |
| `scales` / `biases` length | One pair per hash function. Must equal `num_*_hashes`. |
`build_hash_functions(config)` then wraps `_hash_ids` with the pad offset (next section) and returns `(hash_user, hash_item, hash_author)`.
Published `config.json` keys consumed here:
| Key | Role |
|---|---|
| `hash_params.user_hash_scales` / `user_biases` / `user_modulus` | User LCG |
| `hash_params.item_hash_scales` / `item_biases` / `item_modulus` | Post LCG |
| `hash_params.author_hash_scales` / `author_biases` / `author_modulus` | Author LCG |
| `user_vocab_size` / `item_vocab_size` / `author_vocab_size` | `num_buckets` and table widths |
| `num_user_hashes` / `num_item_hashes` / `num_author_hashes` | Copied into `HashConfig` |
The numeric `scales`, `biases`, and `modulus` values live only in the extracted artifact `config.json` files (Git LFS). This checkout does not embed those constants in source.
`run_pipeline.py` builds **two** hasher triples: one from `retrieval/config.json`, one from `ranker/config.json`. The same `user_id` is hashed twice. Use `hash_user` with the retrieval table and `rank_hash_user` with the ranker table.
## Pad-offset unified table
Exported `embedding_tables.npz` stores three split arrays: `user_embeddings`, `item_embeddings`, `author_embeddings`. `build_unified_emb_table` concatenates them behind a **hard-coded** pad of 65 zero rows so a single integer index can address every entity type.
```text
index: 0 .............. 64 | 65 .......... 65+uv-1 | 65+uv .... 65+uv+iv-1 | 65+uv+iv .... 65+uv+iv+av-1
region: pad (zeros) | user_embeddings | item_embeddings | author_embeddings
offset: 0 | pad | pad + uv | pad + uv + iv
```
`pad = 65` is not a `config.json` key. Both `build_hash_functions` and `build_unified_emb_table` hard-code it.
After LCG, nonzero hashes are shifted:
| Hasher | Index written |
|---|---|
| `hash_user` | `0` if LCG is 0, else `h + 65` |
| `hash_item` | `0` if LCG is 0, else `h + 65 + user_vocab_size` |
| `hash_author` | `0` if LCG is 0, else `h + 65 + user_vocab_size + item_vocab_size` |
Because LCG never emits 0 for a nonzero ID, the first row of each split table (`table[pad]`, `table[pad+uv]`, `table[pad+uv+iv]`) is not selected on the published path. Padding IDs stay at unified index `0` (a zero vector).
Table length is `65 + user_vocab_size + item_vocab_size + author_vocab_size` by `emb_size`, `float32`.
## RecsysBatch hash tensors
`create_dummy_batch_from_config` is the shape contract used at runner init:
| Field | Shape | Dtype in dummy / pipeline |
|---|---|---|
| `user_hashes` | `[B, num_user_hashes]` | `int32` |
| `history_post_hashes` | `[B, history_seq_len, num_item_hashes]` | `int32` |
| `history_author_hashes` | `[B, history_seq_len, num_author_hashes]` | `int32` |
| `candidate_post_hashes` | `[B, candidate_seq_len, num_item_hashes]` | `int32` |
| `candidate_author_hashes` | `[B, candidate_seq_len, num_author_hashes]` | `int32` |
| `user_ip_hashes` | optional | unused on the published path |
Matching `RecsysEmbeddings` ranks add `emb_size` on the last axis, for example `history_post_embeddings`: `[B, S, num_item_hashes, D]`.
Validity is **only** the first hash function:
| Reducer | Valid when |
|---|---|
| `block_user_reduce` | `user_hashes[:, 0] != 0` |
| `block_history_reduce` | `history_post_hashes[:, :, 0] != 0` |
| `block_candidate_reduce` | `candidate_post_hashes[:, :, 0] != 0` |
| Retrieval candidate tower | same first-hash test on `candidate_post_hashes` |
A row whose first hash is 0 is padding even if later hash functions are nonzero.
## Lookup before the transformer
### Published pipeline
`run_pipeline.py` gathers with numpy advanced indexing on the unified table:
```python
user_hashes = hash_user(np.array([user_id], dtype=np.uint64))
hist_post_h = hash_item(history_post_ids).reshape(1, hist_len, -1)
hist_author_h = hash_author(history_author_ids).reshape(1, hist_len, -1)
emb_batch = RecsysEmbeddings(
user_embeddings=ret_emb[user_hashes],
history_post_embeddings=ret_emb[hist_post_h],
history_author_embeddings=ret_emb[hist_author_h],
candidate_post_embeddings=...,
candidate_author_embeddings=...,
)
```
History IDs are written into zero-initialized `uint64` buffers of length `history_seq_len`. Unused tail slots remain `0`, hash to `0`, and look up the pad row.
Ranking walks retrieved posts in chunks of `candidate_seq_len`. A short last chunk is `np.pad(..., constant 0)` on the hash tensors so the ranker still sees a full candidate axis; those pad hashes stay 0 and are masked out by `block_candidate_reduce`.
### Reduce, then transformer
`PhoenixModel.build_inputs` does not index any embedding table. It:
1. Embeds `history_product_surface` and `candidate_product_surface` from a learned `product_surface_embedding_table` (vocab default 16).
2. Projects multi-hot `history_actions` through `action_projection`.
3. Calls `block_user_reduce` → concatenates the `num_user_hashes` vectors to `[B, 1, num_user_hashes * D]`, multiplies by `proj_mat_1`, optional IP add.
4. Calls `block_history_reduce` → concatenates flattened post hashes, author hashes, action embedding, product-surface embedding (plus optional dwell / age), multiplies by `proj_mat_3`.
5. Calls `block_candidate_reduce` → concatenates flattened post hashes, author hashes, product-surface embedding (plus optional post-age), multiplies by `proj_mat_2`.
6. Concatenates `[user, history, candidates]` on the sequence axis and hands that to the transformer.
Retrieval's user tower reuses `block_user_reduce` and `block_history_reduce`, then mean-pools the transformer outputs. The candidate tower does **not** use `block_candidate_reduce`. It concatenates `candidate_post_embeddings` and `candidate_author_embeddings` on the hash axis (`axis=2`) and runs `CandidateTower` (SiLU MLP when `enable_linear_proj=True`, which `build_model_config` sets for retrieval).
During retrieval user encoding, `run_pipeline.py` still materializes the candidate tower by concatenating 64 ghost-negative slots (`N_neg = 64`) with hash tensors of ones and zero embeddings. Those ones are only for Haiku parameter creation; corpus scoring uses precomputed `sports_corpus.npz` `candidate_representations`, not a live candidate-tower pass over the corpus.
### Demo path (no LCG)
`create_example_batch` samples hash indices in `[1, num_*_embeddings)` and synthesizes random embedding tensors of the matching rank. Hash `0` is reserved for padding; history tails are zeroed. `run_ranker.py` and `run_retrieval.py` use this path. They do not read `hash_params` or `embedding_tables.npz`.
Phoenix unit tests (`test_recsys_model.py`, `test_recsys_retrieval_model.py`) also go through `create_example_batch`. They do not assert LCG arithmetic or the pad-65 layout. Those live only in `run_pipeline.py`.
## Reconstruct from artifacts
After Git LFS checkout, unzip `phoenix/artifacts/oss-phoenix-artifacts.zip` so both `retrieval/` and `ranker/` contain `config.json` and `embedding_tables.npz`.
```python
ret_cfg = json.load(open("artifacts/oss-phoenix-artifacts/retrieval/config.json"))
rank_cfg = json.load(open("artifacts/oss-phoenix-artifacts/ranker/config.json"))
hash_user, hash_item, hash_author = build_hash_functions(ret_cfg)
rank_hash_user, rank_hash_item, rank_hash_author = build_hash_functions(rank_cfg)
ret_emb = build_unified_emb_table(load_embedding_table(".../retrieval/embedding_tables.npz"), ret_cfg)
rank_emb = build_unified_emb_table(load_embedding_table(".../ranker/embedding_tables.npz"), rank_cfg)
```
Pass `uint64` IDs into the hasher for that stage. Index the matching unified table. Reshape post/author hashes to `[1, seq, num_hashes]` before building `RecsysBatch` / `RecsysEmbeddings`.
History shorter than `history_seq_len` must hash to all-zero rows. Candidate chunks shorter than `candidate_seq_len` must pad hashes with 0, not with a repeated last ID. After lookup, pad rows are the zero vector at unified index 0.
## Constraints
| Constraint | Source behavior |
|---|---|
| Hash `0` is padding | LCG short-circuits on `id == 0`; reducers mask on first-hash `!= 0` |
| Do not reuse one hasher across stages | Retrieval and ranker each have their own `hash_params` and tables |
| `pad` is 65, not configurable | Changing it without rebuilding both hashers and the table misaligns every index |
| `int64` modular multiply | Required to match training overflow |
| First hash decides validity | Later hash functions cannot un-pad a row |
| Model does not look up IDs | Forgetting the gather and passing empty / random `RecsysEmbeddings` trains or scores the wrong vectors |
| Published vocabs are 1M × 3 | README mini config; confirm against the extracted `*_vocab_size` keys before allocating |
## Failure modes
| Symptom | Likely cause |
|---|---|
| All history embeddings are ~0 | History IDs were left as 0, or hashes were not pad-offset so they landed in the pad region |
| Ranked scores look like a different user | Retrieval hasher used against the ranker table, or the reverse |
| `IndexError` on `table[hashes]` | Hasher from one vocab size applied to a table built with another, or `pad` / offset skipped |
| Candidates in a short last chunk all score similarly | Last IDs were repeated instead of padding hashes with 0; first-hash validity then treats pads as real items |
| `proj_mat_*` shape errors at init | `HashConfig.num_*_hashes` does not match the last-but-one embedding axis produced by lookup |
| LFS / missing `config.json` | `hash_params` never load; see [Troubleshooting](/troubleshooting) |
## Next
`PhoenixModelConfig`, `PhoenixRetrievalModelConfig`, `RecsysBatch`, `RecsysEmbeddings`, and published `config.json` keys.
How the transformer attends after hash embeddings are reduced: candidates see user + history + self only.
`example_sequence.json` fields, history padding to `history_seq_len`, and the IDs that enter these hashers.
Load both checkpoints, hash the example sequence, retrieve from `sports_corpus.npz`, and rank.
---
## 09. Multi-action scoring
> Per-action logits, demo weighted sums in run_pipeline.py, and production WeightedScorer versus RankingScorer weight tables.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/09-multi-action-scoring.md
- Generated: 2026-08-13T19:44:47.522Z
### Source Files
- `phoenix/runners.py`
- `phoenix/run_pipeline.py`
- `home-mixer/scorers/weighted_scorer.rs`
- `home-mixer/scorers/ranking_scorer.rs`
- `phoenix/recsys_model.py`
- `home-mixer/scorers/phoenix_scorer.rs`
---
title: "Multi-action scoring"
description: "Per-action logits, demo weighted sums in run_pipeline.py, and production WeightedScorer versus RankingScorer weight tables."
---
The Phoenix ranker decodes each candidate into a vector of raw engagement logits `[B, C, num_actions]`. Local inference turns those logits into probabilities and collapses a small subset into a display score. Home Mixer first materializes named `PhoenixScores`, then `RankingScorer` applies a much larger weight table, an offset, author-diversity decay, and an out-of-network multiplier. Numeric production weights live in unpublished `crate::params` / feature-switch types; this checkout publishes the formulas, field names, and the demo coefficients only.
`PhoenixCandidatePipeline` wires `PhoenixScorer` → `RankingScorer` → `VMRanker`. `WeightedScorer`, `AuthorDiversityScorer`, and `OONScorer` remain in `home-mixer/scorers/` as compile-time or standalone variants. They are not constructed in the pipeline and are not exported from `scorers/mod.rs`. The root README still diagrams those standalone stages; the snapshot that ships here inlines diversity and OON inside `RankingScorer`.
## Scoring surfaces
```mermaid
flowchart TB
subgraph phoenix ["phoenix/recsys_model.py"]
unemb["unembeddings [emb_size, num_actions]"]
logits["logits [B, C, num_actions]"]
cont["continuous_preds = sigmoid(continuous_unembeddings)"]
unemb --> logits
end
subgraph demo ["Local demo"]
pipe["run_pipeline.py: sigmoid + 4-term sum"]
ranker["run_ranker.py: sigmoid, sort by p_favorite"]
end
subgraph mixer ["Home Mixer PhoenixCandidatePipeline"]
ps["PhoenixScorer → phoenix_scores"]
rs["RankingScorer → weighted_score + score"]
vm["VMRanker if EnableVMRanker → score"]
sel["TopKScoreSelector reads score"]
ps --> rs --> vm --> sel
end
logits --> pipe
logits --> ranker
logits -.->|"production prediction service"| ps
```
| Surface | Input | Collapse | Sort / select key |
|---|---|---|---|
| `RecsysModel.__call__` | candidate token after the transformer | none — raw logits plus a separate continuous head | n/a |
| `run_pipeline.py` | published ranker logits | `fav*1.0 + reply*0.5 + rt*0.3 + dwell*0.2` | that weighted sum |
| `RecsysInferenceRunner.rank` / `run_ranker.py` | in-process ranker logits | none | `probs[:, :, 0]` (`favorite_score` in `ACTIONS` order) |
| `PhoenixScorer` | remote `predict` response | none | writes `phoenix_scores` only |
| `WeightedScorer` | `phoenix_scores` | compile-time `p::*` weights + offset | writes `weighted_score` only; not wired |
| `RankingScorer` | `phoenix_scores` + `query.params` | feature-switch weights + offset + diversity + OON | `weighted_score` then final `score` |
## Logit head
`PhoenixModelConfig.num_actions` sizes both the history action vector and the discrete head. History actions are signed (`2 * actions - 1`) and projected with `action_projection` of shape `[num_actions, emb_size]`. After the transformer and layer-norm, candidate tokens (everything from `candidate_start_offset` onward) are dotted with `unembeddings` of shape `[emb_size, num_actions]`.
A second head, `continuous_unembeddings` `[emb_size, num_continuous_actions]`, produces `continuous_preds` after a sigmoid. The default `num_continuous_actions` is `8`. Discrete logits stay raw; callers apply `jax.nn.sigmoid` when they need probabilities.
The published mini ranker documents **19** action types. `run_pipeline.py` reads `num_actions` from `ranker/config.json` / `retrieval/config.json`.
Last dimension of `RecsysModelOutput.logits` and of `history_actions`. Sourced from the artifact `config.json` in `run_pipeline.py`; set to `len(ACTIONS)` (19) in `run_ranker.py`.
Width of `continuous_preds`. Default `8`. Labels in `runners.CONTINUOUS_ACTIONS`: `reserved`, `dwell_time`, `video_watch_time`, `scroll_depth`, `reserved_3` … `reserved_6`.
## Two action-index spaces
Do not mix these last-dimension conventions.
| Space | Where | Favorite | Reply | Quote | Repost | Dwell | VQV |
|---|---|---|---|---|---|---|---|
| Proto `ActionName` | `example_sequence.json` keys, `run_pipeline.py` `IDX_*` | `1` | `4` | `5` | `6` | `11` | `13` |
| Dense `ACTIONS` | `runners.py` `RankingOutput`, `run_ranker.py` | `0` | `1` | `11` | `2` | `10` | `6` |
`run_pipeline.py` writes `history_actions[i, int(key)]` from the JSON `actions` map and later reads `all_probs[:, IDX_*]` with the proto indices. `RecsysInferenceRunner.hk_rank_candidates` maps `probs[:, :, 0..18]` onto `p_favorite_score` … `p_dwell_time` in `ACTIONS` order and sorts by index `0`. Negative-feedback slots in that dense layout are `14–17` (`NEGATIVE_FEEDBACK_INDICES`): not-interested, block, mute, report.
Home Mixer never indexes the logit tensor. `PhoenixScorer` copies a named `PhoenixScores` struct from `predictions.candidate_scores(original_tweet_id)`.
## Local demo weighted sum
After retrieval, `run_pipeline.py` ranks candidates in chunks of `candidate_seq_len`, applies `jax.nn.sigmoid` to `out.logits`, concatenates the real (unpadded) rows, then:
```python
weighted = (
all_probs[:, IDX_FAV] * 1.0
+ all_probs[:, IDX_REPLY] * 0.5
+ all_probs[:, IDX_RT] * 0.3
+ all_probs[:, IDX_DWELL] * 0.2
)
ranked = np.argsort(-weighted)
```
`IDX_QUOTE` (`5`) and `IDX_VQV` (`13`) are defined and VQV is printed, but neither enters the demo sum. There is no negative-feedback term, no offset, and no author-diversity or OON multiplier.
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file artifacts/oss-phoenix-artifacts/example_sequence.json \
--top_k_retrieval 200 \
--top_k_display 30
```
```text
PIPELINE RESULTS — User
History: N items | Corpus: M posts
Retrieved top 200 → Ranked by engagement model
Rank Score Ret Fav Reply RT Dwell VQV Topics Post URL
1 0.12.. 0.0.. 0.0.. 0.0.. 0.0.. 0.0.. 0.0.. ... https://x.com/a/status/
Weighted score range: [low, high]
```
`Score` is the four-term sum. `Ret` is the retrieval dot product, not an engagement logit. Success is a non-empty table plus that range line.
`run_ranker.py` does not use this sum. It calls `RecsysInferenceRunner.rank` on a synthetic batch and prints every `ACTIONS` probability, ordered by predicted favorite.
## Production scorer sequence
`PhoenixCandidatePipeline` runs scorers in order. Each scorer must return one result per input candidate in the same order; a length mismatch becomes `Scorer length_mismatch` and the stage is skipped.
### PhoenixScorer
`!query.has_cached_posts`. Cached-post requests skip inference.
If `None`, every candidate gets `PostCandidate::default()` — empty `phoenix_scores`.
`HomeTimelineRankedFollowing` when `query.in_network_only`, otherwise `HomeTimelineRanking`.
On success, `update` copies `phoenix_scores`, `prediction_request_id`, and `last_scored_at_ms`. A failed `predict` (after optional egress fallback) returns `Err("Phoenix prediction failed: …")` for every candidate.
### RankingScorer
Always enabled. For each candidate:
1. `combined = Σ apply(score_field, weight)` with missing scores as `0.0`.
2. `offset_score(combined)`.
3. `normalize_score(candidate, raw)` → stored as `weighted_score`.
4. Author-diversity multiplier on those weighted values (score-desc order, per-author occurrence count).
5. If `in_network == Some(false)`, multiply by `effective_oon_weight`.
6. Write `score` as that final value.
`TopKScoreSelector` sorts on `candidate.score`, not `weighted_score`.
### VMRanker
Enabled by `EnableVMRanker`. Forwards the named Phoenix fields plus `vqv_ineligible` into an unpublished ranker and may overwrite `score`.
## Weight tables
Production numeric values are **not** in this checkout. `RankingScorer` reads them from `query.params`. `WeightedScorer` reads compile-time `p::*` constants from the same unpublished `params` module.
| Phoenix field | RankingScorer param | WeightedScorer constant | In demo sum |
|---|---|---|---|
| `favorite_score` | `FavoriteWeight` | `FAVORITE_WEIGHT` | yes (`1.0` at proto index `1`) |
| `reply_score` | `ReplyWeight` | `REPLY_WEIGHT` | yes (`0.5` at `4`) |
| `retweet_score` | `RetweetWeight` | `RETWEET_WEIGHT` | yes (`0.3` at `6`) |
| `photo_expand_score` | `PhotoExpandWeight` | `PHOTO_EXPAND_WEIGHT` | no |
| `click_score` | `ClickWeight` | `CLICK_WEIGHT` | no |
| `profile_click_score` | `ProfileClickWeight` | `PROFILE_CLICK_WEIGHT` | no |
| `vqv_score` | `VqvWeight` (gated) | `VQV_WEIGHT` (gated) | printed only |
| `share_score` | `ShareWeight` | `SHARE_WEIGHT` | no |
| `share_via_dm_score` | `ShareViaDmWeight` | `SHARE_VIA_DM_WEIGHT` | no |
| `share_via_copy_link_score` | `ShareViaCopyLinkWeight` | `SHARE_VIA_COPY_LINK_WEIGHT` | no |
| `dwell_score` | `DwellWeight` | `DWELL_WEIGHT` | yes (`0.2` at `11`) |
| `quote_score` | `QuoteWeight` | `QUOTE_WEIGHT` | no (`IDX_QUOTE` unused) |
| `quoted_click_score` | `QuotedClickWeight` | `QUOTED_CLICK_WEIGHT` | no |
| `quoted_vqv_score` | `QuotedVqvWeight` (gated) | — | no |
| `dwell_time` | `ContDwellTimeWeight` | `CONT_DWELL_TIME_WEIGHT` | no |
| `click_dwell_time` | `ContClickDwellTimeWeight` | — | no |
| `follow_author_score` | `FollowAuthorWeight` | `FOLLOW_AUTHOR_WEIGHT` | no |
| `not_interested_score` | `NotInterestedWeight` | `NOT_INTERESTED_WEIGHT` | no |
| `block_author_score` | `BlockAuthorWeight` | `BLOCK_AUTHOR_WEIGHT` | no |
| `mute_author_score` | `MuteAuthorWeight` | `MUTE_AUTHOR_WEIGHT` | no |
| `report_score` | `ReportWeight` | `REPORT_WEIGHT` | no |
| `not_dwelled_score` | `NotDwelledWeight` | — | no |
`RankingScorer` therefore includes three fields `WeightedScorer` does not: `quoted_vqv_score`, `click_dwell_time`, `not_dwelled_score`. The local `home-mixer/candidate_pipeline/candidate.rs` `PhoenixScores` snapshot also omits those three; the production struct comes from `xai_candidate_pipeline::component_library::models::PhoenixScores`.
Positive mass for `RankingScorer` is the sum of favorite, reply, retweet, photo-expand, click, profile-click, VQV, share, share-via-DM, share-via-copy-link, dwell, quote, quoted-click, quoted-VQV, and follow-author. Negative mass is `-(not_interested + block + mute + report + not_dwelled)`. `total_sum = positive_sum + negative_sum`.
`WeightedScorer` uses precomputed `WEIGHTS_SUM` and `NEGATIVE_WEIGHTS_SUM` instead of summing at request time.
## Offset, VQV gates, diversity, OON
Both combiners share the same offset shape (`NEGATIVE_SCORES_OFFSET` is unpublished):
```text
if total_sum == 0: max(combined, 0)
elif combined < 0: (combined + negative_sum) / total_sum * NEGATIVE_SCORES_OFFSET
else: combined + NEGATIVE_SCORES_OFFSET
```
VQV is not a flat multiply:
- `WeightedScorer`: `VQV_WEIGHT` only when `video_duration_ms > MIN_VIDEO_DURATION_MS`; otherwise `0.0`.
- `RankingScorer`: `candidates_util::vqv_weight(query, candidate, MinVideoDurationMs, VqvWeight)` and `quoted_vqv_weight(..., EnableQuotedVqvDurationCheck)`.
Author diversity (inlined in `RankingScorer`, also implemented standalone in `AuthorDiversityScorer`):
```text
multiplier(position) = (1 - floor) * decay^position + floor
```
`RankingScorer` reads `AuthorDiversityDecay` and `AuthorDiversityFloor` from the query. Candidates are visited in descending weighted-score order; `position` is how many earlier posts from the same `author_id` were already seen. The first post from an author keeps multiplier `1.0` when `position == 0`.
OON (inlined; standalone `OONScorer` only multiplies `score` by `OON_WEIGHT_FACTOR`):
| Condition | Factor |
|---|---|
| `topic_ids` non-empty | `TopicOonWeightFactor` |
| viewer age `< NewUserAgeThresholdSecs` and `followed_user_ids.len() >= NEW_USER_MIN_FOLLOWING` | `NEW_USER_OON_WEIGHT_FACTOR` |
| otherwise | `OonWeightFactor` |
Applied only when `in_network == Some(false)`.
## Constraints and failure modes
- Scorers must preserve length and order. Drops belong in filters, not scorers.
- Missing `PhoenixScores` fields contribute `0.0`.
- No `scoring_sequence` → empty scores → weighted/final scores collapse toward the offset of a zero combined value.
- Phoenix `predict` error → every candidate `Err`; later scorers see whatever `update` skipped.
- Cached posts skip `PhoenixScorer` entirely.
- Demo `weighted` and production `score` are not comparable: different index space, different terms, unpublished production weights, plus offset / diversity / OON.
- `run_ranker.py` success is a favorite-ordered probability dump, not the four-term demo table.
`params` (`FAVORITE_WEIGHT`, `FavoriteWeight`, `NEGATIVE_SCORES_OFFSET`, `NEW_USER_OON_WEIGHT_FACTOR`, …) is not in this tree. You cannot reproduce production `score` from the published ranker logits alone.
## Next
Proto ActionName values, IDX_* constants, ACTIONS labels, and scorer field keys.
Phoenix cluster and egress fallback, VM ranker, TopKScoreSelector.
Load checkpoints, retrieve from sports_corpus.npz, print the ranked table.
Why per-candidate logits stay batch-independent.
PhoenixModelConfig, RecsysBatch, and published config.json keys including num_actions.
---
## 10. Run the inference pipeline
> Load retrieval and ranker checkpoints, encode example_sequence.json, retrieve from sports_corpus.npz, and rank with the published action weights.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/10-run-the-inference-pipeline.md
- Generated: 2026-08-13T19:44:28.570Z
### Source Files
- `phoenix/run_pipeline.py`
- `phoenix/README.md`
- `phoenix/runners.py`
- `phoenix/recsys_model.py`
- `phoenix/recsys_retrieval_model.py`
- `phoenix/run_ranker.py`
- `phoenix/run_retrieval.py`
---
title: "Run the inference pipeline"
description: "Load retrieval and ranker checkpoints, encode example_sequence.json, retrieve from sports_corpus.npz, and rank with the published action weights."
---
`phoenix/run_pipeline.py` is the local retrieval-then-rank entry point. It loads the exported Phoenix checkpoints, encodes a user action sequence, retrieves from the precomputed sports corpus, scores the retrieved posts with the ranker, and prints a weighted engagement table. It does not start Home Mixer, Thunder, or a Phoenix prediction cluster.
`phoenix/artifacts/oss-phoenix-artifacts.zip` is a Git LFS object (about 2.90 GB). If the file is a 135-byte `version https://git-lfs.github.com/spec/v1` pointer, `unzip` will fail. Pull LFS objects before extract. See [Installation](/installation) and [Troubleshooting](/troubleshooting).
## Prerequisites
- Python 3.11+ in `phoenix/` (`requires-python = ">=3.11"` in `phoenix/pyproject.toml`)
- Dependencies from that project: `jax==0.8.1`, `dm-haiku`, `numpy`
- Extracted artifact tree (not the LFS pointer)
```bash
cd phoenix
uv sync
```
```bash
cd phoenix
pip install jax==0.8.1 dm-haiku numpy
```
## Artifact layout
Extract the archive so the runner can see sibling `retrieval/`, `ranker/`, `sports_corpus.npz`, and `example_sequence.json` directories/files:
```bash
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
:::files
artifacts/oss-phoenix-artifacts/
retrieval/
model_params.npz
embedding_tables.npz
config.json
ranker/
model_params.npz
embedding_tables.npz
config.json
sports_corpus.npz
example_sequence.json
:::
The argparse default `--artifacts_dir` is `./artifacts`. After the documented unzip, pass the nested directory:
```bash
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
| Path | Role |
| --- | --- |
| `retrieval/config.json` | Retrieval architecture, vocab sizes, `hash_params` |
| `retrieval/model_params.npz` | Haiku weights for the user tower, candidate tower, and `log_temperature` |
| `retrieval/embedding_tables.npz` | `user_embeddings`, `item_embeddings`, `author_embeddings` |
| `ranker/config.json` | Ranker architecture, `num_actions`, hash params |
| `ranker/model_params.npz` | Ranking transformer + action unembedding |
| `ranker/embedding_tables.npz` | Separate ranker hash tables |
| `sports_corpus.npz` | Precomputed candidate representations |
| `example_sequence.json` | Viewer `user_id` and history items |
Phoenix documents the sports corpus as about 537K Sports-topic posts from a six-hour window, with `example_sequence.json` as three liked/dwelled sports posts (NFL, NBA, NHL). Confirm counts from the extracted NPZ and JSON; the runner logs them at startup.
## CLI
Directory that contains `retrieval/`, `ranker/`, and the default sequence/corpus files.
User sequence JSON. Default: `{artifacts_dir}/example_sequence.json`.
Corpus NPZ. Default: `{artifacts_dir}/sports_corpus.npz`.
How many corpus hits to keep before ranking. Clamped to `len(post_ids)`.
How many ranked rows to print. Clamped to the retrieved count.
```bash uv
cd phoenix
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--top_k_retrieval 200 \
--top_k_display 30
```
```bash python
cd phoenix
python run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--top_k_retrieval 200 \
--top_k_display 30
```
Override sequence or corpus without moving files:
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file /path/to/sequence.json \
--corpus_file /path/to/sports_corpus.npz
```
## Run the pipeline
`oss-phoenix-artifacts.zip` must be a zip, not an LFS pointer. Then extract into `phoenix/artifacts/` so `artifacts/oss-phoenix-artifacts/retrieval/config.json` exists.
From `phoenix/`, run `uv sync` (or pip-install `jax==0.8.1`, `dm-haiku`, `numpy`).
Pass `--artifacts_dir artifacts/oss-phoenix-artifacts`. Optional: `--top_k_retrieval`, `--top_k_display`, `--sequence_file`, `--corpus_file`.
Success is stdout headed `PIPELINE RESULTS — User {user_id}` with `Retrieved top K → Ranked by engagement model`, per-row scores, and a final `Weighted score range: [min, max]`. Logs also print user-repr L2 norm and retrieval score range.
## What the runner does
`run_pipeline.py` does **not** use `RecsysRetrievalInferenceRunner` or `RecsysInferenceRunner`. Those classes power the random-init demos in `run_retrieval.py` / `run_ranker.py` and the unit tests. The artifact runner builds its own Haiku transforms and applies `load_model_params` checkpoints.
```mermaid
flowchart LR
subgraph artifacts["artifacts/oss-phoenix-artifacts"]
retFiles["retrieval/config.json model_params.npz embedding_tables.npz"]
rankFiles["ranker/config.json model_params.npz embedding_tables.npz"]
seq["example_sequence.json"]
corpus["sports_corpus.npz"]
end
subgraph encode["Encode sequence"]
hashFn["build_hash_functions"]
padHist["history[:history_seq_len] right-pad with zeros"]
lookup["unified embedding table lookup"]
end
subgraph retrieve["Retrieval"]
userTower["PhoenixRetrievalModel.build_user_representation"]
dot["corpus_repr @ user_repr"]
topk["argpartition then argsort"]
end
subgraph rank["Ranking"]
chunks["chunks of candidate_seq_len"]
phoenix["PhoenixModel logits"]
sig["jax.nn.sigmoid"]
wsum["fav*1 + reply*0.5 + rt*0.3 + dwell*0.2"]
end
seq --> padHist
retFiles --> hashFn
rankFiles --> hashFn
hashFn --> lookup
padHist --> lookup
lookup --> userTower
retFiles --> userTower
corpus --> dot
userTower --> dot
dot --> topk
topk --> chunks
rankFiles --> phoenix
chunks --> phoenix
phoenix --> sig
sig --> wsum
```
### 1. Load configs and tables
The runner reads `retrieval/config.json` and `ranker/config.json`, then:
- `load_model_params` — NPZ keys are `module/path/param`, reassembled into a Haiku dict
- `load_embedding_table` — raw `user_embeddings` / `item_embeddings` / `author_embeddings`
- `build_unified_emb_table` — concatenates those three tables behind a pad of **65** rows: `[pad | user | item | author]`
`build_model_config` reconstructs `PhoenixRetrievalModelConfig` or `PhoenixModelConfig` from JSON. Retrieval always sets `enable_linear_proj=True`. Both configs hardcode transformer `widening_factor=2.0` and `attn_output_multiplier=0.125`. Ranker also reads `num_actions` and `post_age_granularity_mins` (default 60). History action-vector width comes from **`ret_cfg["num_actions"]`**.
Architecture numbers live in the extracted `config.json` files. `phoenix/README.md` documents a mini snapshot (128-d, 4 layers, 4 heads, `history_seq_len` 127, `candidate_seq_len` 64, 19 actions). The repo root README names a different snapshot (256-d, 2 layers). Trust the JSON next to the checkpoints.
### 2. Encode the user sequence
Expected JSON:
```json example_sequence.json
{
"user_id": 123,
"history": [
{
"post_id": 1987654321098765432,
"author_id": 456,
"actions": {
"1": 1.0,
"11": 1.0
}
}
]
}
```
| Field | Constraint |
| --- | --- |
| `user_id` | Integer viewer id; hashed with the user hash family |
| `history` | List; only the first `history_seq_len` items are used |
| `history[].post_id` | Placed at index `i`; unused slots stay `0` (padding) |
| `history[].author_id` | Same alignment as `post_id` |
| `history[].actions` | Map of `ActionName` index string → float; written into `history_actions[i, idx]` when `idx < num_actions` |
Hashing is the training linear-congruential map (`id * scale + bias) % modulus`), then bucketed into `[1, vocab)` with `0` reserved for pad, then offset into the unified table (`user + 65`, `item + 65 + user_vocab`, `author + 65 + user_vocab + item_vocab`). Retrieval and ranker each use their own `hash_params`.
This path always sets `history_product_surface` and `candidate_product_surface` to zeros. It does not pass `candidate_impr_ts` / `candidate_post_creation_ts`, so ranker post-age buckets are the missing/zero bucket.
Field-level sequence editing belongs on [Customize a user sequence](/customize-user-sequence). Index tables belong on [Action indices](/action-indices).
### 3. Retrieve from `sports_corpus.npz`
Required NPZ keys:
| Array | Use |
| --- | --- |
| `post_ids` | Displayed as `https://x.com/a/status/{id}` |
| `author_ids` | Ranker author hashes |
| `candidate_representations` | Precomputed L2-normalized candidate vectors `[N, D]` |
| `topics` | Optional; missing → empty strings |
The user tower is `PhoenixRetrievalModel.build_user_representation`: user + history through the transformer, masked mean pool, then L2 normalize. The forward also instantiates the candidate tower and `log_temperature` against 64 dummy “graph-negative” slots so those checkpoint parameters bind; those dummies are not scored.
Corpus search is NumPy, not `PhoenixRetrievalModel.__call__` / `jax.lax.top_k`:
```text
scores = corpus_repr @ user_repr[0]
top_idx = argpartition(scores, -K)[-K:]
top_idx = top_idx[argsort(-scores[top_idx])]
```
`K = min(--top_k_retrieval, N)`. The log line is `Retrieved K (score range: low - high)`.
### 4. Rank retrieved posts
The ranker is `PhoenixModel`: user + history + candidates, candidate-isolation attention, then unembedding to `[B, candidate_seq_len, num_actions]` logits. Retrieved hits are scored in chunks of `candidate_seq_len`. A short final chunk is zero-padded; only the real `cs` rows are kept.
```text
probs = sigmoid(logits)
weighted = fav*1.0 + reply*0.5 + retweet*0.3 + dwell*0.2
ranked = argsort(-weighted)
```
Columns used in that sum are proto `ActionName` indices, not `runners.ACTIONS` positions 0..18:
| Constant | Index | Weight in this runner |
| --- | --- | --- |
| `IDX_FAV` (`SERVER_TWEET_FAV`) | 1 | 1.0 |
| `IDX_REPLY` (`SERVER_TWEET_REPLY`) | 4 | 0.5 |
| `IDX_RT` (`SERVER_TWEET_RETWEET`) | 6 | 0.3 |
| `IDX_DWELL` (`CLIENT_TWEET_RECAP_DWELLED`) | 11 | 0.2 |
| `IDX_VQV` (`CLIENT_TWEET_VIDEO_QUALITY_VIEW`) | 13 | printed only |
| `IDX_QUOTE` (`SERVER_TWEET_QUOTE`) | 5 | unused in the sum |
This is the **demo** combiner. Production `WeightedScorer` applies a larger signed weight table (including negative feedback), VQV eligibility, and an offset/normalization step. Do not treat the four coefficients above as Home Mixer weights. See [Multi-action scoring](/multi-action-scoring) and [Scorers and weights](/scorers-and-weights).
`RecsysInferenceRunner.rank` is a different combiner: it sigmoids logits and sorts by column **0** (`favorite_score` in `ACTIONS`). The artifact pipeline does not call it.
## Output
```text
========================================================================================================================
PIPELINE RESULTS — User 123
History: 3 items | Corpus: 537000 posts
Retrieved top 200 → Ranked by engagement model
========================================================================================================================
Rank Score Ret Fav Reply RT Dwell VQV Topics Post URL
------------------------------------------------------------------------------------------------------------------------
1 0.1234 0.2100 0.0800 0.0200 0.0100 0.1500 0.0050 Sports https://x.com/a/status/…
…
Weighted score range: [0.0100, 0.1234]
========================================================================================================================
```
| Column | Source |
| --- | --- |
| `Rank` | Order of `-weighted` |
| `Score` | Demo weighted sum |
| `Ret` | Retrieval dot product |
| `Fav` / `Reply` / `RT` / `Dwell` / `VQV` | `sigmoid(logits)` at the `IDX_*` columns |
| `Topics` | First 28 characters of `topics[i]` |
| `Post URL` | `https://x.com/a/status/{post_id}` |
Log lines to expect before the table: `Loading retrieval model...`, `Loading ranker model...`, `Loading corpus...` (`N posts, repr shape ...`), `User {id}, H history items`, `User repr norm=...`, `Retrieved K (score range: ...)`, `Ranking K candidates...`.
## `config.json` keys the runner reads
| Key | Used for |
| --- | --- |
| `emb_size`, `key_size`, `num_heads`, `num_layers` | `TransformerConfig` (`num_q_heads` = `num_kv_heads` = `num_heads`) |
| `history_seq_len`, `candidate_seq_len` | Sequence pad width and ranker chunk size |
| `num_user_hashes`, `num_item_hashes`, `num_author_hashes` | `HashConfig` |
| `user_vocab_size`, `item_vocab_size`, `author_vocab_size` | Unified table layout and bucket counts |
| `hash_params.{user,item,author}_{hash_scales,biases,modulus}` | Linear-congruential hashes |
| `num_actions` | History / logit width (history width from retrieval JSON) |
| `product_surface_vocab_size` | Optional; default 16 |
| `post_age_granularity_mins` | Ranker only; default 60 |
Full field semantics: [Phoenix model configuration](/phoenix-model-configuration).
## Related scripts (not this path)
| Script | What it actually runs |
| --- | --- |
| `run_pipeline.py` | Published checkpoints + sports corpus + sequence JSON |
| `run_retrieval.py` | Random-init `RecsysRetrievalInferenceRunner` + `create_example_corpus(1000)` |
| `run_ranker.py` | Random-init `RecsysInferenceRunner` + `create_example_batch`; sorts by favorite (column 0) |
`uv run pytest test_recsys_model.py test_recsys_retrieval_model.py` exercises attention masks, L2-normalized towers, and top-k shape/order on synthetic tensors. It does not load `oss-phoenix-artifacts`. See [Test Phoenix](/test-phoenix).
## Constraints and failure modes
| Symptom | Cause |
| --- | --- |
| `unzip` fails or archive is 135 bytes | LFS pointer not pulled |
| `FileNotFoundError` on `retrieval/config.json` | `--artifacts_dir` points at `artifacts/` instead of `artifacts/oss-phoenix-artifacts/` |
| Missing `sports_corpus.npz` / `example_sequence.json` | Defaults resolve under `--artifacts_dir` |
| `KeyError` on `post_ids` / `candidate_representations` / `author_ids` | Corpus NPZ is not the published sports dump |
| Haiku apply / shape errors | `config.json` does not match `model_params.npz`, or embedding table vocabs do not match hash offsets |
| Empty or all-zero history after the first `history_seq_len` items | Extra history is dropped; pad slots stay 0 |
| Ranked order disagrees with `run_ranker.py` | Different index scheme and different sort key |
This checkout does not serve production For You. Home Mixer `PhoenixScorer` calls an unpublished prediction client with a hydrated `scoring_sequence`. Local ranking here is a frozen mini checkpoint plus a four-term printout. See [Runtime boundaries](/runtime-boundaries).
## Next
`example_sequence.json` fields, ActionName indices, history padding, and `--top_k_*` knobs.
Per-action logits, this runner’s weighted sum, and production weight tables.
`PhoenixModelConfig`, `PhoenixRetrievalModelConfig`, `RecsysBatch`, and published `config.json` keys.
LFS pointer failures, artifact paths, and other local run breakages.
---
## 11. Customize a user sequence
> example_sequence.json fields, ActionName indices, history padding to history_seq_len, and --top_k_retrieval / --top_k_display knobs.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/11-customize-a-user-sequence.md
- Generated: 2026-08-13T19:44:47.915Z
### Source Files
- `phoenix/run_pipeline.py`
- `phoenix/README.md`
- `phoenix/runners.py`
- `phoenix/recsys_model.py`
- `home-mixer/query_hydrators/scoring_sequence_query_hydrator.rs`
- `home-mixer/query_hydrators/retrieval_sequence_query_hydrator.rs`
---
title: "Customize a user sequence"
description: "example_sequence.json fields, ActionName indices, history padding to history_seq_len, and --top_k_retrieval / --top_k_display knobs."
---
`phoenix/run_pipeline.py` is the only local entry point that encodes a user action sequence from JSON. It reads `{artifacts_dir}/example_sequence.json` (or `--sequence_file`), left-pads the `history` array to the retrieval checkpoint's `history_seq_len`, hashes `user_id` / `post_id` / `author_id` into a `RecsysBatch`, retrieves `--top_k_retrieval` posts from `sports_corpus.npz`, and prints `--top_k_display` rows of the ranked table.
`phoenix/run_ranker.py` and `phoenix/run_retrieval.py` do **not** read this file. They build a random `create_example_batch` and are not the customization surface.
The shipped `example_sequence.json` lives inside the Git LFS archive `phoenix/artifacts/oss-phoenix-artifacts.zip`. Extract it before editing. The pointer file is not a zip.
## Prerequisites
The sequence file is created by unzipping the Phoenix archive:
```bash
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
Expected path: `artifacts/oss-phoenix-artifacts/example_sequence.json`. The shipped sample is three sports posts (NFL, NBA, NHL) that the user favorited and dwelled on.
```bash
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
Success is a `PIPELINE RESULTS — User ` table. See [Run the inference pipeline](/run-inference-pipeline) for checkpoint layout and [Installation](/installation) if the zip is still an LFS pointer.
## Sequence file schema
The loader does `json.load` and then requires two top-level keys. Extra keys are ignored. `history_product_surface` and `history_continuous_actions` are **not** read from JSON; the pipeline zeros product-surface IDs and omits continuous channels.
```json
{
"user_id": 1234567890123456789,
"history": [
{
"post_id": 1900000000000000001,
"author_id": 44196397,
"actions": {
"1": 1.0,
"11": 1.0
}
}
]
}
```
Hashed with the retrieval and ranker `hash_params` into `RecsysBatch.user_hashes`. Value `0` is reserved as padding by `_hash_ids` and produces a masked user token.
Ordered list of interaction objects. Only the prefix `history[:history_seq_len]` is encoded. Later items are dropped.
Item ID hashed into `history_post_hashes`. Value `0` stays a pad slot; `block_history_reduce` treats `history_post_hashes[:, :, 0] == 0` as padding.
Author ID hashed into `history_author_hashes`. Same pad rule as `post_id`.
Map of `ActionName` index (JSON string key) to a float value. Missing object is treated as `{}`. Keys with `int(key) >= num_actions` are dropped. Typical present value is `1.0`.
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file /tmp/my_sequence.json \
--top_k_retrieval 200 \
--top_k_display 30
```
```text
PIPELINE RESULTS — User 1234567890123456789
History: 1 items | Corpus: posts
Retrieved top 200 → Ranked by engagement model
```
The `History:` line prints `len(history)` from the JSON, not the padded tensor length.
## ActionName indices
`actions` keys are proto `ActionName` integers, the same indices `run_pipeline.py` uses for both history encoding and the printed ranker columns:
| Index | Constant | Comment in `run_pipeline.py` | Role in the demo |
|---|---|---|---|
| `1` | `IDX_FAV` | `SERVER_TWEET_FAV` | History bit + weighted score `× 1.0` + `Fav` column |
| `4` | `IDX_REPLY` | `SERVER_TWEET_REPLY` | History bit + weighted score `× 0.5` + `Reply` column |
| `5` | `IDX_QUOTE` | `SERVER_TWEET_QUOTE` | History bit only. Defined, not used in the demo weighted sum |
| `6` | `IDX_RT` | `SERVER_TWEET_RETWEET` | History bit + weighted score `× 0.3` + `RT` column |
| `11` | `IDX_DWELL` | `CLIENT_TWEET_RECAP_DWELLED` | History bit + weighted score `× 0.2` + `Dwell` column |
| `13` | `IDX_VQV` | `CLIENT_TWEET_VIDEO_QUALITY_VIEW` | History bit + `VQV` column. Not in the demo weighted sum |
`num_actions` comes from `retrieval/config.json` (published mini table: **19**). Index `0` is never written by the documented sample keys.
The action embedding is a signed multi-hot: `(2 * actions - 1)`, then zeroed when the entire vector is `0` (a pad or empty `actions` object). A real history item with only `{"1": 1.0}` therefore contributes `+1` at favorite and `-1` at every other action slot.
Do not reuse `runners.ACTIONS` positions (`favorite_score` at 0, `reply_score` at 1, …) as `example_sequence.json` keys. That list is the `RecsysInferenceRunner` output layout for `run_ranker.py`, not the proto `ActionName` encoding. Full mapping lives on [Action indices](/action-indices).
## History padding and truncation
`hist_len` is `ret_cfg["history_seq_len"]` from `artifacts/.../retrieval/config.json`. The published mini-model table documents **127**. Dataclass defaults on `PhoenixModelConfig` / `PhoenixRetrievalModelConfig` (`128`) are not used once a checkpoint config is loaded.
```text
JSON history length N, encoder width H = history_seq_len
history[:H] copied left-to-right
remaining slots stay 0 (pad post_id, author_id, actions)
N <= H : [item0, item1, ..., itemN-1, 0, 0, ..., 0]
N > H : [item0, item1, ..., itemH-1] # suffix discarded
```
Pad slots hash to `0` and are masked out (`history_post_hashes[:, :, 0] != 0`). They do not attend as real history.
This prefix keep is the opposite of a last-N trim. Put the interactions you want the model to see at the **front** of `history`, or pre-trim the array yourself.
The same padded `history_post_ids` / `history_author_ids` / `history_actions` tensors are reused for retrieval and ranking. Retrieval and ranker hash functions are built separately from each checkpoint's `hash_params`.
## How the sequence becomes a RecsysBatch
After padding, `run_pipeline.py` builds:
| `RecsysBatch` field | Source |
|---|---|
| `user_hashes` | `hash_user([user_id])` |
| `history_post_hashes` | `hash_item(history_post_ids)` shaped `[1, H, num_item_hashes]` |
| `history_author_hashes` | `hash_author(history_author_ids)` |
| `history_actions` | `[1, H, num_actions]` float32 from `actions` |
| `history_product_surface` | zeros `[1, H]` |
| `candidate_*` | zeros at retrieval time; filled with retrieved IDs at ranking time |
Embeddings are looked up from the reconstructed unified table **before** the transformer. Candidate isolation (candidates attend to user + history + self only) is unchanged by sequence edits. See [Hash embeddings](/hash-embeddings) and [Candidate isolation](/candidate-isolation).
## CLI knobs
These flags change retrieval depth and printed rows. They do not change encoder width, `num_actions`, or the JSON schema.
Default `./artifacts`. Must contain `retrieval/`, `ranker/`, and (unless overridden) `example_sequence.json` plus `sports_corpus.npz`.
Default `{artifacts_dir}/example_sequence.json`. Point this at an edited copy instead of overwriting the shipped sample.
Default `{artifacts_dir}/sports_corpus.npz`. Retrieved `post_id`s come from this corpus, not from `history`.
Default `200`. Effective `TOP_K = min(top_k_retrieval, len(corpus_post_ids))`. Dot-product of the user representation against `candidate_representations`, then `argpartition` + sort. Ranking walks this list in chunks of `candidate_seq_len` (published mini table: **64**).
Default `30`. Effective `DISPLAY = min(top_k_display, TOP_K)`. Only the print loop is truncated; every retrieved candidate is still scored.
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file /tmp/my_sequence.json \
--top_k_retrieval 500
```
Ranks 500 corpus hits (or the whole corpus if smaller). The table still shows 30 rows unless you also raise `--top_k_display`.
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file /tmp/my_sequence.json \
--top_k_display 50
```
Prints 50 ranked rows from the default 200 retrieved candidates.
Demo rank order is **not** the production `WeightedScorer` table. The pipeline uses:
```text
weighted = P(fav)*1.0 + P(reply)*0.5 + P(rt)*0.3 + P(dwell)*0.2
```
Quote and VQV probabilities are printed and can be driven by history bits, but they do not move this demo rank. Production weights are on [Multi-action scoring](/multi-action-scoring) and [Scorers and weights](/scorers-and-weights).
## Edit and verify
```bash
cp artifacts/oss-phoenix-artifacts/example_sequence.json /tmp/my_sequence.json
```
Keep `user_id` non-zero. Each history object needs integer `post_id` and `author_id` plus an `actions` map using the `ActionName` keys above. Multiple keys on one item are allowed (`"1"` and `"11"` together is the shipped “liked and dwelled” pattern).
If `len(history)` exceeds `history_seq_len` from `retrieval/config.json`, only the prefix is encoded. Trim or reorder first.
```bash
uv run run_pipeline.py \
--artifacts_dir artifacts/oss-phoenix-artifacts \
--sequence_file /tmp/my_sequence.json \
--top_k_retrieval 200 \
--top_k_display 30
```
Confirm `User` matches `user_id` and `History: N items` matches the JSON array length. Then inspect `Fav` / `Reply` / `RT` / `Dwell` / `VQV` and the `Score` column. Changing only `--top_k_display` must not change the first printed rows; changing `--top_k_retrieval` can, because a different candidate pool is ranked.
History IDs do not have to appear in `sports_corpus.npz`. Corpus posts are retrieved by embedding similarity; history only conditions the user tower.
## Production analog
Local JSON replaces two Home Mixer query hydrators that `PhoenixCandidatePipeline` registers first:
| Local field | Production field on `ScoredPostsQuery` | Hydrator | Consumer |
|---|---|---|---|
| Same `history` tensor for both stages | `retrieval_sequence` (+ `columnar_retrieval_sequence`) | `RetrievalSequenceQueryHydrator` | `PhoenixSource` (errors with `PhoenixSource: missing retrieval_sequence`) |
| Same `history` tensor for both stages | `scoring_sequence` (+ `columnar_scoring_sequence`) | `ScoringSequenceQueryHydrator` | `PhoenixScorer` (returns default candidates if `scoring_sequence` is `None`) |
Both hydrators call `UserActionAggregationClient.fetch_aggregated_sequence` with `UAS_WINDOW_TIME_MS`. Retrieval uses `MaxSeqLengthRetrieval` and aggregation type `PhoenixRetrievalAggregationType` (fallback `Dense`). Scoring uses `MaxSeqLengthScoring` and `PhoenixAggregationType` (fallback `DenseWithNotInterestedIn`). Those params live in unpublished `home-mixer` crates; this checkout cannot fetch a live sequence.
Editing `example_sequence.json` does not change For You / Scored Posts gRPC behavior. Production scoring still requires a hydrated `scoring_sequence`. See [Assemble a Home Mixer request](/assemble-home-mixer-request) and [Troubleshooting](/troubleshooting).
## Constraints and errors
| Condition | Behavior |
|---|---|
| Missing `--sequence_file` and no `{artifacts_dir}/example_sequence.json` | `FileNotFoundError` on `open` |
| JSON missing `user_id` or `history` | `KeyError` |
| History item missing `post_id` or `author_id` | `KeyError` |
| `user_id`, `post_id`, or `author_id` is `0` | Slot hashes to pad `0` and is masked |
| `len(history) > history_seq_len` | Silent prefix truncate |
| Action key `>= num_actions` | Silent drop |
| Empty `actions` on a real `post_id` | Action embedding zeroed (`jnp.any` mask) |
| `--top_k_retrieval` larger than the corpus | Capped to corpus length |
| `--top_k_display` larger than `TOP_K` | Capped to `TOP_K` |
| Artifacts still an LFS pointer | Unzip fails; see [Troubleshooting](/troubleshooting) |
`run_pipeline.py` does not validate snowflake ID ranges, topic membership, or that history posts exist in the corpus.
## Next
Load retrieval and ranker checkpoints, encode the sequence, retrieve from `sports_corpus.npz`.
`ActionName` vs `IDX_*` vs `runners.ACTIONS` vs scorer weight keys.
Demo weighted sum versus production `WeightedScorer` / `RankingScorer`.
`history_seq_len`, `candidate_seq_len`, `num_actions`, and published `config.json` keys.
---
## 12. Assemble a Home Mixer request
> HomeMixerServer CLI flags, QueryBuilder.viewer_id validation, ScoredPostsQuery construction, and For You versus ScoredPosts entry points.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/12-assemble-a-home-mixer-request.md
- Generated: 2026-08-13T19:56:12.127Z
### Source Files
- `home-mixer/main.rs`
- `home-mixer/server.rs`
- `home-mixer/scored_posts_server.rs`
- `home-mixer/for_you_server.rs`
- `home-mixer/models/query.rs`
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`
- `home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs`
---
title: "Assemble a Home Mixer request"
description: "HomeMixerServer CLI flags, QueryBuilder.viewer_id validation, ScoredPostsQuery construction, and For You versus ScoredPosts entry points."
---
`HomeMixerServer` is the gRPC process started from `home-mixer/main.rs`. It parses CLI flags, boots `XServiceBuilder` under the service name `home-mixer`, and registers `ScoredPostsService` plus `ForYouFeedService`. Every inbound RPC that carries a `pb::ScoredPostsQuery` goes through `QueryBuilder::build`, which rejects `viewer_id == 0` with `INVALID_ARGUMENT` `"viewer_id must be specified"` and constructs the in-memory `models::query::ScoredPostsQuery` both pipelines execute.
This checkout does not ship `home-mixer` crate files such as `params`, `clients`, or `xai_home_mixer_proto`. The assembly path below is in source; the binary is not locally runnable. Use the Phoenix Python pipeline for on-box inference.
`HomeMixerServer` is a production gRPC surface. It requires mTLS (`TlsMode::server_mtls_from_env`), StringCenter, feature-switch, and decider bundles referenced by unpublished `params` constants. A missing or zero `viewer_id` never reaches a pipeline.
## Start the server
`clap` defines the process flags. `shard_coordinate < 0` (the default `-1`) leaves `HomeMixerConfig.shard_coordinate` as `None`; a non-negative value becomes `ShardCoordinate { ordinal, total_size }`.
gRPC listen port.
Metrics listen port.
Wily shard ordinal. Values `>= 0` enable `ShardCoordinate`; `-1` disables sharding.
Shard ring size. Used only when `shard-coordinate >= 0`.
Datacenter string. Passed to `XServiceBuilder`, `QueryBuilder`, Gizmoduck, and both pipeline `prod` constructors. Also injected into feature-switch recipients as custom string `datacenter`.
OpenTelemetry collector endpoint. Empty string leaves OTel unset.
```bash
# Conceptual production start. Unpublished params/clients/proto crates
# are required; this checkout cannot link the binary.
home-mixer \
--grpc-port 50051 \
--metrics-port 9090 \
--shard-coordinate -1 \
--shard-total-size 500 \
--datacenter atla
```
Boot sequence after `Args::parse()`:
1. `xai_stringcenter::init_from_file(params::STRINGCENTER_BUNDLE_PATH)`
2. `XServiceBuilder::new("home-mixer")` with gRPC port, metrics port, datacenter, OTel
3. Feature switches from `params::FS_PATH` (second arg `true`)
4. Decider from `params::decider_path()`
5. mTLS from the environment
6. Max connection age `300` seconds
7. gRPC reflection from `pb::FILE_DESCRIPTOR_SET`
8. Dark-traffic layers: `dark_traffic_setup::resolve_layer()` then `RejectDarkTrafficLayer::from_env()`
9. HTTP profiling routes from `xai_profiling::profiling_router()`
10. `run::(HomeMixerConfig { shard_coordinate })`
Both registered services accept and send Gzip and Zstd, and cap decode/encode size at `params::MAX_GRPC_MESSAGE_SIZE`.
## Service wiring
`HomeMixerServer::build` constructs one shared `QueryBuilder`, one `PhoenixCandidatePipeline`, then wraps that pipeline in `ScoredPostsServer`. `ForYouCandidatePipeline` takes `Arc` and the datacenter so For You organic posts reuse the same scored-posts path.
```text
XServiceBuilder("home-mixer")
│
▼
HomeMixerServer
├─ QueryBuilder ── feature_switches, decider, datacenter, GizmoduckClient
├─ ScoredPostsServer ── PhoenixCandidatePipeline → ScoredPost[]
└─ ForYouFeedServer ── ForYouCandidatePipeline
├─ ScoredPostsSource → run_pipeline()
├─ AdsSource
├─ WhoToFollowSource
├─ PromptsSource
└─ PushToHomeSource
```
Gizmoduck is `ProdGizmoduckClient::new(shard_coordinate, datacenter, Some("home-mixer.prod"))`. `QueryBuilder::mock()` exists for tests: empty feature switches, empty `DeciderStore`, datacenter `"mock"`, `MockGizmoduckClient`.
## Request path
```mermaid
sequenceDiagram
participant Client
participant QB as QueryBuilder
participant Giz as GizmoduckClient
participant SP as ScoredPostsServer
participant FY as ForYouFeedServer
participant Phoenix as PhoenixCandidatePipeline
participant FYPipes as ForYouCandidatePipeline
Client->>QB: pb.ScoredPostsQuery
QB->>QB: reject viewer_id == 0
alt viewer_id in TRACE_USER_IDS
QB->>QB: B3RequestInfo.force_sample()
end
QB->>Giz: get_viewer_data (200 ms)
Giz-->>QB: ViewerData or default
QB->>QB: feature switches + ScoredPostsQuery::new
alt GetScoredPosts / GetDebugScoredPosts
QB->>SP: run_pipeline(query)
alt user_id in TEST_USER_IDS
SP-->>Client: empty ScoredPostsResponse
else
SP->>Phoenix: execute(query)
Phoenix-->>Client: ScoredPost[]
end
else GetForYouFeed / GetForYouFeedUrt
Note over FY: URT patches cursor, is_polling, request_context after build
QB->>FY: get_for_you_feed(query)
alt user_id in TEST_USER_IDS
FY-->>Client: empty FeedItem[]
else
FY->>FYPipes: execute(query)
FYPipes->>SP: ScoredPostsSource.run_pipeline
FYPipes-->>Client: FeedItem[] or URT bytes
end
end
```
B3 trace headers are extracted from request metadata and injected on the response. `GetDebugScoredPosts` always calls `force_sample()`. Root spans use endpoint names `scored_posts`, `debug_scored_posts`, `for_you_feed`, or `for_you_feed_urt`.
## Validate viewer_id
`QueryBuilder::build` is the only constructor used by the four RPCs.
If `proto_query.viewer_id == 0`, return `Status::invalid_argument("viewer_id must be specified")`. Proto3 numeric defaults are `0`, so omitting the field fails the same way.
If `params::TRACE_USER_IDS` contains `viewer_id`, call `b3_info.force_sample()`. The ID list lives in unpublished `params`.
`fetch_viewer_data` calls `gizmoduck_client.get_viewer_data(viewer_id)` with a `200` ms timeout (`VIEWER_ROLES_TIMEOUT_MS`). Timeout or error becomes `ViewerData::default()`.
`in_network_only = proto_query.in_network_only || viewer_data.allow_for_you_recommendations == Some(false)`. `None` (the default after a Gizmoduck miss) does **not** flip the flag.
`RecipientBuilder` gets `user_id`, `country`, `language`, `client_app_id`, custom `datacenter`, custom `account_age_days` from `days_since_creation(viewer_id)`, custom `has_phone_number`, and `user_roles` when nonempty. Debug RPCs then apply `feature_switch_overrides`.
`ScoredPostsQuery::new(...)` copies proto fields, Gizmoduck fields, and generated IDs. `request_id = resolve_request_id(proto_query.request_id)`; `prediction_id = generate_request_id()`; `push_to_home_post_id = non_zero(proto_query.push_to_home_post_id)` (`0` becomes `None`); `is_shadow_traffic = is_sampled(request_id, 0.5)`.
Viewer fields consumed from Gizmoduck (inferred from `QueryBuilder` usage; `ViewerData` itself is unpublished): `roles`, `has_phone_number`, `allow_for_you_recommendations`, `muted_keywords`, `follower_count`, `subscription_level`, `age_in_years`.
`TEST_USER_IDS` is checked **after** `QueryBuilder`. A test user still needs a nonzero `viewer_id`. Both `ScoredPostsServer::run_pipeline` and `ForYouFeedServer::get_for_you_feed` return empty results without executing a pipeline.
## Proto fields QueryBuilder reads
`xai_home_mixer_proto::ScoredPostsQuery` is unpublished. These are the fields `QueryBuilder` and the For You URT handler actually read:
| Proto field | Destination | Notes |
|---|---|---|
| `viewer_id` | `query.user_id` | Required; `0` is invalid |
| `client_app_id` | `query.client_app_id` | Also feature-switch recipient |
| `country_code` | `query.country_code` | Feature-switch recipient |
| `language_code` | `query.language_code` | Feature-switch recipient |
| `seen_ids` | `query.seen_ids` | |
| `served_ids` | `query.served_ids` | |
| `in_network_only` | `query.in_network_only` | OR'd with Gizmoduck For You opt-out |
| `is_bottom_request` | `query.is_bottom_request` | URT `BOTTOM` cursor can overwrite |
| `topic_ids` | `query.topic_ids` | `is_topic_request()` / `is_bulk_topic_request()` (`len > 6`) |
| `excluded_topic_ids` | `query.excluded_topic_ids` | |
| `exclude_videos` | `query.exclude_videos` | |
| `request_id` | `query.request_id` | Via `resolve_request_id` |
| `is_preview` | `query.is_preview` | Disables `AdsSource` when `true` |
| `push_to_home_post_id` | `query.push_to_home_post_id` | `0` dropped; enables `PushToHomeSource` |
| `device_status.*` | device / IP fields | Missing message → default empty values |
| `cursor` | `query.cursor` | Applied only on `GetForYouFeedUrt` after `build` |
| `request_context` | `query.request_context` | URT only, after `build` |
| `is_polling` | `query.is_polling` | URT only, after `build` |
`device_status` fields mapped: `ip_address`, `user_agent`, `time_zone` (`timezone_string_to_enum`), `device_network_type` (`network_type_string_to_enum`), `client_version`, `device_id`, `mobile_device_id`, `mobile_device_ad_id`.
`ForYouFeedQuery` must wrap that proto: `feed_query.query` missing returns `"query must be specified"`. `DebugScoredPostsQuery` uses `query.unwrap_or_default()` — a missing inner query becomes `viewer_id == 0` and then fails validation. Debug also passes `feature_switch_overrides: HashMap`.
## Constructor defaults
`ScoredPostsQuery::new` copies the arguments above, then fills the rest:
| Field | Constructor default |
|---|---|
| `is_top_request` | `false` (URT `TOP` cursor later sets `true`) |
| `bloom_filter_entries` | `[]` |
| `scoring_sequence` / `retrieval_sequence` | `None` (query hydrators fill later) |
| `columnar_scoring_sequence` / `columnar_retrieval_sequence` | `None` |
| `user_features` | `muted_keywords` + `follower_count`; other lists empty |
| `request_time_ms` | `current_time_ms()` |
| `cached_posts` / `has_cached_posts` | `[]` / `false` |
| `new_user_topic_ids` | `[]` |
| `in_network_replies` | default |
| `viewer_minhash` | `None` |
| `user_demographics` / `ip_location` | `None` |
| `user_inferred_gender` / score | `None` |
| `followed_grok_topics` / `inferred_grok_topics` / `followed_starter_packs` | `None` |
| `is_polling` | `false` |
| `cursor` | `None` |
| `request_context` | `""` |
| `served_history` | `[]` |
| `who_to_follow_eligible` | `false` (`ServedHistoryQueryHydrator` may set it on For You) |
| `non_polling_timestamps` | `None` |
| `impressed_post_ids` | `[]` |
`GetTwitterContextViewer` exposes `user_id`, `client_application_id`, `request_country_code`, and `request_language_code`. `PipelineQuery` exposes `params` and `decider`.
`home-mixer/candidate_pipeline/query.rs` defines a narrower leftover `ScoredPostsQuery` (`user_id: i64`, string `request_id`). Live RPCs and both candidate pipelines use `models::query::ScoredPostsQuery`.
## Choose an entry point
| RPC | Request | Pipeline | Response | Extra vs QueryBuilder |
|---|---|---|---|---|
| `ScoredPostsService.GetScoredPosts` | `pb::ScoredPostsQuery` | `PhoenixCandidatePipeline` | `ScoredPostsResponse { scored_posts }` | None |
| `ScoredPostsService.GetDebugScoredPosts` | `pb::DebugScoredPostsQuery` | same | `DebugScoredPostsResponse { scored_posts, debug_json }` | Always sampled; FS overrides |
| `ForYouFeedService.GetForYouFeed` | `pb::ForYouFeedQuery` | `ForYouCandidatePipeline` | `ForYouFeedResponse { items }` | Requires nested `query` |
| `ForYouFeedService.GetForYouFeedUrt` | `pb::ForYouFeedQuery` | same | `ForYouFeedUrtResponse { urt }` | Cursor / polling / `request_context`; Thrift URT |
**Scored posts** is the organic ranker. `PhoenixCandidatePipeline` sources Thunder, Tweet Mixer, Phoenix, Phoenix topics, Phoenix MoE, and cached posts; selects with `TopKScoreSelector`; result size `params::RESULT_SIZE`. `in_network_only` disables Phoenix / Tweet Mixer / Phoenix MoE / Phoenix topics. Topic requests flip product-surface logs to `topics`; excluded topics log as `for_you_with_snoozed_topics`; otherwise `for_you` or `ranked_following`.
**For You** is the blended feed. `ForYouCandidatePipeline` hydrates served history and past request timestamps, sources organic posts through `ScoredPostsSource` (`ScoredPostsServer::run_pipeline` → `FeedItem::Post`), then ads, who-to-follow, prompts, and push-to-home. Selector is `BlenderSelector`. Result size `params::FOR_YOU_MAX_RESULT_SIZE`. `is_preview` skips ads. `push_to_home_post_id` enables `PushToHomeSource`.
URT-only post-processing after `build`:
- Copy `request_context` and `is_polling` from the proto.
- If `cursor` is nonempty, `cursor_utils::decode_ordered_cursor` sets `query.cursor`. `CursorType::BOTTOM` sets `is_bottom_request`; `CursorType::TOP` sets `is_top_request`. Decode errors are logged and ignored.
- `urt::make_urt_timeline` then `xai_urt_thrift::serialize_binary`. Serialize failure is `Status::internal("failed to serialize URT: …")`.
- `in_network_only` — Thunder still runs; Phoenix / Tweet Mixer / topics / MoE disable.
- `topic_ids` — `is_topic_request()`; `len > 6` is `is_bulk_topic_request()` (Phoenix stays enabled for bulk).
- `is_preview` — `AdsSource` off.
- `push_to_home_post_id` — `PushToHomeSource` on.
- `who_to_follow_eligible` — starts `false`; For You `ServedHistoryQueryHydrator` may flip it before `WhoToFollowSource`.
- `has_cached_posts` — starts `false`; cache hydrators later can skip live sources.
## Assemble a client request
Use `GetScoredPosts` for ranked organic `ScoredPost`s. Use `GetForYouFeed` / `GetForYouFeedUrt` for a blended `FeedItem` timeline. Use `GetDebugScoredPosts` only when you need `debug_json` and FS overrides.
Required on `pb::ScoredPostsQuery`. For You and debug wrap that message; they do not replace it.
Set `client_app_id`, `country_code`, `language_code` (feature switches). Pass `seen_ids` / `served_ids` to suppress already-shown posts. Set `in_network_only` only when the client wants following-only (Gizmoduck can still force it).
`topic_ids`, `excluded_topic_ids`, `exclude_videos`, `is_preview`, `push_to_home_post_id`, `device_status`, `request_id`. Leave `push_to_home_post_id` at `0` unless a focal post should be injected.
Put the proto in `ForYouFeedQuery.query`. For URT, also set `cursor`, `request_context`, and `is_polling` on the inner `ScoredPostsQuery` — `QueryBuilder` ignores those three; the URT handler copies them after `build`.
A zero or omitted `viewer_id` must return `INVALID_ARGUMENT` / `viewer_id must be specified`. A successful scored-posts call returns `scored_posts`; For You returns `items` or URT bytes. IDs in `TEST_USER_IDS` return empty payloads with no pipeline work.
```text
# Minimal scored-posts query (field names as used in QueryBuilder)
viewer_id:
client_app_id:
country_code: "US"
language_code: "en"
seen_ids: []
served_ids: []
in_network_only: false
is_bottom_request: false
topic_ids: []
excluded_topic_ids: []
exclude_videos: false
is_preview: false
push_to_home_post_id: 0
request_id: 0 # resolve_request_id decides the stored id
device_status: { ... } # optional; defaults if omitted
# ForYouFeedQuery
query:
# DebugScoredPostsQuery
query:
feature_switch_overrides: { "SomeFS": "true" }
```
```text
# GetScoredPosts — ScoredPostsResponse
scored_posts: [
tweet_id, author_id, score, in_network, served_type, ...
]
# GetForYouFeed — ForYouFeedResponse
items: [ FeedItem { position, item: Post | Ad | ... } ]
# GetForYouFeedUrt — ForYouFeedUrtResponse
urt:
# GetDebugScoredPosts also includes
debug_json: { query, retrieved_candidates, filtered_candidates,
selected_candidates, stats }
```
## Errors and empty feeds
| Condition | Status / result |
|---|---|
| `viewer_id == 0` or omitted | `INVALID_ARGUMENT` `"viewer_id must be specified"` |
| For You missing `ForYouFeedQuery.query` | `INVALID_ARGUMENT` `"query must be specified"` |
| Debug missing inner `query` | Default proto → same `viewer_id` error |
| `user_id` in `TEST_USER_IDS` | `200`-style success with empty `scored_posts` / `items` |
| Gizmoduck timeout or error | Continue with `ViewerData::default()` |
| Bad URT cursor | Warn, ignore cursor, continue |
| URT serialize failure | `INTERNAL` `"failed to serialize URT: …"` |
`PhoenixSource` later fails with `"PhoenixSource: missing retrieval_sequence"` if query hydrators never fill that field. That is post-assembly; it is not a `QueryBuilder` error.
## Related pages
Full query fields, TEST_USER_IDS empty responses, and ScoredPost / ForYouFeed / URT mapping.
CandidatePipeline.execute stages and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline.
What this checkout can execute locally versus production Home Mixer.
How in_network_only changes Thunder versus Phoenix sourcing.
AdsSource, blenders, and For You insertion after scored posts.
viewer_id must be specified, TEST_USER_IDS empty feeds, and unpublished crates.
---
## 13. Blend ads into the feed
> AdsSource intake, SafeGapAdsBlender versus PartitionOrganicAdsBlender, AdsBlenderType selection, and prompt / who-to-follow / push-to-home insertion.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/13-blend-ads-into-the-feed.md
- Generated: 2026-08-13T19:45:23.566Z
### Source Files
- `home-mixer/ads/mod.rs`
- `home-mixer/ads/safe_gap_blender.rs`
- `home-mixer/ads/partition_organic_blender.rs`
- `home-mixer/ads/util.rs`
- `home-mixer/selectors/blender_selector.rs`
- `home-mixer/sources/ads_source.rs`
- `home-mixer/candidate_hydrators/ads_brand_safety_hydrator.rs`
---
title: "Blend ads into the feed"
description: "AdsSource intake, SafeGapAdsBlender versus PartitionOrganicAdsBlender, AdsBlenderType selection, and prompt / who-to-follow / push-to-home insertion."
---
`ForYouCandidatePipeline` assembles the Home timeline by fetching organic `ScoredPost`s, ads, prompts, Who to Follow, and push-to-home in parallel, then running `BlenderSelector`. That selector partitions the mixed `FeedItem` stream, chooses `SafeGapAdsBlender` or `PartitionOrganicAdsBlender` from `AdsBlenderType`, and inserts the remaining modules around the blended result.
This checkout publishes the blender, source, hydrator, and selector logic. `crate::params` and `crate::clients` (including `AdIndexClient`) are production Home Mixer modules and are not in this snapshot. Local `phoenix/run_pipeline.py` does not run ads blending.
```mermaid
flowchart TB
subgraph Phoenix["PhoenixCandidatePipeline"]
TopK["TopKScoreSelector"]
BSH["AdsBrandSafetyHydrator or AdsBrandSafetyVfHydrator"]
SP["ScoredPost.brand_safety_verdict"]
TopK --> BSH --> SP
end
subgraph External["Unpublished production clients"]
AdIdx["AdIndexClient.get_eligible_ads"]
PromptsC["PromptsClient.get_injections"]
WtfC["WhoToFollowClient.get_wtf_recommendations"]
Tes["TESClient + ReplyMixerClient"]
end
subgraph ForYou["ForYouCandidatePipeline"]
SPS["ScoredPostsSource"]
ADS["AdsSource"]
PR["PromptsSource"]
WTF["WhoToFollowSource"]
PTH["PushToHomeSource"]
BS["BlenderSelector"]
SPS --> BS
ADS --> BS
PR --> BS
WTF --> BS
PTH --> BS
end
SP --> SPS
AdIdx --> ADS
PromptsC --> PR
WtfC --> WTF
Tes --> PTH
BS --> Feed["Vec FeedItem"]
```
`ForYouCandidatePipeline` has empty hydrator, filter, and scorer lists. Organic ranking and brand-safety labels happen in `PhoenixCandidatePipeline` before `ScoredPostsSource` wraps each `ScoredPost` as `feed_item::Item::Post`. After `BlenderSelector`, `CandidatePipeline::execute` truncates to `params::FOR_YOU_MAX_RESULT_SIZE`.
## Ads intake
`AdsSource` is enabled when `EnableAdsSource` is true and `ScoredPostsQuery.is_preview` is false. Preview requests skip ads entirely.
A failed `get_eligible_ads` call returns `Err("AdsSource: …")`. `fetch_candidates` flattens source results, so a failed ads fetch contributes nothing and organic sources still produce a feed.
Feature-switch key on `query.params`. Combined with `!query.is_preview`.
Copied from the gRPC `ScoredPostsQuery`. When true, `AdsSource.enable` is false.
The request sent to Ad Index:
```rust AdIndexRequest from AdsSource
AdIndexRequest {
user_id: query.user_id as i64,
product_surface: ProductSurface::HomeTimelineRanking as i32,
client_context: Some(ClientContext {
user_id: query.user_id as i64,
app_id: query.client_app_id as i64,
country_code: query.country_code.clone(),
language_code: query.language_code.clone(),
ip_address: query.ip_address.clone(),
user_agent: query.user_agent.clone(),
user_roles: query.user_roles.clone(),
device_id: query.device_id.clone(),
mobile_device_id: query.mobile_device_id.clone(),
mobile_device_ad_id: query.mobile_device_ad_id.clone(),
..Default::default()
}),
..Default::default()
}
```
Each `AdIndexInfo` in `response.ad_info` becomes `FeedItem { position: 0, item: Some(Ad(ad)) }`. Fields the blender later reads:
| Field | Used for |
| --- | --- |
| `insert_position` | Spacing from the first four ads; first ideal gap for `SafeGapAdsBlender` |
| `ad_adjacency_control.brand_safety_risk` | BSR-low / IAS drop in `PartitionOrganicAdsBlender` |
| `ad_adjacency_control.handles` | Adjacent-author drop |
| `ad_adjacency_control.keywords` | Adjacent-text drop |
| `post_id`, `author_id`, `impression_id`, `account_id` | Kafka logging and served-candidate details |
## Brand-safety verdicts
Organic posts enter the blender as `ScoredPost`s. `PhoenixCandidatePipeline` hydrates verdicts after `TopKScoreSelector`, then `candidates_to_scored_posts` copies them. A missing verdict becomes `BrandSafetyVerdict::MediumRisk`.
Two hydrators share `EnableAdsBrandSafetyHydrator` and are mutually exclusive via the `vf_brand_safety_dark_traffic` decider:
| Hydrator | Runs when | Backend |
| --- | --- | --- |
| `AdsBrandSafetyHydrator` | flag on and decider **off** | `SafetyLabelStoreClient.batch_get_all_labels`, Moka cache of 1_000_000 keys |
| `AdsBrandSafetyVfHydrator` | flag on and decider **on** | `VfClient.get_safety_labels` |
Both look up `retweeted_tweet_id.unwrap_or(tweet_id)` plus `quoted_tweet_id`. A quote lookup failure raises the candidate to at least `MediumRisk`. A primary lookup `Err` skips `update`; the candidate keeps its previous (usually unset) verdict and later maps to `MediumRisk`.
`compute_verdict` in `home-mixer/models/brand_safety.rs`:
1. Any label in `MEDIUM_RISK_LABELS` → `MediumRisk`.
2. Missing `GROK_SFA` and `GROK_NSFA_LIMITED` → `MediumRisk`.
3. `tweet_id >= 2054275414225846272` without `PTOS_REVIEWED` → `MediumRisk`.
4. Any label in `LOW_RISK_LABELS` → `LowRisk`.
5. Otherwise → `Safe`.
The blender treats `MediumRisk` as **avoid**: `has_avoid` is true only for that verdict. `Safe` and `LowRisk` are eligible neighbors. `LowRisk` additionally blocks `BsrLow` / `BsrIas` ads in the partition blender.
Hydrators must return one result per input candidate and must not drop posts. Length mismatches become per-candidate errors and leave fields unchanged.
## AdsBlenderType
`BlenderSelector` reads `query.params.get(AdsBlenderType)` as a string:
| Value | Implementation |
| --- | --- |
| `"safe_gap"` | `SafeGapAdsBlender` |
| any other string | `PartitionOrganicAdsBlender` |
Feature-switch string. Exact match `"safe_gap"` selects the gap placer. Every other value, including an unset or unexpected default, selects `PartitionOrganicAdsBlender`.
The numeric default is not published in this checkout. `ForYouResponseStatsSideEffect` tags `ForYouFeed.response` with `blender=`.
## Shared blender rules
Both implementations go through `AdsBlender::blend`, which first records:
- `AdsBlender.post_brand_safety_verdict` per post (`verdict` = proto `as_str_name()`)
- `AdsBlender.ad_brand_safety_risk` per ad (`risk` from `ad_adjacency_control`, else `BsrUnknown`)
Constants in `home-mixer/ads/util.rs`:
| Identifier | Value | Role |
| --- | --- | --- |
| `MIN_POSTS_FOR_ADS` | `5` | If `ads` is empty or `scored_posts.len() < 5`, return organic-only `FeedItem`s |
| `MIN_REQUESTED_GAP` | `3` | Minimum accepted gap derived from ad `insert_position`s |
| `DEFAULT_SPACING` | `requested: 3`, `min: 2` | Used when fewer than two ads, or the min positive delta is `< 3` |
`compute_spacing` sorts the first four `insert_position`s and takes the minimum positive adjacent difference. If that value is `>= 3`, requested spacing is that delta and min spacing is `requested.div_ceil(2)`.
Both blenders then:
1. `truncate` the item list to `params::RESULT_SIZE` (same constant Phoenix uses as its pipeline `result_size`).
2. Pop a trailing `feed_item::Item::Ad` so the page never ends on an ad.
3. Rewrite `FeedItem.position` to `0..len` as `i32`.
`RESULT_SIZE` itself is defined in the unpublished `params` module.
## SafeGapAdsBlender
Keeps organic order. Ads occupy **safe gaps**: index `g` in `1..n` where neither `posts[g-1]` nor `posts[g]` is `MediumRisk`. Gap `0` is never eligible, so an ad cannot sit before the first post.
Placement loop (`assign_ads_to_gaps`):
| Ad | Ideal gap | Minimum gap |
| --- | --- | --- |
| First | `ads[0].insert_position.max(0)` | `1` |
| Later | previous **ideal** + `spacing.requested` | `max(previous ideal + spacing.min, last actual + 2)` |
`find_best_gap` takes gaps `>= min` and picks the closest to ideal. A tie (`ideal - below <= above - ideal`) chooses the lower gap. If no remaining gap satisfies `min`, remaining ads are dropped.
`interleave_and_finalize` inserts each chosen ad **before** the post at that gap index, then applies the shared truncate / no-trailing-ad / reindex rules.
```text
organic order preserved
P0 P1 [ad] P2 P3 [ad] P4
^safe gap 2 ^safe gap 4
MediumRisk posts block the gaps on either side
```
## PartitionOrganicAdsBlender
Rebuilds the feed around `above / ad / below` triples taken from **safe** posts only.
1. `actual_ads = min(ads.len(), (n - 1) / spacing.requested, safe_count / 2)`. `spacing.requested == 0` yields zero ads.
2. Split posts into `safe` (`!has_avoid`) and `unsafe_posts` (`MediumRisk`), preserving relative order inside each bucket.
3. `group_size = num_safe / actual_ads` (at least 2 when `actual_ads > 0`).
4. For each ad, try the current group's first two remaining safe posts. Drops **do not** advance `group_idx`, so the next ad retries the same pair:
| Check | Drops the ad when |
| --- | --- |
| `should_drop_bsr_low` | Ad risk is `BsrLow` or `BsrIas` **and** above or below is `LowRisk` |
| `should_drop_handle` | `ad_adjacency_control.handles` contains `author_id` of above or below |
| `should_drop_keyword` | Tokenized `keywords` are a subsequence of `tweet_text` on above or below (`TweetTokenizer`) |
5. Leftover safe posts plus all unsafe posts are sorted by `score` descending and used as filler after each triple. Remainder filler goes to the last groups.
6. If zero ads survive, every leftover post is score-sorted and returned as organic-only.
Enforcement counters on `PartitionOrganic.enforcement`: `drop` (BSR), `ok` (BSR-low ad placed), `handle_drop`, `keyword_drop`.
```text
safe group 0 filler (score desc) later triples
P_safe [ad] P_safe P_fill … P_safe [ad] P_safe …
MediumRisk posts only appear in filler, never as ad neighbors
```
## Prompt, who-to-follow, and push-to-home insertion
`BlenderSelector` partitions incoming `FeedItem`s, blends posts+ads, then inserts modules in this order:
1. `insert_prompts` — every prompt at index `0, 1, …` (order preserved at the front). `FeedItem.position` is `PROMPTS_POSITION`.
2. `insert_who_to_follow` — **first** `WhoToFollowModule` only, at `min(WHO_TO_FOLLOW_POSITION.saturating_sub(1), blended.len())`. `FeedItem.position` is `WHO_TO_FOLLOW_POSITION as i32`.
3. `pin_push_to_home` — at most one item, inserted at index `0` with `position: 0`.
Those helpers do **not** rewrite positions of already-blended items. Kafka ads-injection logging uses enumerate order; `ServedCandidatesKafkaSideEffect` uses `item.position`.
### PromptsSource
Must be true or the source is skipped.
Requests `GetInjectionsRequest` with `DisplayLocation::HOME_TIMELINE` and supported types `INLINE_PROMPT`, `FULL_COVER`, `HALF_COVER`, `RELEVANCE_PROMPT`. Each injection is Thrift-serialized into `Prompt.injection`. Serialization failure fails the source (`"PromptsSource: serialization failed: …"`).
### WhoToFollowSource
Must be true **and** `query.who_to_follow_eligible`.
Starts `false` in `ScoredPostsQuery::new`. `ServedHistoryQueryHydrator` sets it when `EnableUrtMigrationComponents` is on: eligible if no prior `EntityIdType::WHO_TO_FOLLOW` entry, or last serve is at least `WhoToFollowFatigueHours` hours before `request_time_ms`.
Empty recommendations return no items. Otherwise the source keeps at most `MAX_WHO_TO_FOLLOW_USERS` (`3`) users. Exclusions are up to `200` previously served Who-to-Follow `user_id`s from `served_history`.
### PushToHomeSource
Enabled only when `query.push_to_home_post_id` is `Some`. `QueryBuilder` copies a non-zero proto `push_to_home_post_id`.
- TES miss or `Ok(None)` → empty vec (not an error).
- TES error → `Err("PushToHomeSource: TES error for tweet …")`.
- Root posts (`in_reply_to_tweet_id` none) request up to `3` facepile repliers from `ReplyMixerClient`, excluding the author. Reply-mixer errors log a warning and leave the facepile empty.
- `served_type` is `ServedType::ForYouPushToHome`.
Final visual order after a full insert:
```text
[PushToHome] [Prompt…] …organic+ads… [WhoToFollow at WHO_TO_FOLLOW_POSITION-1, then shifted by PTH] …
```
## Dropped items and pipeline cap
`BlenderSelector` compares input post/ad counts to the blended output and emits `non_selected` **placeholders** (`ScoredPost::default()` / `AdIndexInfo::default()`) for the difference. Those placeholders exist so fetched-vs-response stats can recover dropped counts. They are not real ranked candidates.
`CandidatePipeline::execute` then `split_off`s anything beyond `FOR_YOU_MAX_RESULT_SIZE` into `non_selected` as well.
`ForYouFeedServer` short-circuits `params::TEST_USER_IDS` to an empty item list and never runs the blender.
## Side effects and stats
| Surface | Gate | Behavior |
| --- | --- | --- |
| `AdsInjectionLoggingSideEffect` | `is_prod()` and `EnableAdsInjectionLogging` | Publishes `AdsInjectedTimeline` to `ADS_INJECTION_TOPIC` on the Ads Kafka cluster. Counts `retrieved_*` from selected + non-selected, `response_*` from selected. `display_location` is `TimelineHome`. |
| `ForYouResponseStatsSideEffect` | always | Logs post/ad counts and increments `ForYouFeed.response` with `subscription`, `blender`, and country bucket. Empty / sufficient thresholds: `0` ads, `0` posts, `>= 5` ads, `>= 20` posts. |
| `ServedCandidatesKafkaSideEffect` | (its own enable) | Ads become `PROMOTED_TWEET` with `insert_position` and `impression_id`. |
## Failure modes
| Symptom | Cause |
| --- | --- |
| Organic-only feed, ads expected | `EnableAdsSource` off, `is_preview`, `AdsSource` RPC error, fewer than 5 posts, `actual_ads == 0`, or every ad dropped by adjacency |
| Feed never starts with an ad | Safe-gap minimum is 1; partition triples always start with a post; trailing ads are popped |
| Who to Follow missing | `EnableWhoToFollowModule` off, `who_to_follow_eligible` still false (`EnableUrtMigrationComponents` off or fatigue window), or empty recommendations |
| Prompts missing | `EnablePrompts` off or injection client / serialize error |
| Push-to-home missing | `push_to_home_post_id` unset/zero, or TES returned no core data |
| Ads sit next to `MediumRisk` | Brand-safety hydrator skipped or failed (`None` → `MediumRisk` only on the **organic** side). Ads themselves are not labeled by these hydrators |
| BSR-low ads never appear next to `LowRisk` | `PartitionOrganicAdsBlender` only; `SafeGapAdsBlender` does not run handle / keyword / BSR-low adjacency drops |
| Cannot compile or run Home Mixer locally | Unpublished `params`, `clients`, and proto crates. See runtime boundaries |
## Next
How CandidatePipeline.execute stages wrap ForYouCandidatePipeline around PhoenixCandidatePipeline.
Enable predicates and served_type assignment for Phoenix, Thunder, ads, prompts, and Who to Follow.
Post-selection AdsBrandSafety hydrators and the length-match / no-drop hydrator contract.
QueryBuilder fields including is_preview and push_to_home_post_id.
What this checkout can execute versus unpublished Home Mixer crates.
TEST_USER_IDS empty feeds, viewer_id validation, and unpublished-module failures.
---
## 14. Execute Grox content plans
> Engine, Dispatcher, and GrpcServer startup, PlanMaster fan-out, TaskEligibility gating, and merged TaskResult fields.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/14-execute-grox-content-plans.md
- Generated: 2026-08-13T19:45:35.844Z
### Source Files
- `grox/main.py`
- `grox/engine.py`
- `grox/dispatcher.py`
- `grox/plans/plan.py`
- `grox/plans/plan_master.py`
- `grox/schedules/types.py`
- `grox/plans/plan_post_safety.py`
---
title: "Execute Grox content plans"
description: "Engine, Dispatcher, and GrpcServer startup, PlanMaster fan-out, TaskEligibility gating, and merged TaskResult fields."
---
`grox/main.py` `serve()` builds one shared `ScheduleContext`, then starts `Engine`, `Dispatcher`, and `GrpcServer` in that order. The dispatcher process pulls Kafka-backed `TaskPayload`s, injects `TaskEligibility` values, and writes them to `task_queue`. The engine process reads that queue, runs `PlanMaster.exec`, and writes a merged `TaskResult` to `resp_queue`. The published snapshot does not include `grox.service`, `grox.config`, `grox.data_loaders.data_types`, `grox.data_loaders.media_processor`, or `monitor`, so `python grox/main.py` cannot start from this checkout.
This is a production-shaped snapshot, not a local runner. Phoenix inference in `phoenix/run_pipeline.py` is the checkout that executes. Grox plan execution needs the unpublished config, gRPC, media, and metrics modules listed under [Runtime boundary](#runtime-boundary).
## Process layout
Three OS processes share a `multiprocessing.Manager` dict. Child processes call `init_proc`, which ignores `SIGINT` / `SIGTERM`; only the main `serve()` loop handles those signals.
```mermaid
flowchart TB
subgraph Main["main process — grox/main.py serve()"]
Init["init_proc('main')"]
Ctx["new_context()"]
Signals["SIGINT / SIGTERM → shutdown Event"]
Grpc["GrpcServer(context)"]
end
subgraph Shared["ScheduleContext manager.dict"]
TQ["task_queue"]
RQ["resp_queue"]
SE["shutdown_event"]
QSE["queue_connection_shutdown_event"]
LTQ["live_task_queue"]
LRQ["live_resp_queue"]
end
subgraph Disp["grox-dispatcher process"]
Fill["_fill_loop"]
Result["_result_loop"]
Gens["PriorityTaskGenerator"]
Loaders["KafkaPostLoader / MessageQueueLoader"]
end
subgraph Eng["grox-engine process"]
Poll["_poll_task"]
PM["PlanMaster.exec"]
Plans["ALL_PLANS DAGs"]
ASR["ASRProcessor.start"]
Media["MediaProcessor.start"]
end
Kafka["Kafka topics via KafkaTopicName"] --> Loaders
Loaders --> Gens
Gens --> Fill
Fill --> TQ
TQ --> Poll
Poll --> PM
PM --> Plans
Plans --> RQ
RQ --> Result
Result --> Gens
Init --> Ctx
Ctx --> Shared
Signals --> QSE
QSE --> Gens
Signals --> SE
SE --> Poll
SE --> Fill
Grpc --> Shared
```
`Engine` and `Dispatcher` only use `task_queue`, `resp_queue`, `shutdown_event`, and `queue_connection_shutdown_event`. `live_task_queue` and `live_resp_queue` are allocated and unused by those two classes. `GrpcServer` is constructed with the same context; its RPCs are not in this tree.
## Start the service
The published entry is `grox/main.py`:
```python
if __name__ == "__main__":
asyncio.run(serve())
```
There is no `grox/` `pyproject.toml` or console script in this checkout.
`new_context()` returns a `SyncManager` dict with six keys:
| Key | Type | Used by |
|---|---|---|
| `task_queue` | `Queue[TaskPayload]` | Dispatcher put, Engine get |
| `resp_queue` | `Queue[TaskResult]` | Engine put, Dispatcher get |
| `live_task_queue` | `Queue` | Allocated only |
| `live_resp_queue` | `Queue` | Allocated only |
| `shutdown_event` | `Event` | Engine / Dispatcher loops |
| `queue_connection_shutdown_event` | `Event` | Dispatcher generator stop |
`serve()` waits for each child `started_event` before continuing:
1. `Engine.start()` — process name `grox-engine`. Child runs `init_proc("engine")`, then `MediaProcessor.start()` and `ASRProcessor.start()`, then sets `started_event`.
2. `Dispatcher.start()` — process name `grox-dispatcher`. Child `init_proc("dispatcher")` is wrapped in tenacity: 3 attempts, wait 1s then +3s (cap 9s). It builds `PriorityTaskGenerator` from `grox_config.dispatcher.task_generators` and calls `start()` on every generator (each starts its Kafka loader).
3. `GrpcServer(context).start()` — implementation is unpublished.
Logs `Grox server started` only after all three `start()` calls return.
Expected log sequence on a complete deployment:
```text
Starting grox server...
Starting Grox engine...
Grox engine started
Starting Grox dispatcher...
Grox dispatcher started
Grox server started
```
An `ImportError` for `grox.service`, `grox.config.config`, `monitor.logging`, or `grox.data_loaders.media_processor` is the local-checkout failure mode.
## Dispatcher intake
`Dispatcher` runs three coroutines: `_fill_loop`, `_result_loop`, and `_wait_for_queue_connection_shutdown`.
### Generator selection
`grox_config.dispatcher.task_generators` is a list of `{type, max_qps, weight}`. Unknown `TaskGeneratorType` raises `ValueError`. `PriorityTaskGenerator` requires a non-empty list and strictly positive weights. Each poll round samples remaining generators with `random.choices` by weight; a generator that yields `None` is dropped from that round only.
`TaskGenerator.poll` applies an in-process `FixedWindowRateLimiter` when `max_qps` is set (`RateLimitItemPerSecond(max_qps, 1)`).
### Eligibility injection
`StreamTaskGenerator._poll` copies loader fields onto `TaskPayload` and sets `eligibilities` from the generator class constant. `payload_id` is a new `uuid.uuid4().hex` from `KafkaPostLoader`, not the Kafka offset.
| `TaskGeneratorType` | Injected `TaskEligibility` |
|---|---|
| `POST_STREAM` | `spam_comment`, `reply_ranking` |
| `POST_STREAM_RECOVERY` | `banger_initial_screen` |
| `POST_STREAM_TEST` | `banger_initial_screen` |
| `POST_STREAM_DELAYED` | none (`{}`) |
| `POST_SAFETY_STREAM` | `post_safety` |
| `POST_MIN_TRACTION_STREAM_FOR_GROX` | `banger_initial_screen` |
| `POST_MIN_TRACTION_STREAM_FOR_GROX_PTOS` | `safety_ptos` |
| `POST_MIN_TRACTION_STREAM_FOR_GROX_MULTI_MODAL` | `post_embedding_with_summary_for_reply` |
| `POST_EMBEDDING_REQUEST_STREAM_WITH_SUMMARY` | `post_embedding_with_summary` |
| `POST_EMBEDDING_REQUEST_STREAM_WITH_SUMMARY_RECOVERY` | `post_embedding_with_summary` |
| `POST_EMBEDDING_REQUEST_STREAM_WITH_SUMMARY_FOR_REPLY_RECOVERY` | `post_embedding_with_summary_for_reply` |
| `POST_EMBEDDING_V5_STREAM` | `mm_emb_v5` |
| `POST_EMBEDDING_V5_FOR_REPLY_STREAM` | `mm_emb_v5_for_reply` |
| `REPLY_RANKING_RECOVERY` | `reply_ranking` |
| `SAFETY_PTOS_RECOVERY` | `safety_ptos` |
| `SAFETY_PTOS_DELUXE` | `safety_ptos` |
`TaskEligibility.MM_EMB_V4` exists on the enum and is not injected by any published generator and not required by any published plan.
`POST_STREAM_DELAYED` injects an empty eligibility set. Every `Plan.execute` returns `None`, `PlanMaster.merge_results` then calls `min` / `max` on an empty list, and the engine records a failed `TaskResult`.
### In-flight cap and retry
`grox_config.dispatcher.max_in_flight`. Fill loop sleeps 10ms while `len(_in_flights) >= max_in_flight`.
`grox_config.dispatcher.max_attempts`. Compared against `TaskPayload.attempt` (starts at `0`).
`grox_config.dispatcher.graceful_shutdown_timeout`. Parent `Process.join` timeout.
```mermaid
sequenceDiagram
participant Kafka
participant Gen as StreamTaskGenerator
participant Fill as Dispatcher._fill_loop
participant TQ as task_queue
participant Eng as Engine
participant PM as PlanMaster
participant RQ as resp_queue
participant Res as Dispatcher._result_loop
Kafka->>Gen: MessageQueuePayload
Gen->>Gen: eligibilities = ELIGIBILITIES_TO_INJECT
Fill->>Fill: wait while inflight >= max_in_flight
Fill->>TQ: put TaskPayload
Eng->>TQ: get_nowait
Eng->>PM: exec(task)
PM->>PM: gather ALL_PLANS.execute
PM->>PM: merge_results(non-None)
Eng->>RQ: put TaskResult
Res->>RQ: get_nowait
alt success
Res->>Gen: ack(payload_id, success=True)
else attempt < max_attempts
Res->>TQ: put same payload, attempt += 1
else final failure
Res->>Gen: ack(payload_id, success=False) if origin known
end
```
On success the payload id is removed from `_in_flights` and `PriorityTaskGenerator.ack` pops the origin label. On retry the id stays in `_in_flights` and is put on `task_queue` again. On final failure the dispatcher looks up origin via `_result_cache`; if missing, it logs and skips `ack`.
Published `KafkaLoader.ack` is a no-op (`pass`). The dispatcher still calls it.
## Engine execution
`Engine._run` non-blocking-gets from `task_queue`. Empty queue sleeps 100ms. Each payload is `asyncio.create_task`'d with no engine-side concurrency cap — the dispatcher in-flight set is the only backpressure.
`_process_task` is `PlanMaster.exec(task)`. Logging context keys: `task=payload_id`, `post=post.id` when present, `user=user.id` or `user_context.user.id`. A tracer span `task.root` is opened under `Metrics.tracer("engine")`.
If `PlanMaster.exec` raises, the engine still puts a `TaskResult` so the dispatcher can retry: `success=False`, `error=str(e)`, `task_finished_at` set to the pre-task `perf_counter`, `task_started_at` set to now. Those two timestamps are swapped relative to the field names.
Loop condition: `while not shutdown_event or not task_queue.empty()`. After the loop, `run()` calls `os._exit(0)` without awaiting outstanding `_run_task` tasks. In-flight plan work can be killed once the queue is empty and shutdown is set.
`Engine.stop()` in the parent joins `grox-engine` for `grox_config.engine.graceful_shutdown_timeout`, then awaits `MediaProcessor.stop()` and `ASRProcessor.stop()` (ASR stop default timeout is 5s). Those processors were started inside the child.
## PlanMaster fan-out
`PlanMaster.ALL_PLANS` is a class-level list of already-constructed plan instances, in this order:
1. `PlanInitialBanger`
2. `PlanPostSafety`
3. `PlanSpamComment`
4. `PlanPostEmbeddingWithSummary`
5. `PlanPostEmbeddingWithSummaryForReply`
6. `PlanPostEmbeddingV5`
7. `PlanPostEmbeddingV5ForReply`
8. `PlanReplyRanking`
9. `PlanSafetyPtos`
`exec` gathers every `plan.execute(task)` concurrently on the same `TaskPayload`. Ineligible plans return `None` and are dropped before merge. Gather order matches `ALL_PLANS`, which is also merge order.
## TaskEligibility gating
Each `Plan` declares `REQUIRED_ELIGIBILITY`. `Plan.execute` returns `None` immediately when that value is not in `task.eligibilities`. No `plan.execute.*` metrics are recorded for a skipped plan.
| Plan | `REQUIRED_ELIGIBILITY` |
|---|---|
| `PlanInitialBanger` | `banger_initial_screen` |
| `PlanPostSafety` | `post_safety` |
| `PlanSpamComment` | `spam_comment` |
| `PlanPostEmbeddingWithSummary` | `post_embedding_with_summary` |
| `PlanPostEmbeddingWithSummaryForReply` | `post_embedding_with_summary_for_reply` |
| `PlanPostEmbeddingV5` | `mm_emb_v5` |
| `PlanPostEmbeddingV5ForReply` | `mm_emb_v5_for_reply` |
| `PlanReplyRanking` | `reply_ranking` |
| `PlanSafetyPtos` | `safety_ptos` |
One payload can carry multiple eligibilities. `POST_STREAM` therefore runs `PlanSpamComment` and `PlanReplyRanking` in the same `PlanMaster.exec` call; the other seven plans return `None`.
Plan and task DAGs, classifier wiring, and per-plan `TASK_DEPENDENCIES` live on [Grox plans and tasks](/grox-plans-and-tasks). The runtime rule that matters here: every task name in `TASKS` is launched concurrently; a task waits on dependency futures; a dependency result of `TaskResultCategory.SKIPPED` skips the dependent; a raised exception is set on the future, re-raised, appended to `TaskContext.errors`, and marks the plan `success=False`. Leaf tasks (nothing depends on them) still run. `Task.exec` itself retries twice with a 1s wait via tenacity.
## Merged TaskResult
Stable id for inflight tracking and ack. New hex UUID per Kafka message.
Copied from the stream generator. Default empty set.
Retry counter. Starts at `0`. Dispatcher increments before resubmit.
Generator type that created the payload. Used by some deluxe/PTOS tasks.
`now + loader_config.task_deadline_secs` at Kafka decode. Carried on `TaskPayload`; Engine and Plan do not enforce it.
Eligible `Plan.execute` always returns a `TaskResult`:
The original payload.
`TaskContext.start_time` (`perf_counter` at context create). Merge takes `min`.
`perf_counter` at plan return. Merge takes `max`.
Merge concatenates every plan list, each entry `model_copy()`.
Merge keeps the first non-`None` embedding in `ALL_PLANS` order. Later embeddings are discarded.
Merge joins non-empty plan reasons with `\n`.
Plan: `len(ctx.errors) == 0`. Merge: `all(r.success)`.
Plan: `"\n".join(str(e) for e in ctx.errors)`. Merge: joins `r.error or "unknown error"` for unsuccessful plans only.
```json
{
"payload_id": "a1b2c3d4e5f64789a0b1c2d3e4f50617",
"attempt": 0,
"task_type": "POST_STREAM",
"eligibilities": ["spam_comment", "reply_ranking"],
"deadline_ts_secs": 1773600000
}
```
```json
{
"success": true,
"error": "",
"reason": "",
"content_categories": [],
"multimodal_post_embedding": null,
"task_started_at": 1024.11,
"task_finished_at": 1026.40
}
```
`merge_results` assumes `results` is non-empty. Zero eligible plans (`POST_STREAM_DELAYED`, or a payload whose eligibilities match no plan) raises inside `min` / `max` and becomes an engine-level failed `TaskResult`.
## Shutdown
Main `serve()` registers `SIGINT` and `SIGTERM` to set a local `asyncio.Event` named `shutdown`. Child processes ignore those signals.
After the signal, `queue_connection_shutdown_context` sets `queue_connection_shutdown_event`. Dispatcher stops all task generators (Kafka loaders). Main then `await asyncio.sleep(300)` before touching `shutdown_event`. Engine keeps consuming `task_queue` during that window.
`shutdown_context` sets `shutdown_event`. Dispatcher fill loop exits; result loop continues while `_in_flights` is non-empty. Engine loop exits when the queue is empty, then `os._exit(0)`.
Main gathers `grpc_server.stop()`, `dispatcher.stop()`, `engine.stop()`, then `cleanup()` shuts down the `SyncManager`. Join timeouts come from `grox_config.*.graceful_shutdown_timeout`.
Expected teardown logs: `Grox server shutting down...` then `Grox server stopped`.
## Referenced config
`grox.config.config.grox_config` is unpublished. These attributes are read by the published execution path:
| Attribute | Reader |
|---|---|
| `dispatcher.task_generators[].type` | `Dispatcher._get_task_generators` |
| `dispatcher.task_generators[].max_qps` | generator constructor |
| `dispatcher.task_generators[].weight` | `PriorityTaskGenerator` |
| `dispatcher.max_in_flight` | fill loop |
| `dispatcher.max_attempts` | result loop |
| `dispatcher.graceful_shutdown_timeout` | `Dispatcher.stop` |
| `engine.graceful_shutdown_timeout` | `Engine.stop` |
| `logging` | `init_proc` → `Logging.config` |
| `metrics` | `init_proc` → `Metrics.init` |
| `periodic_gc.interval`, `periodic_gc.jitter` | `periodic_gc` task |
| `asr.max_workers` | `ASRProcessor.start` |
| `get_kafka_loader_topic` / `get_kafka_consumer_topic` | `KafkaLoader` |
| loader `prefetching_threshold`, `prefetching_batch_size`, `task_deadline_secs` | Kafka prefetcher / deadline |
## Metrics
| Name | Kind | When |
|---|---|---|
| `dispatcher.inflight.count` | gauge | after add/remove of `_in_flights` |
| `dispatcher.task.sent.count` | counter | every put, attribute `task_type` |
| `dispatcher.result.received.count` | counter | every `resp_queue` get |
| `dispatcher.result.success.count` | counter | `result.success` |
| `dispatcher.result.failed.count` | counter | retry path |
| `dispatcher.result.failed.final.count` | counter | exhausted retries, attribute `origin` |
| `engine.task.received.count` | counter | successful `task_queue` get |
| `engine.task.success.count` | counter | `PlanMaster.exec` returned |
| `engine.task.failed.count` | counter | exception wrapper |
| `engine.task.processing_time` | histogram | seconds around `PlanMaster.exec` |
| `plan.execute.count` | counter | eligible plan start, `plan_name` |
| `plan.execute.success.count` | counter | no exception in gather |
| `plan.execute.failed.count` | counter | gather exception |
| `plan.execute.duration` | histogram | eligible plan wall time |
| `task.exec.count` / `.intaken` / `.success` / `.skipped` / `.failed` | counter | per `Task.exec`, `task_name` |
`plan_name` is `camel_to_snake` of the plan class (`PlanPostSafety` → `plan_post_safety`).
## Runtime boundary
These imports are required to start or execute plans and are not present under `grox/`:
- `grox.service.GrpcServer`
- `grox.config.config` (`grox_config`, `TaskGeneratorType`, `KafkaTopicName`)
- `grox.config.env` (task disable rules)
- `grox.data_loaders.data_types`
- `grox.data_loaders.media_processor.MediaProcessor`
- `monitor.logging`, `monitor.metrics`
- `kafka_cli.consumer`, `kafka_cli.config`, `thrifts.serdes`
`BrokenPipeError` on a manager `Event` or queue is treated as shutdown (`Engine` / `Dispatcher` `_is_shutdown` return `True`; poll methods return `None`).
## Failure modes
| Symptom | Cause | What to check |
|---|---|---|
| `ImportError` on `grox.service` / `grox.config` / `monitor` | Unpublished modules | This checkout cannot run Grox |
| `ValueError: Invalid task generator type` | Unknown `TaskGeneratorType` in config | Dispatcher match arms listed above |
| `ValueError: No generators provided` | Empty `task_generators` | Config list must be non-empty |
| `ValueError: All weights must be positive` | `weight <= 0` | Priority mixer |
| `ValueError: Not every task in TASK_DEPENDENCIES is defined in TASKS` | Plan DAG / `TASKS` mismatch at construct | Plan class body |
| Engine failed result with empty plans | No matching eligibility (`POST_STREAM_DELAYED` or unknown set) | `merge_results` empty `min`/`max` |
| Fill loop idle, engine idle | `len(_in_flights) >= max_in_flight` | `dispatcher.inflight.count` |
| Task retried then dropped | `attempt` reached `max_attempts` | `dispatcher.result.failed.final.count` |
| `No origin found ... skipping ack` | Missing `_result_cache` label | Priority mixer cache vs payload id |
| Child ignores Ctrl-C | `prevent_default()` in `init_proc` | Signal the main process |
| Engine dies mid-plan after drain | `os._exit(0)` when queue empty | 300s window then shutdown_event |
| Kafka ack appears to do nothing | `KafkaLoader.ack` is `pass` | Offset commit is unpublished |
## Next
`ALL_PLANS`, per-plan DAGs, `TaskEligibility` values, classifiers, and `TaskResult` merge rules.
What this checkout can run locally versus Home Mixer, Thunder, and Grox snapshots that need unpublished modules.
LFS/artifact failures, empty feeds, Thunder `RESOURCE_EXHAUSTED`, and unpublished Grox imports.
---
## 15. Add a pipeline component
> Source, Hydrator, Filter, Scorer, Selector, QueryHydrator, and SideEffect contracts, including length-match and no-drop rules.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/15-add-a-pipeline-component.md
- Generated: 2026-08-13T19:45:30.803Z
### Source Files
- `candidate-pipeline/candidate_pipeline.rs`
- `candidate-pipeline/source.rs`
- `candidate-pipeline/hydrator.rs`
- `candidate-pipeline/filter.rs`
- `candidate-pipeline/scorer.rs`
- `candidate-pipeline/selector.rs`
- `candidate-pipeline/query_hydrator.rs`
- `candidate-pipeline/side_effect.rs`
---
title: "Add a pipeline component"
description: "Source, Hydrator, Filter, Scorer, Selector, QueryHydrator, and SideEffect contracts, including length-match and no-drop rules."
---
`CandidatePipeline::execute` in `xai_candidate_pipeline` runs a typed `Q: PipelineQuery` / `C: PipelineCandidate` request through fixed stages. New behavior is a new trait object pushed into `PhoenixCandidatePipeline` or `ForYouCandidatePipeline`. Hydrators and scorers must return one result per input candidate in the same order; only a `Filter` or `Selector` may drop candidates.
Home Mixer and these traits depend on unpublished crates (`xai_candidate_pipeline::component_library`, feature-switch params, gRPC clients). This checkout can compile the trait sources and show registration, but it cannot run Home Mixer locally. The runnable surface in this repo is the Phoenix Python inference path.
## Choose a stage
| Stage | Trait | Parallelism | May drop candidates? | Failure behavior |
| --- | --- | --- | --- | --- |
| `QueryHydrator` | `QueryHydrator` | Parallel `hydrate`, then sequential `update` | n/a (query only) | `Err` is logged; that hydrator's `update` is skipped |
| `DependentQueryHydrator` | same trait, later wave | Same | n/a | Same |
| `Source` | `Source` | Parallel; results appended | n/a (creates candidates) | `Err` is logged and discarded by `flatten` |
| `Hydrator` / `PostSelectionHydrator` | `Hydrator` | Parallel `hydrate`, sequential `update_all` | **No** | Length mismatch skips the whole batch; per-item `Err` skips that `update` |
| `Filter` / `PostSelectionFilter` | `Filter` | Sequential | **Yes** — partition into `kept` / `removed` | Filters are sync and do not return `Result` |
| `Scorer` | `Scorer` | Sequential `run` + `update_all` | **No** | Same length-match skip as hydrators |
| `Selector` | `Selector` | Single | **Yes** — `selected` vs `non_selected` | If `enable` is false, all candidates stay selected |
| `SideEffect` | `SideEffect` | `tokio::spawn` + parallel | No effect on the returned feed | `join_all` result is discarded |
`PhoenixCandidatePipeline` fills every slot for `ScoredPostsQuery` / `PostCandidate`. `ForYouCandidatePipeline` uses `FeedItem` and returns empty slices for hydrators, filters, scorers, and post-selection stages; it only owns query hydrators, sources, `BlenderSelector`, and side effects.
```mermaid
flowchart TB
subgraph queryLayer [Query layer - parallel hydrate, sequential update]
QH["query_hydrators()"]
DQH["dependent_query_hydrators()"]
end
subgraph fetchLayer [Fetch - parallel append]
SRC["sources()"]
end
subgraph mutateLayer [Pre-score - hydrate cannot drop]
HYD["hydrators() parallel"]
FIL["filters() sequential"]
SCO["scorers() sequential"]
end
subgraph selectLayer [Select and trim]
SEL["selector()"]
PSH["post_selection_hydrators()"]
PSF["post_selection_filters()"]
TRUNC["result_size() split_off"]
end
subgraph effectLayer [After return]
SE["side_effects() spawned"]
end
QH --> DQH --> SRC --> HYD --> FIL --> SCO --> SEL --> PSH --> PSF --> TRUNC --> SE
```
## Shared surface
Every component is `Send + Sync`. `enable` defaults to `true`. `name()` defaults to the last `::` segment of `type_name_of_val(self)` via `util::short_type_name`. Tracing spans record `total_count`, `enabled_count`, and a comma-joined `disabled` list.
Skip the component for this request. `SideEffect::enable` takes `Arc`, not `&Q`.
Used in tracing, stats (`{name}.run`, `{name}.cache`, `{pipeline}.execute`), and `CandidatePipeline::components()`.
`Q` must implement `PipelineQuery` (`Clone + Send + Sync + 'static`, plus `params()` and `decider()`). `ScoredPostsQuery` is the published query. `C` is any `Clone + Send + Sync + 'static` type; Home Mixer uses `PostCandidate` in Phoenix and `FeedItem` in For You.
`CandidatePipeline` also requires:
Hard cap after post-selection filters. Excess candidates move to `SideEffectInput.non_selected_candidates`. Phoenix uses `params::RESULT_SIZE`; For You uses `params::FOR_YOU_MAX_RESULT_SIZE`.
Runs after truncation, before side effects. Neither published pipeline overrides it.
Second query-hydration wave. It sees fields written by the first wave. Neither published pipeline overrides this accessor.
## Length-match and no-drop
`Hydrator::hydrate` and `Scorer::score` take `&[C]` and must return `Vec>` with **the same length and order**. The trait comments state dropping is not allowed — use a filter.
`Hydrator::run` / `Scorer::run` enforce that:
1. If `result.len() == candidates.len()`, the vector is passed to `update_all`.
2. Otherwise the stage logs `Skipped: length_mismatch expected=N got=M`, replaces the vector with `N` copies of `Err("… length_mismatch expected=N got=M")`, and `update_all` copies **no** fields.
`update_all` zips by index and calls `update` only on `Ok`. A per-candidate `Err` leaves that candidate unchanged.
`CachedHydrator` has a second check: `hydrate_from_client` must return one result per cache miss. A miss-list mismatch returns `Err("CachedHydrator length_mismatch …")` for **every** input candidate, including cache hits.
Use a defaulted sibling struct in `hydrate` / `score` and copy only owned fields in `update`. Sibling hydrators run in parallel on the pre-hydration slice, so they cannot read each other's new fields. Scorers run sequentially, so later scorers can read earlier scores.
`CoreDataHydrationFilter` drops `author_id == 0` after `CoreDataCandidateHydrator`. Missing TES data is `Ok(PostCandidate::default())` (length preserved); the filter, not the hydrator, removes the candidate.
## QueryHydrator
Fills `ScoredPostsQuery` fields before sources run. All enabled hydrators receive the **same original query** in parallel. `update` then applies each `Ok` result in vec order.
```rust
#[async_trait]
pub trait QueryHydrator: Any + Send + Sync
where
Q: PipelineQuery,
{
fn enable(&self, _query: &Q) -> bool { true }
async fn hydrate(&self, query: &Q) -> Result;
fn update(&self, query: &mut Q, hydrated: Q);
}
```
`update` must copy only this hydrator's fields. `FollowedUserIdsQueryHydrator` writes `user_features.followed_user_ids`. `RetrievalSequenceQueryHydrator` writes `retrieval_sequence` and `columnar_retrieval_sequence`.
Query hydrators cannot see sibling results in the first wave. A hydrator that needs `followed_user_ids` or `retrieval_sequence` belongs in `dependent_query_hydrators()`, or it must not depend on another hydrator's output.
`PhoenixCandidatePipeline::build_with_clients` constructs `ImpressedPostsQueryHydrator` into `_impressed_posts_hydrator` and never inserts it. Construction is not registration.
## Source
Creates candidates. Enabled sources run in parallel; `Ok` vectors are appended. Source order in the vec is the append order.
```rust
#[async_trait]
pub trait Source: Any + Send + Sync
where
Q: PipelineQuery,
C: PipelineCandidate,
{
fn enable(&self, _query: &Q) -> bool { true }
async fn source(&self, query: &Q) -> Result, String>;
}
```
`Source::run` logs `Fetched N candidates` or `Failed: …`. A failed source contributes nothing; other sources still run.
Typical `enable` predicates:
- `ThunderSource`: `!query.has_cached_posts`
- `PhoenixSource`: not a non-bulk topic request, not new-user topic retrieval, `!query.in_network_only`, `!query.has_cached_posts`
`PhoenixSource` errors with `PhoenixSource: missing retrieval_sequence` when the query hydrator left that field empty. That is a source `Err`, not a hydrator length mismatch.
## Hydrator
Enriches candidates. Same trait is used for `hydrators()` and `post_selection_hydrators()`.
```rust
#[async_trait]
pub trait Hydrator: Any + Send + Sync {
async fn hydrate(&self, query: &Q, candidates: &[C]) -> Vec>;
fn update(&self, candidate: &mut C, hydrated: C);
}
```
`InNetworkCandidateHydrator` is the minimal pattern: map one output per input, set only `in_network`, copy that field in `update`.
TES-backed hydrators (`CoreDataCandidateHydrator`, `HasMediaHydrator`, `LanguageCodeHydrator`, …) implement `CachedHydrator` instead. The blanket `impl Hydrator for T: CachedHydrator` handles cache lookup. Implement:
| Method | Role |
| --- | --- |
| `cache_store` | `CacheStore` (Home Mixer uses `MokaCache`) |
| `cache_key` | Usually `candidate.tweet_id` |
| `cache_value` / `hydrate_from_cache` | Round-trip the owned fields |
| `hydrate_from_client` | One `Result` per **miss**, same order as the miss slice |
| `update` | Copy owned fields onto the live candidate |
Successful client hydrations are inserted into the cache. `Err` results are not cached.
Post-selection hydrators (`VFCandidateHydrator`, brand-safety, `TweetTypeMetricsHydrator`, …) run only on the selected set.
## Filter
The only pre-score stage that may remove candidates. Filters run **sequentially**; each sees the previous `kept` list.
```rust
pub struct FilterResult {
pub kept: Vec,
pub removed: Vec,
}
pub trait Filter: Any + Send + Sync {
fn filter(&self, query: &Q, candidates: Vec) -> FilterResult;
}
```
`kept` continues. `removed` accumulates into `PipelineResult.filtered_candidates` (pre-score removals plus post-selection removals). Post-selection removals are **not** added to `SideEffectInput.non_selected_candidates`.
`DropDuplicatesFilter` keeps first `tweet_id`. `SelfTweetFilter` drops `author_id == query.user_id`. `TopicIdsFilter` overrides `enable` to `query.is_topic_request() || query.has_excluded_topics()`.
`Filter::run` records `kept_count`, `removed_count`, `filter_rate`, and increments `{name}.run` kept/removed stats.
Phoenix pre-score filters: `DropDuplicatesFilter`, `CoreDataHydrationFilter`, `AgeFilter`, `SelfTweetFilter`, `RetweetDeduplicationFilter`, `IneligibleSubscriptionFilter`, `PreviouslySeenPostsFilter`, `PreviouslySeenPostsBackupFilter`, `PreviouslyServedPostsFilter`, `MutedKeywordFilter`, `AuthorSocialgraphFilter`, `VideoFilter`, `TopicIdsFilter`, `NewUserTopicIdsFilter`.
Phoenix post-selection filters: `VFFilter`, `AncillaryVFFilter`, `DedupConversationFilter`.
## Scorer
Writes score fields. Scorers run **one at a time**: `run` then `update_all`, then the next scorer. Later scorers may read earlier fields.
```rust
#[async_trait]
pub trait Scorer: Send + Sync {
async fn score(&self, query: &Q, candidates: &[C]) -> Vec>;
fn update(&self, candidate: &mut C, scored: C);
}
```
Phoenix registers `PhoenixScorer`, `RankingScorer`, then `VMRanker`. `PhoenixScorer` is disabled when `query.has_cached_posts`. If `scoring_sequence` is missing it still returns `vec![Ok(PostCandidate::default()); candidates.len()]` so length-match holds.
`RankingScorer::update` copies `weighted_score` and `score`. `TopKScoreSelector` reads `candidate.score.unwrap_or(f64::NEG_INFINITY)`.
`home-mixer/scorers/weighted_scorer.rs` and `oon_scorer.rs` exist on disk but are not in `scorers/mod.rs` and are not in the Phoenix scorer vec. Registering a scorer requires both the module and the vec.
## Selector
Exactly one selector per pipeline. Default `select` sorts by `score()` descending and truncates to `size()` when `Some`.
```rust
pub struct SelectResult {
pub selected: Vec,
pub non_selected: Vec,
}
pub trait Selector: Send + Sync {
fn score(&self, candidate: &C) -> f64;
fn size(&self) -> Option { None }
fn select(&self, query: &Q, candidates: Vec) -> SelectResult { /* sort + split_off */ }
}
```
`TopKScoreSelector` (Phoenix) uses `params::TOP_K_CANDIDATES_TO_SELECT`. `BlenderSelector` (For You) overrides `select` to blend ads / prompts / who-to-follow / push-to-home and does not use the default sort path (`score()` returns `0.0`).
If `selector.enable(query)` is false, every candidate is selected and `non_selected` is empty.
After selection, `execute` still truncates `final_candidates` to `result_size()` and appends the tail to `non_selected`.
## SideEffect
Runs after `finalize`. Cannot change `PipelineResult`. `execute` returns while the spawned task is still running.
```rust
pub struct SideEffectInput {
pub query: Arc,
pub selected_candidates: Vec,
pub non_selected_candidates: Vec,
}
#[async_trait]
pub trait SideEffect: Send + Sync {
fn enable(&self, _query: Arc) -> bool { true }
async fn side_effect(&self, input: Arc>) -> Result<(), String>;
}
```
`non_selected_candidates` is selector leftovers plus `result_size` overflow. It does not include filter removals.
`RedisPostCandidateCacheSideEffect::enable` is `is_prod() && !query.has_cached_posts`. `ScoredStatsSideEffect` always enables and samples stats from selected (and sometimes non-selected) candidates.
## Register the component
Create the file next to siblings:
- Phoenix query hydrators: `home-mixer/query_hydrators/`
- Phoenix candidate hydrators: `home-mixer/candidate_hydrators/`
- Phoenix filters: `home-mixer/filters/`
- Phoenix scorers: `home-mixer/scorers/`
- Phoenix sources: `home-mixer/sources/`
- Phoenix side effects: `home-mixer/side_effects/`
- For You extras: same trees; `ForYouCandidatePipeline::build` owns the For You vecs
Declare `pub mod …;` in that folder's `mod.rs`.
Use `ScoredPostsQuery` plus `PostCandidate` (Phoenix) or `FeedItem` (For You). Override `enable` when the component is request-conditional. Keep `hydrate` / `score` length-matched. Copy only owned fields in `update`.
Phoenix: `PhoenixCandidatePipeline::build_with_clients`. For You: `ForYouCandidatePipeline::build`.
```rust
// Phoenix hydrators vec — same pattern for every stage
hydrators.push(Box::new(YourHydrator { /* clients */ }));
```
For You hydrators, filters, scorers, and post-selection stages currently return `&[]`. Adding one of those types to For You also requires storing a `Vec` on the struct and changing the `CandidatePipeline` accessor.
Confirm `CandidatePipeline::components()` lists the new `name()`. Hydrator/scorer length bugs show up as `Skipped: length_mismatch` and unchanged candidate fields. Source `Err` only appears as a missing batch, not a pipeline abort. Side effects must not be required for a correct `PipelineResult`.
### Phoenix registration order
`PhoenixCandidatePipeline::build_with_clients` currently wires:
| Accessor | Types |
| --- | --- |
| `query_hydrators` | Scoring + retrieval sequences, blocked/muted/followed/subscribed IDs, cached posts, mutual follow, demographics, Grok topics, starter packs, inferred topics, impression bloom, IP, inferred gender |
| `sources` | `ThunderSource`, `TweetMixerSource`, `PhoenixSource`, `PhoenixTopicsSource`, `PhoenixMOESource`, `CachedPostsSource` |
| `hydrators` | In-network, TES core/quote/video/media/subscription/language, Gizmoduck, blocked-by, filtered topics |
| `filters` | Dedup through topic filters listed above |
| `scorers` | `PhoenixScorer`, `RankingScorer`, `VMRanker` |
| `selector` | `TopKScoreSelector` |
| `post_selection_hydrators` | VF, ads brand safety, tweet-type metrics, following-replied, mutual-follow Jaccard |
| `post_selection_filters` | VF, ancillary VF, conversation dedup |
| `side_effects` | Phoenix experiments Kafka, reranking Kafka, Redis cache, scored stats, mutual-follow stats, Phoenix request cache |
### For You registration order
| Accessor | Types |
| --- | --- |
| `query_hydrators` | `ServedHistoryQueryHydrator`, `PastRequestTimestampsQueryHydrator` |
| `sources` | `ScoredPostsSource`, `AdsSource`, `WhoToFollowSource`, `PromptsSource`, `PushToHomeSource` |
| `selector` | `BlenderSelector` |
| `side_effects` | Ads injection log, seen-ids Kafka, served-candidates Kafka, client events, For You stats, past-request timestamps, served-history update + truncate |
## Error and skip matrix
| Symptom | Likely cause |
| --- | --- |
| Component never appears in traces | Not in the pipeline vec, or `mod.rs` omitted |
| `disabled=` in the stage span | `enable` returned false |
| Hydrator/scorer fields stay `None` / default | Length mismatch, or `update` does not copy the field, or per-item `Err` |
| Query field stays default | `hydrate` returned `Err`, or two hydrators overwrite the same field, or the hydrator is first-wave but needs a sibling's output |
| Missing candidate batch, pipeline still returns | Source `Err` flattened away |
| Feed unchanged by a "post-process" | Implemented as `SideEffect` (fire-and-forget) instead of hydrator/filter/scorer |
| For You hydrator never runs | `ForYouCandidatePipeline::hydrators()` returns `&[]` |
`PipelineResult` after `execute`:
Candidates after the first hydrate wave, before pre-score filters.
Union of pre-score and post-selection `removed` lists.
Post-selection kept set after `result_size` truncation and `finalize`.
Query after both hydration waves.
## Next
Stage order inside `CandidatePipeline::execute` and how For You wraps Phoenix.
Enable predicates, cluster resolution, and `served_type` assignment.
Pre-score vs post-selection lists and query-hydrator field ownership.
`PhoenixScorer`, `RankingScorer`, diversity, and `TopKScoreSelector`.
What this checkout can run versus unpublished Home Mixer crates.
---
## 16. Phoenix model configuration
> PhoenixModelConfig and PhoenixRetrievalModelConfig fields, TransformerConfig, RecsysBatch, RecsysEmbeddings, and published config.json keys.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/16-phoenix-model-configuration.md
- Generated: 2026-08-13T19:45:48.005Z
### Source Files
- `phoenix/recsys_model.py`
- `phoenix/recsys_retrieval_model.py`
- `phoenix/grok.py`
- `phoenix/run_pipeline.py`
- `phoenix/runners.py`
- `phoenix/README.md`
- `phoenix/pyproject.toml`
---
title: "Phoenix model configuration"
description: "PhoenixModelConfig and PhoenixRetrievalModelConfig fields, TransformerConfig, RecsysBatch, RecsysEmbeddings, and published config.json keys."
---
`PhoenixModelConfig` and `PhoenixRetrievalModelConfig` are the Haiku construction records for the ranking and retrieval models. `run_pipeline.build_model_config` maps each published `config.json` into those records plus a nested `TransformerConfig` and `HashConfig`. The models do not read IDs or raw embedding tables: they consume a `RecsysBatch` of hashes and actions together with a `RecsysEmbeddings` of already-looked-up vectors.
`phoenix/artifacts/oss-phoenix-artifacts.zip` is a Git LFS pointer until pulled. After extract, treat `retrieval/config.json` and `ranker/config.json` as the runtime source of truth. The two READMEs disagree on mini-model size: `phoenix/README.md` documents 128-dim / 4 layers / 4 heads / key 32; the root README documents 256-dim / 2 layers / 4 heads. `build_model_config` does not hardcode those sizes.
## Construct from published artifacts
Unzip `phoenix/artifacts/oss-phoenix-artifacts.zip` so both `retrieval/config.json` and `ranker/config.json` sit next to their `model_params.npz` and `embedding_tables.npz`. See [Installation](/installation).
`run_pipeline.py` reads both files. Sequence lengths, `emb_size`, and `num_actions` used to allocate the live batch come from the **retrieval** JSON. Each model is still constructed from its own JSON.
`build_model_config(config, PhoenixRetrievalModelConfig)` or `build_model_config(config, PhoenixModelConfig)` fills `HashConfig` and `TransformerConfig`, then calls `initialize()`.
Hash IDs with `build_hash_functions`, index the unified table from `build_unified_emb_table`, wrap the rows in `RecsysEmbeddings`, and call `config.make()` inside a `hk.transform` forward.
## Config object graph
```mermaid
classDiagram
class TransformerConfig {
+emb_size int
+key_size int
+num_q_heads int
+num_kv_heads int
+num_layers int
+widening_factor float
+attn_output_multiplier float
+make() Transformer
}
class HashConfig {
+num_user_hashes int
+num_item_hashes int
+num_author_hashes int
+num_ip_hashes int
}
class PhoenixModelConfig {
+model TransformerConfig
+emb_size int
+num_actions int
+history_seq_len int
+candidate_seq_len int
+hash_config HashConfig
+make() PhoenixModel
}
class PhoenixRetrievalModelConfig {
+model TransformerConfig
+emb_size int
+history_seq_len int
+candidate_seq_len int
+enable_linear_proj bool
+hash_config HashConfig
+make() PhoenixRetrievalModel
}
class RecsysBatch {
+user_hashes
+history_* hashes actions surface
+candidate_* hashes surface
}
class RecsysEmbeddings {
+user_embeddings
+history_post_embeddings
+candidate_post_embeddings
+history_author_embeddings
+candidate_author_embeddings
}
PhoenixModelConfig --> TransformerConfig
PhoenixRetrievalModelConfig --> TransformerConfig
PhoenixModelConfig --> HashConfig
PhoenixRetrievalModelConfig --> HashConfig
PhoenixModelConfig ..> RecsysBatch : ranks
PhoenixRetrievalModelConfig ..> RecsysBatch : encodes user
PhoenixModelConfig ..> RecsysEmbeddings
PhoenixRetrievalModelConfig ..> RecsysEmbeddings
```
`TransformerConfig.make()` does not pass `emb_size` into `Transformer`. The stack width is the last dimension of the input embeddings (`[B, T, D]`), so `PhoenixModelConfig.emb_size` and `TransformerConfig.emb_size` must match the looked-up tables.
## PhoenixModelConfig
Ranking construction record. `make()` returns a `PhoenixModel` whose forward is `(RecsysBatch, RecsysEmbeddings) -> RecsysModelOutput`.
Nested transformer hyperparameters. Required; there is no default.
Embedding width `D`. Must match hash-table columns and `model.emb_size`.
Width of the discrete action head. Published JSON supplies this; `runners.ACTIONS` has 19 labels.
History positions `S`. Default `128`. Used for right-anchored RoPE when that flag is on; batch history must be padded to this length in the pipeline.
Candidate positions `C`. Default `32`. `run_pipeline.py` ranks in chunks of this length.
Optional label used only in the "not initialized" warning. Default `None`.
Forward dtype. Default `jnp.bfloat16`. `BaseModelRunner.initialize` also forces this to `bfloat16`.
Hash counts. Default `HashConfig()` (`2` user / item / author hashes, `0` IP hashes).
Rows in `product_surface_embedding_table`. Default `16`. Shared table for history and candidate surfaces.
Minutes per post-age bucket. Default `60`. Vocab size is `(4800 // granularity) + 2` (`POST_AGE_MAX_MINUTES = 4800`).
Columns of `continuous_unembeddings`. Default `8`. `runners.CONTINUOUS_ACTIONS[1]` is `dwell_time`.
Hidden width of the history-dwell MLP. Default `64`.
Normalization and unused training-loss knobs. Default `ContinuousActionConfig()` with `NormConfig(norm_scale=30.0, use_log=False)`.
Stored on the config. Default `False`. The forward path does **not** read this flag; IP addition is gated by `user_ip_embeddings is not None` and `hash_config.num_ip_hashes > 0`.
When `True`, `right_anchored_rope_positions` replaces sequential RoPE indices so the newest history token is fixed and every candidate shares position `1 + history_seq_len`. Default `False`. `build_model_config` does not set this from JSON.
Stored on the config. Default `True`. No consumer in `PhoenixModel.__call__`.
`initialize()` sets the initialized flag and returns `self`. `make()` warns and initializes if that flag is still false, then builds `PhoenixModel(model=self.model.make(), config=self, fprop_dtype=self.fprop_dtype)`.
### Ranker-only nested records
| Record | Field | Default | Used at inference |
|---|---|---|---|
| `NormConfig` | `norm_scale` | `30.0` | Yes — clamps then scales dwell |
| `NormConfig` | `use_log` | `False` | Yes — `log1p` path when true |
| `ContinuousActionConfig` | `loss_weight` | `0.0` | No |
| `ContinuousActionConfig` | `loss_type` | `"mae"` | No |
| `ContinuousActionConfig` | `tweedie_power` | `1.5` | No |
`normalize_continuous_value` clips to `[0, norm_scale]`, then divides by `norm_scale` or by `log1p(norm_scale)` when `use_log` is set. History dwell is taken from `history_continuous_actions[:, :, 1]`; if that tensor is missing, the ranker feeds zeros.
### Ranker outputs
`[B, C, num_actions]` discrete engagement logits from `unembeddings` (`[emb_size, num_actions]`).
`[B, C, num_continuous_actions]` — `sigmoid(candidate_embeddings @ continuous_unembeddings)`.
Candidate tokens start at offset `1 + S`. After the transformer, `layer_norm` is applied, then only the candidate slice is projected. Isolation masking is described on [Candidate isolation](/candidate-isolation).
## PhoenixRetrievalModelConfig
Two-tower construction record. The user tower is the same `Transformer` stack as the ranker; the candidate tower is `CandidateTower`.
User-tower transformer. Required.
Shared width `D` for user pooling and the candidate tower.
Default `128`.
Default `32`. Sizes the dummy candidate slots `run_pipeline.py` concatenates with 64 Gaussian-noise rows so `candidate_tower_projection_*` stay in the Haiku tree.
Default `jnp.bfloat16`.
Default `HashConfig()`.
Default `16`. History product-surface table only; candidates are not transformer tokens on this path.
`CandidateTower` mode. Default `True`. `build_model_config` **forces** `True` for published checkpoints and does not read a JSON key.
The retrieval transformer is called with `candidate_start_offset=None`, so it uses a standard causal mask over `[user, history]` only. User representation is the masked mean of those outputs, then L2-normalized (`EPS = 1e-12`).
`CandidateTower` when `enable_linear_proj` is true: flatten hash embeddings, SiLU MLP (`D_in → 2D → D`), L2-normalize. When false: mean over the hash axis, L2-normalize, no learned parameters.
`[B, D]` L2-normalized user vector.
`[B, K]` corpus indices from `jax.lax.top_k` on `user @ corpus.T`. Invalid corpus rows are set to `-1e12` when `corpus_mask` is passed.
`[B, K]` corresponding dot products.
`run_pipeline.py` does not call that `__call__` path. It uses `build_user_representation` and scores `corpus["candidate_representations"]` in NumPy.
## TransformerConfig
Defined in `phoenix/grok.py`. Ported Grok-1 decoder stack with recsys attention when `candidate_start_offset` is set.
Documented width. Not forwarded into `Transformer.__init__`.
Per-head RoPE / key dim. `model_size` defaults to `key_size * num_q_heads`.
Query heads. Must be a multiple of `num_kv_heads`.
Key/value heads. Published JSON sets both head counts from one `num_heads` key.
`decoder_layer_{i}` count.
FFN expansion. Dataclass default `4.0`. Published loader hardcodes `2.0`.
Multiplies attention logits before a `30 * tanh(x / 30)` clip. Dataclass default `1.0`. Published loader hardcodes `0.125`.
`ffn_size(emb_size, widening_factor)` is `int(widening_factor * emb_size) * 2 // 3`, then rounded up to a multiple of 8. Local demos in `run_ranker.py` / `run_retrieval.py` use `emb_size=128`, `widening_factor=2`, `key_size=64`, `num_q_heads=num_kv_heads=2`, `num_layers=2`, `attn_output_multiplier=0.125`, `history_seq_len=32`, `candidate_seq_len=8` — those are untrained demo configs, not the checkpoint.
## HashConfig
Default `2`. Width of `user_hashes` / `user_embeddings`.
Default `2`. Width of post-hash tensors.
Default `2`. Width of author-hash tensors.
Default `0`. Ranker `block_user_reduce` sums IP embeddings into the user token only when this is `> 0` and `user_ip_embeddings` is present.
Hash `0` is padding. Reduce functions take validity from the first hash lane (`[..., 0] != 0`). Linear-congruential hashing, pad offset `65`, and the unified table are specified on [Hash embeddings](/hash-embeddings).
## RecsysBatch
`NamedTuple` of feature indices. Embeddings travel separately.
| Field | Shape | Required |
|---|---|---|
| `user_hashes` | `[B, num_user_hashes]` | yes |
| `history_post_hashes` | `[B, S, num_item_hashes]` | yes |
| `history_author_hashes` | `[B, S, num_author_hashes]` | yes |
| `history_actions` | `[B, S, num_actions]` | yes — multi-hot; retrieval infers `num_actions` from this axis |
| `history_product_surface` | `[B, S]` | yes |
| `candidate_post_hashes` | `[B, C, num_item_hashes]` | yes |
| `candidate_author_hashes` | `[B, C, num_author_hashes]` | yes |
| `candidate_product_surface` | `[B, C]` | yes |
| `history_continuous_actions` | `[B, S, num_continuous_actions]` | no — ranker dwell at index `1` |
| `candidate_impr_ts` | `[B, C]` | no — seconds; both timestamp fields required to bucket age |
| `candidate_post_creation_ts` | `[B, C]` | no |
| `user_ip_hashes` | `[B, num_ip_hashes]` | no — carried on the batch; lookup is the caller's job |
Action embeddings are `(2 * actions - 1) @ action_projection`, then zeroed where the whole action vector is false. Signed `0/1` inputs therefore become `-1/+1`.
`run_pipeline.py` sets every product-surface index to `0` and omits timestamps and continuous actions. Missing age timestamps become bucket `0` (the missing/invalid bucket). Negative age and zero timestamps also map to `0`. Overflow age (`> 4800` minutes at 60-minute granularity) is bucket `81`.
## RecsysEmbeddings
Dataclass of pre-looked-up rows. `block_*_reduce` concatenates hash lanes and projects with `proj_mat_1` (user), `proj_mat_3` (history), `proj_mat_2` (ranker candidates).
| Field | Shape |
|---|---|
| `user_embeddings` | `[B, num_user_hashes, D]` |
| `history_post_embeddings` | `[B, S, num_item_hashes, D]` |
| `candidate_post_embeddings` | `[B, C, num_item_hashes, D]` |
| `history_author_embeddings` | `[B, S, num_author_hashes, D]` |
| `candidate_author_embeddings` | `[B, C, num_author_hashes, D]` |
| `user_ip_embeddings` | optional `[B, num_ip_hashes, D]` |
The pipeline indexes a unified table: `ret_emb[user_hashes]`, `ret_emb[hist_post_h]`, and so on. Dummy candidate embeddings are zeros during retrieval user encoding.
## Published config.json keys
Both `retrieval/config.json` and `ranker/config.json` are consumed by the same helpers. Keys below are those the loader actually reads.
### Required by `build_model_config`
| Key | Maps to |
|---|---|
| `emb_size` | `Phoenix*ModelConfig.emb_size` and `TransformerConfig.emb_size` |
| `history_seq_len` | `history_seq_len` |
| `candidate_seq_len` | `candidate_seq_len` |
| `num_user_hashes` | `HashConfig.num_user_hashes` |
| `num_item_hashes` | `HashConfig.num_item_hashes` |
| `num_author_hashes` | `HashConfig.num_author_hashes` |
| `key_size` | `TransformerConfig.key_size` |
| `num_heads` | both `num_q_heads` and `num_kv_heads` |
| `num_layers` | `TransformerConfig.num_layers` |
| `num_actions` | **ranker only** — `PhoenixModelConfig.num_actions`. Retrieval still reads this key at the top of `main()` to size `history_actions`. |
### Optional JSON keys
| Key | Default if absent |
|---|---|
| `product_surface_vocab_size` | `16` |
| `post_age_granularity_mins` | `60` (ranker only) |
### Required by hashing and the unified table
| Key | Role |
|---|---|
| `user_vocab_size` | user table rows; also the item/author index offsets |
| `item_vocab_size` | item table rows |
| `author_vocab_size` | author table rows |
| `hash_params.user_hash_scales` | per-hash LCG scales |
| `hash_params.user_biases` | per-hash LCG biases |
| `hash_params.user_modulus` | LCG modulus |
| `hash_params.item_hash_scales` | item LCG scales |
| `hash_params.item_biases` | item LCG biases |
| `hash_params.item_modulus` | item modulus |
| `hash_params.author_hash_scales` | author LCG scales |
| `hash_params.author_biases` | author LCG biases |
| `hash_params.author_modulus` | author modulus |
### Hardcoded in the loader, not JSON
| Value | Where |
|---|---|
| `widening_factor=2.0` | `TransformerConfig` in `build_model_config` |
| `attn_output_multiplier=0.125` | same |
| `enable_linear_proj=True` | retrieval branch |
| pad offset `65` | `build_hash_functions` / `build_unified_emb_table` |
```text
unified table rows = 65 + user_vocab_size + item_vocab_size + author_vocab_size
[0 .. 64] [65 .. 65+U) [65+U .. 65+U+I) [65+U+I .. end)
pad zeros user_embeddings item_embeddings author_embeddings
```
ID `0` stays hash `0` (padding). Non-zero IDs hash into `[1, vocab)` then add the pad/user/item offset. `embedding_tables.npz` must contain `user_embeddings`, `item_embeddings`, and `author_embeddings`.
```python
# phoenix/run_pipeline.py — published JSON → live config
kwargs = dict(
emb_size=config["emb_size"],
history_seq_len=config["history_seq_len"],
candidate_seq_len=config["candidate_seq_len"],
hash_config=HashConfig(
num_user_hashes=config["num_user_hashes"],
num_item_hashes=config["num_item_hashes"],
num_author_hashes=config["num_author_hashes"],
),
product_surface_vocab_size=config.get("product_surface_vocab_size", 16),
model=TransformerConfig(
emb_size=config["emb_size"],
key_size=config["key_size"],
num_q_heads=config["num_heads"],
num_kv_heads=config["num_heads"],
num_layers=config["num_layers"],
widening_factor=2.0,
attn_output_multiplier=0.125,
),
)
if config_class == PhoenixModelConfig:
kwargs["num_actions"] = config["num_actions"]
kwargs["post_age_granularity_mins"] = config.get("post_age_granularity_mins", 60)
elif config_class == PhoenixRetrievalModelConfig:
kwargs["enable_linear_proj"] = True
mc = config_class(**kwargs)
mc.initialize()
```
### Documented mini-checkpoint table
`phoenix/README.md` lists these published mini values. Confirm against extracted JSON before depending on them.
| Parameter | Documented mini value |
|---|---|
| Embedding dimension | 128 |
| Transformer layers | 4 |
| Attention heads | 4 |
| Key size | 32 |
| Widening factor | 2 |
| History sequence length | 127 |
| Candidate sequence length | 64 |
| User / item / author vocab | 1,000,000 each |
| Hashes per entity | 2 |
| Action types | 19 |
Dataclass defaults differ (`history_seq_len=128`, `candidate_seq_len=32`). Production Phoenix is described as a larger continuously trained model; this checkout ships a frozen mini snapshot.
## Reduce projections and parameter names
These Haiku names must exist in the matching `model_params.npz` after `hk.transform`:
| Name | Owner | Shape intent |
|---|---|---|
| `proj_mat_1` | user reduce | `[num_user_hashes * D, D]` |
| `proj_mat_3` | history reduce | `[concat_width, D]` |
| `proj_mat_2` | ranker candidate reduce | `[concat_width, D]` |
| `action_projection` | both | `[num_actions, D]` |
| `product_surface_embedding_table` | both | `[product_surface_vocab_size, D]` |
| `unembeddings` | ranker | `[D, num_actions]` |
| `continuous_unembeddings` | ranker | `[D, num_continuous_actions]` |
| `history_dwell_time_proj1` / `_proj2` | ranker | `[1, hidden]` / `[hidden, D]` |
| `post_age_embedding_table` | ranker | `[post_age_vocab_size, D]` |
| `candidate_tower_projection_1` / `_2` | retrieval | `[hash_concat, 2D]` / `[2D, D]` |
| `log_temperature` | retrieval forward in `run_pipeline.py` | scalar, initialized to `0` so the published key is consumed |
History concat (ranker) is `[post hashes | author hashes | action emb | surface emb | dwell emb]`. Retrieval history omit dwell. Ranker candidates add post-age embeddings; they do not take action embeddings.
## Constraints and failures
- **Missing JSON key.** `build_model_config` / `build_hash_functions` raise `KeyError` on any required key listed above.
- **Uninitialized `make()`.** Logs `PhoenixModel {name} is not initialized. Initializing.` (or the retrieval equivalent) and continues.
- **Vocab / offset mismatch.** Hashes that land past `65 + U + I + A` index off the unified table.
- **Sequence-length split.** Live `S` and `C` come from **retrieval** JSON even when the ranker JSON differs. Rank in chunks of `candidate_seq_len`; a short last chunk is zero-padded.
- **Separate hash spaces.** Retrieval and ranker each have their own `hash_params` and embedding tables. Do not mix `ret_emb` rows with `rank_hash_*` indices.
- **Dtype.** Inference runners force `fprop_dtype = bfloat16`.
- **IP and negative-feedback flags.** `use_ip_address` and `mask_neg_feedback_on_negatives` are not wired in the published forward.
- **LFS.** Until `git lfs pull`, `config.json` is not on disk. See [Troubleshooting](/troubleshooting).
Verify a constructed config by running `uv run pytest test_recsys_model.py test_recsys_retrieval_model.py` and, after extract, `uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts`. The ranked table is the success signal.
## Related pages
LCG hashing, pad-offset unified tables, and RecsysBatch lookup before the transformer.
`make_recsys_attn_mask` rules so candidate scores stay batch-independent.
Load both checkpoints, encode example_sequence.json, retrieve, rank.
Per-action logits, demo weighted sums, and production weight tables.
example_sequence.json fields and history padding to history_seq_len.
Attention-mask, retrieval, and CandidateTower assertions.
---
## 17. Action indices
> ActionName indices used in example_sequence.json, run_pipeline.py IDX_* constants, runners.ACTIONS labels, and scorer weight keys.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/17-action-indices.md
- Generated: 2026-08-13T19:48:07.373Z
### Source Files
- `phoenix/run_pipeline.py`
- `phoenix/runners.py`
- `phoenix/README.md`
- `home-mixer/scorers/weighted_scorer.rs`
- `home-mixer/scorers/ranking_scorer.rs`
- `phoenix/recsys_model.py`
---
title: "Action indices"
description: "ActionName indices used in example_sequence.json, run_pipeline.py IDX_* constants, runners.ACTIONS labels, and scorer weight keys."
---
Phoenix stores each history item as a multi-hot vector of length `num_actions` and emits ranking logits of shape `[B, num_candidates, num_actions]`. The published `run_pipeline.py` path indexes that last dimension with proto `ActionName` ordinals. `runners.ACTIONS` and Home Mixer `PhoenixScores` name the same engagement types as fields, not as those ordinals.
Do not treat `runners.ACTIONS[i]` as `ActionName` value `i`. Favorite is ordinal `1` (`IDX_FAV`) in `example_sequence.json` and `run_pipeline.py`, and slot `0` (`favorite_score`) in `RecsysInferenceRunner`. Mixing the two spaces writes the wrong history bit or reads the wrong logit column.
The `ActionName` proto is not in this checkout. Only the six ordinals named on the `IDX_*` constants and in `phoenix/README.md` are documented here.
## Index spaces
| Space | Used by | Coordinate | Width |
| --- | --- | --- | --- |
| `ActionName` ordinals | `example_sequence.json` keys, `history_actions[..., idx]`, `run_pipeline.py` logit columns | Integer proto enum values | `config.json` `num_actions` (documented mini width `19`) |
| `runners.ACTIONS` | `run_ranker.py`, `run_retrieval.py`, `RecsysInferenceRunner.rank_candidates` | Dense `0..len(ACTIONS)-1` | `len(ACTIONS)` = `19` |
| `PhoenixScores` fields | `PhoenixScorer` output, `WeightedScorer`, `RankingScorer`, Kafka / experiment logs | Named `*_score` / `*_time` fields | 19 fields on the local snapshot; production crate adds more |
```text
ActionName last-dim (example_sequence.json / run_pipeline.py)
1 SERVER_TWEET_FAV 4 SERVER_TWEET_REPLY
5 SERVER_TWEET_QUOTE 6 SERVER_TWEET_RETWEET
11 CLIENT_TWEET_RECAP_DWELLED 13 CLIENT_TWEET_VIDEO_QUALITY_VIEW
other slots exist in [0, num_actions) and are unnamed in this repo
runners.ACTIONS last-dim (run_ranker.py dummy)
0 favorite_score 1 reply_score 2 repost_score ... 18 dwell_time
Home Mixer PhoenixScores (named fields, not indices)
favorite_score, reply_score, retweet_score, ...
```
`num_actions` is not hardcoded in `PhoenixModel`. `run_pipeline.py` reads it from `retrieval/config.json` and sizes both the history tensor and the ranker unembedding from that width. `PhoenixModelConfig.num_actions` is set from `ranker/config.json`. Tests and dummy runners default to `19`. `BaseInferenceRunner._get_num_actions()` also falls back to `19` when the model config has no `num_actions` (retrieval).
## ActionName ordinals
These are the only proto names this repository prints. Use them as JSON keys and as `all_probs[:, idx]` columns after `run_pipeline.py` applies `jax.nn.sigmoid` to ranker logits.
| Ordinal | `IDX_*` | Proto comment | Demo weight | Pipeline table column |
| --- | --- | --- | --- | --- |
| `1` | `IDX_FAV` | `SERVER_TWEET_FAV` | `1.0` | `Fav` |
| `4` | `IDX_REPLY` | `SERVER_TWEET_REPLY` | `0.5` | `Reply` |
| `5` | `IDX_QUOTE` | `SERVER_TWEET_QUOTE` | unused | not printed |
| `6` | `IDX_RT` | `SERVER_TWEET_RETWEET` | `0.3` | `RT` |
| `11` | `IDX_DWELL` | `CLIENT_TWEET_RECAP_DWELLED` | `0.2` | `Dwell` |
| `13` | `IDX_VQV` | `CLIENT_TWEET_VIDEO_QUALITY_VIEW` | unused | `VQV` |
`1`. Favorite / like. Included in the demo weighted sum at `1.0`.
`4`. Reply. Demo weight `0.5`.
`5`. Quote. Defined for encoding; not used in the demo weighted sum or printed columns.
`6`. Retweet / repost. Demo weight `0.3`.
`11`. Recap dwelled. Demo weight `0.2`.
`13`. Video quality view. Printed only. Production `WeightedScorer` / `RankingScorer` gate VQV on video duration.
The published demo score is:
```python
weighted = (
all_probs[:, IDX_FAV] * 1.0
+ all_probs[:, IDX_REPLY] * 0.5
+ all_probs[:, IDX_RT] * 0.3
+ all_probs[:, IDX_DWELL] * 0.2
)
```
Those four coefficients are local to `run_pipeline.py`. They are not the Home Mixer weight tables.
## History encoding
`run_pipeline.py` loads `--sequence_file` (default `/example_sequence.json`) and writes a left-aligned `[history_seq_len, num_actions]` matrix. Extra history items past `history_seq_len` are dropped. Missing keys stay `0.0`. Keys `>= num_actions` are ignored.
Viewer id hashed into user embeddings.
Item id hashed into history post embeddings. Padding slots stay `0`.
Author id hashed into history author embeddings.
Map of stringified `ActionName` ordinal → float. Typical present value is `1.0`.
```json example_sequence.json
{
"user_id": 123456789,
"history": [
{
"post_id": 2001,
"author_id": 3001,
"actions": { "1": 1.0, "11": 1.0 }
}
]
}
```
That example sets favorite (`1`) and dwell (`11`) on one post. The shipped artifact sequence is described as three sports posts the user liked and dwelled on; the zip itself is a Git LFS pointer until you extract `oss-phoenix-artifacts`.
```python
history_actions = np.zeros((hist_len, num_actions), dtype=np.float32)
for i, item in enumerate(history[:hist_len]):
history_post_ids[i] = item["post_id"]
history_author_ids[i] = item["author_id"]
for act_idx_str, act_val in item.get("actions", {}).items():
idx = int(act_idx_str)
if idx < num_actions:
history_actions[i, idx] = float(act_val)
```
Both the retrieval user tower and the ranker project that vector with a learned `action_projection` of shape `[num_actions, D]`:
```python
actions_signed = (2 * actions - 1) # 0 → -1, 1 → +1
action_emb = actions_signed @ action_projection
action_emb = action_emb * jnp.any(actions, axis=-1, keepdims=True)
```
An all-zero action row (padding, or a history item with no `actions`) contributes a zero action embedding. `run_pipeline.py` does not populate `history_continuous_actions`, so the ranker's continuous dwell head sees zeros for history.
Use `"1"` for favorite, not `"0"`. A `"0": 1.0` bit is a valid last-dim slot but is not a documented favorite.
`if idx < num_actions` silently drops anything at or above the published width. Confirm `num_actions` in `retrieval/config.json` and `ranker/config.json` after extract.
The encoder allocates `history_seq_len` zeros, then fills `history[:hist_len]`. You do not pad the JSON.
After `uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts`, the printed `Fav` / `Reply` / `RT` / `Dwell` / `VQV` columns are `all_probs` at ordinals `1`, `4`, `6`, `11`, `13`.
## runners.ACTIONS labels
`ACTIONS` is the dummy ranker / retrieval vocabulary. `run_ranker.py` and `run_retrieval.py` set `num_actions = len(ACTIONS)` and print labels in this order. `RecsysInferenceRunner` maps `probs[:, :, i]` onto `p_` and ranks by column `0`.
| Dense index | `ACTIONS` label | `RankingOutput` field |
| --- | --- | --- |
| `0` | `favorite_score` | `p_favorite_score` |
| `1` | `reply_score` | `p_reply_score` |
| `2` | `repost_score` | `p_repost_score` |
| `3` | `photo_expand_score` | `p_photo_expand_score` |
| `4` | `click_score` | `p_click_score` |
| `5` | `profile_click_score` | `p_profile_click_score` |
| `6` | `vqv_score` | `p_vqv_score` |
| `7` | `share_score` | `p_share_score` |
| `8` | `share_via_dm_score` | `p_share_via_dm_score` |
| `9` | `share_via_copy_link_score` | `p_share_via_copy_link_score` |
| `10` | `dwell_score` | `p_dwell_score` |
| `11` | `quote_score` | `p_quote_score` |
| `12` | `quoted_click_score` | `p_quoted_click_score` |
| `13` | `follow_author_score` | `p_follow_author_score` |
| `14` | `not_interested_score` | `p_not_interested_score` |
| `15` | `block_author_score` | `p_block_author_score` |
| `16` | `mute_author_score` | `p_mute_author_score` |
| `17` | `report_score` | `p_report_score` |
| `18` | `dwell_time` | `p_dwell_time` |
`NEGATIVE_FEEDBACK_INDICES = [14, 15, 16, 17]` refers to this dense space (`not_interested`, `block_author`, `mute_author`, `report`). It is not an `ActionName` table.
`CONTINUOUS_ACTIONS` is a separate length-8 vocabulary for `history_continuous_actions`:
| Continuous index | Label |
| --- | --- |
| `0` | `reserved` |
| `1` | `dwell_time` |
| `2` | `video_watch_time` |
| `3` | `scroll_depth` |
| `4` | `reserved_3` |
| `5` | `reserved_4` |
| `6` | `reserved_5` |
| `7` | `reserved_6` |
`PhoenixModel` reads only index `1` (`dwell_time`) from history. The ranker also predicts `continuous_preds` of width `PhoenixModelConfig.num_continuous_actions` (default `8`) via `continuous_unembeddings`. Dummy `create_example_batch(..., include_continuous_actions=True)` writes exponential noise into `[:, :, 1]`.
## PhoenixScores and scorer weight keys
Home Mixer never indexes logits by integer. `PhoenixScorer` fills `PostCandidate.phoenix_scores`, then `WeightedScorer` / `RankingScorer` multiply named fields by weight constants or feature-switch params.
The local snapshot in `home-mixer/candidate_pipeline/candidate.rs` has these optional `f64` fields:
| Field | Kafka / experiment key | `WeightedScorer` weight | `RankingScorer` weight |
| --- | --- | --- | --- |
| `favorite_score` | `favorite` | `FAVORITE_WEIGHT` | `FavoriteWeight` |
| `reply_score` | `reply` | `REPLY_WEIGHT` | `ReplyWeight` |
| `retweet_score` | `retweet` | `RETWEET_WEIGHT` | `RetweetWeight` |
| `photo_expand_score` | `photo_expand` | `PHOTO_EXPAND_WEIGHT` | `PhotoExpandWeight` |
| `click_score` | `click` | `CLICK_WEIGHT` | `ClickWeight` |
| `profile_click_score` | `profile_click` | `PROFILE_CLICK_WEIGHT` | `ProfileClickWeight` |
| `vqv_score` | `vqv` | `VQV_WEIGHT` if `video_duration_ms > MIN_VIDEO_DURATION_MS`, else `0` | `VqvWeight` via `vqv_weight()` |
| `share_score` | `share` | `SHARE_WEIGHT` | `ShareWeight` |
| `share_via_dm_score` | `share_via_dm` | `SHARE_VIA_DM_WEIGHT` | `ShareViaDmWeight` |
| `share_via_copy_link_score` | `share_via_copy_link` | `SHARE_VIA_COPY_LINK_WEIGHT` | `ShareViaCopyLinkWeight` |
| `dwell_score` | `dwell` | `DWELL_WEIGHT` | `DwellWeight` |
| `quote_score` | `quote` | `QUOTE_WEIGHT` | `QuoteWeight` |
| `quoted_click_score` | `quoted_click` | `QUOTED_CLICK_WEIGHT` | `QuotedClickWeight` |
| `follow_author_score` | `follow_author` | `FOLLOW_AUTHOR_WEIGHT` | `FollowAuthorWeight` |
| `not_interested_score` | `not_interested` | `NOT_INTERESTED_WEIGHT` | `NotInterestedWeight` (negative group) |
| `block_author_score` | `block_author` | `BLOCK_AUTHOR_WEIGHT` | `BlockAuthorWeight` (negative group) |
| `mute_author_score` | `mute_author` | `MUTE_AUTHOR_WEIGHT` | `MuteAuthorWeight` (negative group) |
| `report_score` | `report` | `REPORT_WEIGHT` | `ReportWeight` (negative group) |
| `dwell_time` | `dwell_time` | `CONT_DWELL_TIME_WEIGHT` | `ContDwellTimeWeight` |
`ACTIONS` uses `repost_score` for the same engagement `PhoenixScores` and Kafka call `retweet_score` / `retweet`. Missing scores become `0.0` (`score.unwrap_or(0.0) * weight`).
Production `RankingScorer` and `vm_ranker.rs` also read fields that exist on the unpublished `xai_candidate_pipeline` `PhoenixScores` (re-exported from `home-mixer/models/candidate.rs`), not on the local snapshot:
| Extra field | Kafka / experiment key | `RankingScorer` weight |
| --- | --- | --- |
| `quoted_vqv_score` | `quoted_vqv` | `QuotedVqvWeight` via `quoted_vqv_weight()` |
| `click_dwell_time` | not in the Kafka insert list | `ContClickDwellTimeWeight` |
| `not_dwelled_score` | `not_dwelled` | `NotDwelledWeight` (negative group) |
`RankingScorer` treats `not_interested + block_author + mute_author + report + not_dwelled` as the negative sum used by `offset_score`. Numeric weight values live in unpublished `crate::params` / feature switches, not in this checkout.
`PhoenixScorer` returns default (empty) scores when `query.scoring_sequence` is missing. `ScoringSequenceQueryHydrator` loads that sequence from the unpublished user-action aggregation service; it does not parse `example_sequence.json`.
## Constraints
- History JSON keys are strings of integers. `int("1")` is `ActionName` favorite; `"favorite_score"` raises `ValueError`.
- Last-dim width must match the checkpoint unembedding `[emb_size, num_actions]`. Retrieval and ranker each have their own `config.json`.
- `run_pipeline.py` uses retrieval `num_actions` for the history tensor passed to both stages.
- Dummy `run_ranker.py` ranks by `probs[:, :, 0]` (`ACTIONS` favorite). The published pipeline ranks by the four-term `ActionName` weighted sum.
- `mask_neg_feedback_on_negatives` exists on `PhoenixModelConfig` (default `True`) and is unused in the published forward pass.
- Production VQV weight is zero for short or missing video; the demo always prints `IDX_VQV` and never multiplies it.
## Errors and verification
| Symptom | Cause |
| --- | --- |
| History has no effect | All-zero `actions`, or keys that do not match `ActionName` ordinals |
| Action silently missing | Key `>= num_actions` |
| Favorite column looks empty after editing JSON | Wrote `"0"` (dense `ACTIONS` favorite) instead of `"1"` |
| `FileNotFoundError` on the sequence file | Artifacts not extracted; default path is `/example_sequence.json` |
| Shape error on `history_actions` | `num_actions` mismatch between the JSON-built tensor and the loaded `action_projection` / `unembeddings` |
Success signal for the published path is the ranked table: `Score` is the four-term weighted sum, `Ret` is retrieval dot-product, and `Fav` / `Reply` / `RT` / `Dwell` / `VQV` are sigmoid probabilities at ordinals `1`, `4`, `6`, `11`, `13`. Dummy `run_ranker.py` instead prints all 19 `ACTIONS` labels ordered by predicted favorite at dense index `0`. Retrieval tests construct batches with `num_actions = 19`.
## Next
`example_sequence.json` fields, padding to `history_seq_len`, and `--top_k_retrieval` / `--top_k_display`.
Per-action logits, the demo weighted sum, and production `WeightedScorer` versus `RankingScorer`.
Cluster fallback, weight formulas, VQV gates, author diversity, and `TopKScoreSelector`.
Load checkpoints, encode the sequence, retrieve from `sports_corpus.npz`, and print the ranked table.
---
## 18. ScoredPostsQuery and gRPC
> ScoredPostsQuery fields, QueryBuilder defaults, TEST_USER_IDS empty responses, and ScoredPost / ForYouFeed / URT response mapping.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/18-scoredpostsquery-and-grpc.md
- Generated: 2026-08-13T19:49:19.541Z
### Source Files
- `home-mixer/models/query.rs`
- `home-mixer/server.rs`
- `home-mixer/scored_posts_server.rs`
- `home-mixer/for_you_server.rs`
- `home-mixer/main.rs`
- `home-mixer/models/candidate.rs`
- `home-mixer/models/user_features.rs`
---
title: "ScoredPostsQuery and gRPC"
description: "ScoredPostsQuery fields, QueryBuilder defaults, TEST_USER_IDS empty responses, and ScoredPost / ForYouFeed / URT response mapping."
---
Home Mixer serves ranked posts through two gRPC services on the same process: `ScoredPostsService` and `ForYouFeedService`. Both convert `xai_home_mixer_proto::ScoredPostsQuery` into the Rust `ScoredPostsQuery` in `home-mixer/models/query.rs` via `QueryBuilder`, then either short-circuit empty for `params::TEST_USER_IDS` or run a candidate pipeline. `xai_home_mixer_proto`, `crate::params`, Gizmoduck, and the URT serializer live in unpublished crates; this checkout shows the request and response mapping, not a runnable server.
```mermaid
sequenceDiagram
participant Client
participant QB as QueryBuilder
participant SPS as ScoredPostsServer
participant FYS as ForYouFeedServer
participant Phoenix as PhoenixCandidatePipeline
participant ForYou as ForYouCandidatePipeline
Client->>QB: proto ScoredPostsQuery
alt viewer_id == 0
QB-->>Client: INVALID_ARGUMENT viewer_id must be specified
else valid viewer
QB->>QB: Gizmoduck ViewerData (200 ms) + feature switches
QB-->>SPS: Rust ScoredPostsQuery
QB-->>FYS: Rust ScoredPostsQuery
alt user_id in TEST_USER_IDS
SPS-->>Client: ScoredPostsResponse scored_posts = []
FYS-->>Client: ForYouFeedResponse items = [] / empty URT
else live request
SPS->>Phoenix: execute(query)
Phoenix-->>SPS: selected PostCandidates
SPS-->>Client: ScoredPostsResponse
FYS->>ForYou: execute(query)
ForYou->>SPS: ScoredPostsSource.run_pipeline
ForYou-->>FYS: Vec FeedItem
FYS-->>Client: ForYouFeedResponse or ForYouFeedUrtResponse
end
end
```
## gRPC process
`home-mixer/main.rs` starts `HomeMixerServer` as service name `home-mixer` with mTLS from the environment, gRPC reflection from `xai_home_mixer_proto::FILE_DESCRIPTOR_SET`, feature switches at `params::FS_PATH`, and dark-traffic reject layers.
| Flag | Default | Role |
|------|---------|------|
| `--grpc-port` | `50051` | gRPC listen port |
| `--metrics-port` | `9090` | Metrics / profiling HTTP |
| `--shard-coordinate` | `-1` | Shard ordinal; `< 0` means no `ShardCoordinate` |
| `--shard-total-size` | `500` | Used only when a shard is set |
| `--datacenter` | `atla` | Feature-switch recipient + client wiring |
| `--otel-endpoint` | empty | Optional OTel export |
Both services accept and send Gzip and Zstd, and cap decode/encode size at `params::MAX_GRPC_MESSAGE_SIZE`. `HomeMixerServer::build` constructs one shared `QueryBuilder` (feature switches, decider, datacenter, prod Gizmoduck), one `PhoenixCandidatePipeline`, a `ScoredPostsServer` wrapping that pipeline, then a `ForYouCandidatePipeline` that calls back into the same `ScoredPostsServer`.
| Service | RPC | Request | Response |
|---------|-----|---------|----------|
| `ScoredPostsService` | `GetScoredPosts` | `ScoredPostsQuery` | `ScoredPostsResponse` |
| `ScoredPostsService` | `GetDebugScoredPosts` | `DebugScoredPostsQuery` | `DebugScoredPostsResponse` |
| `ForYouFeedService` | `GetForYouFeed` | `ForYouFeedQuery` | `ForYouFeedResponse` |
| `ForYouFeedService` | `GetForYouFeedUrt` | `ForYouFeedQuery` | `ForYouFeedUrtResponse` |
`GetForYouFeed` requires `ForYouFeedQuery.query`. `GetDebugScoredPosts` unwraps a missing nested query to proto default, which then fails `viewer_id` validation. Every successful RPC injects B3 trace headers on the response.
## Proto request fields `QueryBuilder` reads
The proto crate is unpublished. The following fields are the ones `QueryBuilder::build` and the For You URT handler actually consume.
Mapped to `ScoredPostsQuery.user_id`. `0` is rejected as `INVALID_ARGUMENT` with `viewer_id must be specified`.
Copied onto the query and used as the feature-switch recipient `client_app_id`.
Copied onto the query and the feature-switch recipient.
Copied onto the query and the feature-switch recipient.
Viewer-seen tweet IDs. Copied as-is.
Initial served IDs. For You later overwrites this from served-history hydration when that hydrator is enabled.
Forced `true` if Gizmoduck returns `allow_for_you_recommendations == Some(false)`.
Copied into `ScoredPostsQuery`. URT cursor decode can overwrite it.
Copied into `ScoredPostsQuery.is_preview`.
Copied into `ScoredPostsQuery.exclude_videos`.
Non-empty marks a topics request (`is_topic_request()`). More than 6 IDs is a bulk topic request.
Snoozed/excluded topics. Non-empty is `has_excluded_topics()`.
Passed through `resolve_request_id`. A separate `prediction_id` is always newly generated.
`0` becomes `None` via `non_zero`.
Missing status becomes proto default. Fields copied: `ip_address`, `user_agent`, `time_zone`, `device_network_type`, `client_version`, `device_id`, `mobile_device_id`, `mobile_device_ad_id`. Time zone and network type go through `timezone_string_to_enum` / `network_type_string_to_enum`.
For You RPCs also read these proto fields **after** `QueryBuilder::build`:
| Proto field | Applied on |
|-------------|------------|
| `request_context` | `query.request_context` |
| `is_polling` | `query.is_polling` |
| `cursor` | Decoded with `cursor_utils::decode_ordered_cursor`; sets `cursor`, `is_bottom_request`, `is_top_request` |
`GetDebugScoredPosts` also applies `feature_switch_overrides: map` after recipient matching.
```json
{
"viewer_id": 123456789,
"client_app_id": 3033300,
"country_code": "US",
"language_code": "en",
"seen_ids": [1, 2],
"served_ids": [],
"in_network_only": false,
"is_bottom_request": false,
"is_preview": false,
"exclude_videos": false,
"topic_ids": [],
"excluded_topic_ids": [],
"request_id": 0,
"push_to_home_post_id": 0,
"device_status": {
"ip_address": "203.0.113.10",
"user_agent": "XAndroid",
"time_zone": "America/New_York",
"device_network_type": "WIFI",
"client_version": "10.0",
"device_id": "dev",
"mobile_device_id": "",
"mobile_device_ad_id": ""
}
}
```
That JSON is reconstructed from `QueryBuilder` usage, not a checked-in `.proto`.
## QueryBuilder construction
`QueryBuilder::build` is the only path from proto to the live Rust query.
`viewer_id == 0` returns `Status::invalid_argument("viewer_id must be specified")` before any hydration.
If `viewer_id` is in `params::TRACE_USER_IDS`, B3 sampling is forced. The ID list is not in this checkout.
`gizmoduck_client.get_viewer_data(viewer_id)` is awaited with a **200 ms** timeout. Error or timeout becomes `ViewerData::default()`.
`in_network_only = proto.in_network_only || allow_for_you_recommendations == Some(false)`.
Recipient fields: `user_id`, `country`, `language`, `client_app_id`, custom `datacenter`, `account_age_days` from `days_since_creation(viewer_id)`, `has_phone_number`, and `user_roles` when non-empty. Debug overrides call `override_fs` per key.
`prediction_id = generate_request_id()`, `request_id = resolve_request_id(proto.request_id)`, `is_shadow_traffic = is_sampled(request_id, 0.5)`, then `ScoredPostsQuery::new(...)`.
Fields taken from `ViewerData` when the Gizmoduck call succeeds: `roles`, `muted_keywords`, `follower_count`, `subscription_level`, `age_in_years`, `has_phone_number`, `allow_for_you_recommendations`. The client struct itself is unpublished; those names are the ones `QueryBuilder` reads.
`QueryBuilder::mock()` builds empty feature switches, an empty decider store, datacenter `"mock"`, and `MockGizmoduckClient`.
## Live `ScoredPostsQuery` fields
The gRPC servers, Phoenix pipeline, and For You pipeline all use `home-mixer/models/query.rs`. `home-mixer/candidate_pipeline/query.rs` defines a narrower leftover type (`user_id: i64`, string `request_id`, no sequences) that the servers do not use.
### Set at construction
| Field | Construction default / source |
|-------|-------------------------------|
| `user_id` | proto `viewer_id` |
| `client_app_id`, `country_code`, `language_code` | proto |
| `seen_ids`, `served_ids` | proto |
| `in_network_only` | proto **or** Gizmoduck For You opt-out |
| `is_bottom_request` | proto; URT cursor may overwrite |
| `is_top_request` | `false`; URT `CursorType::TOP` sets `true` |
| `params` | feature-switch match (+ debug overrides) |
| `decider` | `Some(decider.with_recipient(viewer_id))` |
| `user_roles` | Gizmoduck roles, else empty |
| `user_features.muted_keywords` | Gizmoduck |
| `user_features.follower_count` | Gizmoduck |
| `topic_ids`, `excluded_topic_ids`, `exclude_videos` | proto |
| `request_id`, `prediction_id` | resolved / generated |
| `request_time_ms` | `current_time_ms()` |
| `ip_address`, `user_agent`, `time_zone`, `device_network_type`, `client_version`, `device_id`, `mobile_device_id`, `mobile_device_ad_id` | `device_status` |
| `subscription_level` | Gizmoduck |
| `is_shadow_traffic` | 50% sample on `request_id` |
| `is_preview` | proto |
| `user_age_in_years` | Gizmoduck `age_in_years` |
| `push_to_home_post_id` | proto, `0` → `None` |
All other struct fields start empty / `None` / `false` in `ScoredPostsQuery::new`.
Helpers on the live type:
- `is_topic_request()` — `!topic_ids.is_empty()`
- `is_bulk_topic_request()` — `topic_ids.len() > 6`
- `has_excluded_topics()` — `!excluded_topic_ids.is_empty()`
- `has_new_user_topic_ids()` — `!new_user_topic_ids.is_empty()`
`PipelineQuery` exposes `params` and `decider`. `GetTwitterContextViewer` builds a viewer with `user_id`, `client_application_id`, `request_country_code`, and `request_language_code`.
### Filled by query hydrators
Phoenix (`PhoenixCandidatePipeline`) hydrates scoring/retrieval sequences, social-graph IDs, cached posts, mutual-follow minhash, demographics, Grok topics, starter packs, impression bloom filters, IP location, and inferred gender. For You (`ForYouCandidatePipeline`) hydrates served history and non-polling timestamps, then calls Phoenix through `ScoredPostsSource`.
| Hydrator | Writes |
|----------|--------|
| `ScoringSequenceQueryHydrator` | `scoring_sequence`, `columnar_scoring_sequence` |
| `RetrievalSequenceQueryHydrator` | `retrieval_sequence`, `columnar_retrieval_sequence` |
| `BlockedUserIdsQueryHydrator` | `user_features.blocked_user_ids` |
| `MutedUserIdsQueryHydrator` | `user_features.muted_user_ids` |
| `FollowedUserIdsQueryHydrator` | `user_features.followed_user_ids` |
| `SubscribedUserIdsQueryHydrator` | `user_features.subscribed_user_ids` |
| `CachedPostsQueryHydrator` | `cached_posts`, `has_cached_posts` |
| `MutualFollowQueryHydrator` | `viewer_minhash` |
| `UserDemographicsQueryHydrator` | `user_demographics` |
| `FollowedGrokTopicsQueryHydrator` | `followed_grok_topics` (`[bool; 32]`), optionally `new_user_topic_ids` |
| `FollowedStarterPacksQueryHydrator` | `followed_starter_packs` (`[bool; 20]`) |
| `InferredGrokTopicsQueryHydrator` | `inferred_grok_topics` (`[bool; 32]`) |
| `ImpressionBloomFilterQueryHydrator` | `bloom_filter_entries` |
| `IpQueryHydrator` | `ip_location` |
| `UserInferredGenderQueryHydrator` | `user_inferred_gender`, `user_inferred_gender_score` |
| `ServedHistoryQueryHydrator` | `served_history`, `served_ids`, `who_to_follow_eligible` |
| `PastRequestTimestampsQueryHydrator` | `non_polling_timestamps` |
`UserFeatures` also stores `muted_keywords` and `follower_count` from Gizmoduck at construction. The struct is a Strato `MValCodec` with camelCase serde names.
`ImpressedPostsQueryHydrator` exists and writes `impressed_post_ids`, but Phoenix constructs it as `_impressed_posts_hydrator` and does not put it in the hydrator list. `impressed_post_ids` therefore stays empty on the live path.
`UserFeaturesQueryHydrator` and `UserActionSeqQueryHydrator` target the leftover `candidate_pipeline::query::ScoredPostsQuery` and are not registered on either live pipeline.
## TEST_USER_IDS empty responses
Both servers check `params::TEST_USER_IDS.contains(&query.user_id)` **after** query construction and **before** pipeline execute. The ID list is unpublished.
| Entry point | Short-circuit result |
|-------------|----------------------|
| `ScoredPostsServer::run_pipeline` | `PipelineOutput { scored_posts: [], pipeline_result: PipelineResult::empty() }` |
| `ForYouFeedServer::get_for_you_feed` | `ForYouFeedOutput { items: [] }` |
| `get_for_you_feed_urt` | Calls `get_for_you_feed`, so URT is serialized from an empty item list |
`PipelineResult::empty()` uses `ScoredPostsQuery::default()` as the stored query (not the incoming query) and empty retrieved/filtered/selected vectors. Debug JSON for a test user therefore does not include the real request. For You URT still runs `urt::make_urt_timeline` on the empty list.
`TRACE_USER_IDS` only forces B3 sampling. It does not empty the feed. Empty feeds for listed test users are expected, not a scoring failure.
## Scored Posts surface classification
`log_request_info` labels each Scored Posts request for logs and `ScoredPostsServer.product_surface`:
| Condition | Surface |
|-----------|---------|
| `in_network_only` | `ranked_following` |
| else non-empty `topic_ids` | `topics` |
| else `has_excluded_topics()` | `for_you_with_snoozed_topics` |
| else | `for_you` |
Single-topic requests also increment `ScoredPosts.topic` with `type=request` and `type=empty` when selection is empty.
## Response mapping
### `ScoredPost`
`candidates_to_scored_posts` maps each selected `PostCandidate` to `xai_home_mixer_proto::ScoredPost`. Missing optionals become `0`, `false`, empty bytes, or proto default.
| `ScoredPost` field | Source |
|--------------------|--------|
| `tweet_id` | `candidate.tweet_id` |
| `author_id` | `candidate.author_id` |
| `retweeted_tweet_id` | `retweeted_tweet_id.unwrap_or(0)` |
| `retweeted_user_id` | `retweeted_user_id.unwrap_or(0)` |
| `in_reply_to_tweet_id` | `in_reply_to_tweet_id.unwrap_or(0)` |
| `score` | `candidate.score.unwrap_or(0.0) as f32` — **not** `weighted_score` |
| `in_network` | `in_network.unwrap_or(false)` |
| `served_type` | `served_type as i32`, else `0` |
| `last_scored_timestamp_ms` | `last_scored_at_ms.unwrap_or(0)` |
| `prediction_request_id` | `prediction_request_id.unwrap_or(0)` |
| `ancestors` | cloned |
| `screen_names` | `get_screen_names()` — author and retweeted-author screen names only |
| `visibility_reason` | `FilteredReason` converted into proto, if present |
| `tweet_type_metrics` | bytes, default empty |
| `following_replied_user_ids` | cloned |
| `brand_safety_verdict` | `BrandSafetyVerdict` as `i32`; missing → `MediumRisk` (`3`) |
| `safety_label_types` | mapped `SafetyLabelType` values; unmapped types dropped |
| `tweet_text` | cloned |
`BrandSafetyVerdict` values: `Unspecified = 0`, `Safe = 1`, `LowRisk = 2`, `MediumRisk = 3`.
Mapped safety labels include NSFW / NSFA / gore / PDNA / Grok SFA-NSFA variants listed in `safety_label_to_proto`. Anything else is omitted.
`PostCandidate` fields that stay off the proto include `phoenix_scores`, `weighted_score`, quote IDs, video durations, engagement counts, topic IDs, `has_media`, `language_code`, and `mutual_follow_jaccard`.
```json
{
"scored_posts": [
{
"tweet_id": 1001,
"author_id": 42,
"retweeted_tweet_id": 0,
"retweeted_user_id": 0,
"in_reply_to_tweet_id": 0,
"score": 1.25,
"in_network": true,
"served_type": 1,
"last_scored_timestamp_ms": 0,
"prediction_request_id": 0,
"ancestors": [],
"screen_names": { "42": "example" },
"tweet_type_metrics": "",
"following_replied_user_ids": [],
"brand_safety_verdict": 3,
"safety_label_types": [],
"tweet_text": ""
}
]
}
```
`GetDebugScoredPosts` returns the same `scored_posts` plus `debug_json`: serialized `query`, `retrieved_candidates`, `filtered_candidates`, `selected_candidates`, and count stats. Serialization failure becomes `{"error": "Failed to serialize debug info: ..."}`.
### `ForYouFeed` items
`ScoredPostsSource` wraps each `ScoredPost` as:
```rust
FeedItem {
position: 0,
item: Some(feed_item::Item::Post(post)),
}
```
For You sources add other `feed_item::Item` variants. `BlenderSelector` partitions them, blends ads, then inserts prompts, who-to-follow, and push-to-home:
| Variant | Source | Placement |
|---------|--------|-----------|
| `Post(ScoredPost)` | `ScoredPostsSource` → Phoenix | Organic list, then ads blender |
| `Ad(AdIndexInfo)` | `AdsSource` | `SafeGapAdsBlender` or `PartitionOrganicAdsBlender` |
| `Prompt` | `PromptsSource` | Inserted at index `i` with `position = PROMPTS_POSITION` |
| `WhoToFollow` | `WhoToFollowSource` | First module only; insert index `WHO_TO_FOLLOW_POSITION - 1` |
| `PushToHome` | `PushToHomeSource` | Pinned at index `0` |
`WhoToFollowSource` enables only when `EnableWhoToFollowModule` is on **and** `who_to_follow_eligible` is true (served-history fatigue). `GetForYouFeed` returns `ForYouFeedResponse { items }` as the blender's selected `FeedItem`s.
### URT
`GetForYouFeedUrt` reuses `get_for_you_feed`, then:
1. `urt::make_urt_timeline(items, cursor, request_context, client_app_id, viewer_id, language_code, country_code)`
2. `xai_urt_thrift::serialize_binary` → `ForYouFeedUrtResponse.urt`
Empty `country_code` is passed as `None`. Cursor decode failure logs a warning and ignores the cursor (`is_bottom_request` / `is_top_request` stay at QueryBuilder values). Serialize failure is `Status::internal("failed to serialize URT: …")`. The URT builder itself is unpublished (`crate::util::urt`).
## Errors and empty-feed cases
| Symptom | Cause |
|---------|--------|
| `INVALID_ARGUMENT: viewer_id must be specified` | proto `viewer_id == 0`, including debug with a missing nested query |
| `INVALID_ARGUMENT: query must be specified` | `ForYouFeedQuery.query` absent |
| Empty `scored_posts` / `items` / URT | `user_id` in `TEST_USER_IDS`, or a live pipeline that selected nothing |
| Missing roles / muted keywords / age | Gizmoduck timeout or error (`ViewerData::default()`) |
| Forced `in_network_only` | `allow_for_you_recommendations == Some(false)` |
| URT cursor ignored | `decode_ordered_cursor` error |
| `INTERNAL: failed to serialize URT` | Thrift serialize failure after a successful pipeline |
| `impressed_post_ids` always empty | hydrator not registered |
This checkout cannot start `HomeMixerServer`: `params`, proto stubs, Gizmoduck, and URT crates are not published here.
## Next
CLI flags, viewer_id validation, and For You versus Scored Posts entry points.
`CandidatePipeline.execute` stages and how `ForYouCandidatePipeline` wraps Phoenix.
Query hydrators that fill `ScoredPostsQuery` and candidate hydrators that must preserve order.
`viewer_id must be specified`, `TEST_USER_IDS` empty feeds, and other Home Mixer failures.
---
## 19. Candidate sources
> PhoenixCandidatePipeline and ForYouCandidatePipeline sources, enable predicates, cluster resolution, and served_type assignment.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/19-candidate-sources.md
- Generated: 2026-08-13T19:48:09.013Z
### Source Files
- `home-mixer/sources/mod.rs`
- `home-mixer/sources/thunder_source.rs`
- `home-mixer/sources/phoenix_source.rs`
- `home-mixer/sources/phoenix_moe_source.rs`
- `home-mixer/sources/phoenix_topics_source.rs`
- `home-mixer/sources/ads_source.rs`
- `home-mixer/sources/cached_posts_source.rs`
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`
---
title: "Candidate sources"
description: "PhoenixCandidatePipeline and ForYouCandidatePipeline sources, enable predicates, cluster resolution, and served_type assignment."
---
`PhoenixCandidatePipeline` and `ForYouCandidatePipeline` each own a `Source` list. After query hydration, `CandidatePipeline::fetch_candidates` keeps sources whose `enable(&query)` is true, runs those `source` methods in parallel with `join_all`, and concatenates successful `Ok` vectors. `Err` results are logged by `Source::run` and dropped. Phoenix sources emit `PostCandidate`. For You sources emit `FeedItem`. Organic scored posts enter For You only through `ScoredPostsSource`, which executes `PhoenixCandidatePipeline` via `ScoredPostsServer::run_pipeline`.
Feature-switch identifiers such as `PhoenixMaxResults` live in unpublished `crate::params`. This checkout shows the names and how they are read, not the default values.
```mermaid
flowchart TB
subgraph FY["ForYouCandidatePipeline — FeedItem"]
SPS[ScoredPostsSource]
ADS[AdsSource]
WTF[WhoToFollowSource]
PRM[PromptsSource]
PTH[PushToHomeSource]
BS[BlenderSelector]
SPS --> BS
ADS --> BS
WTF --> BS
PRM --> BS
PTH --> BS
end
subgraph PHX["PhoenixCandidatePipeline — PostCandidate"]
TH[ThunderSource]
TM[TweetMixerSource]
PX[PhoenixSource]
PT[PhoenixTopicsSource]
MOE[PhoenixMOESource]
CP[CachedPostsSource]
TK[TopKScoreSelector]
TH --> TK
TM --> TK
PX --> TK
PT --> TK
MOE --> TK
CP --> TK
end
SPS -->|run_pipeline| PHX
QH[CachedPostsQueryHydrator / FollowedGrokTopicsQueryHydrator / RetrievalSequenceQueryHydrator] --> PHX
SH[ServedHistoryQueryHydrator] --> FY
```
## Source contract
`xai_candidate_pipeline::source::Source`:
| Method | Default | Role |
|---|---|---|
| `enable(&Q) -> bool` | `true` | Gate before `run` |
| `source(&Q) -> Result, String>` | required | Fetch candidates |
| `run(&Q)` | wraps `source` | Logs count or error, then returns the result |
| `name()` | short type name | Tracing / stats |
`fetch_candidates` records enabled versus disabled names, then flattens only `Ok` vectors. One failed source does not fail the request.
:::files
home-mixer/sources/
├── thunder_source.rs
├── tweet_mixer_source.rs
├── phoenix_source.rs
├── phoenix_topics_source.rs
├── phoenix_moe_source.rs
├── cached_posts_source.rs
├── scored_posts_source.rs
├── ads_source.rs
├── who_to_follow_source.rs
├── prompts_source.rs
└── push_to_home_source.rs
:::
## PhoenixCandidatePipeline sources
`build_with_clients` registers sources in this order:
1. `ThunderSource`
2. `TweetMixerSource`
3. `PhoenixSource`
4. `PhoenixTopicsSource`
5. `PhoenixMOESource`
6. `CachedPostsSource`
Order does not serialize execution. Enabled sources run concurrently; results are appended in that registration order.
| Source | Candidate type | Backend | `served_type` |
|---|---|---|---|
| `ThunderSource` | `PostCandidate` | Thunder `GetInNetworkPosts` | `ForYouInNetwork` or `RankedFollowing` |
| `TweetMixerSource` | `PostCandidate` | Tweet Mixer `HOME_RECOMMENDED_TWEETS` | `ForYouTweetMixer` |
| `PhoenixSource` | `PostCandidate` | Phoenix retrieval | `ForYouPhoenixRetrieval` |
| `PhoenixTopicsSource` | `PostCandidate` | Phoenix retrieval with topic IDs | `ForYouPhoenixRetrieval` |
| `PhoenixMOESource` | `PostCandidate` | Phoenix retrieval | `ForYouPhoenixRetrievalMoe` |
| `CachedPostsSource` | `PostCandidate` | `query.cached_posts` | Preserved from cache |
Prod retrieval client construction pins `PhoenixRetrievalCluster::Experiment1Fou` and `PhoenixRetrievalCluster::Experiment1Lap7`. Cluster choice per request is still resolved from params and deciders.
## Enable predicates
Shared query flags used by almost every Phoenix-side gate:
True when the proto sets `in_network_only` or viewer `allow_for_you_recommendations == Some(false)`. Disables every out-of-network source.
True when `CachedPostsQueryHydrator` loads at least 500 cached `PostCandidate`s. Disables Thunder, Tweet Mixer, and all Phoenix retrieval sources; enables `CachedPostsSource`.
`!topic_ids.is_empty()`.
`topic_ids.len() > 6`. Bulk topic requests take the general Phoenix path, not `PhoenixTopicsSource`.
`!new_user_topic_ids.is_empty()`. Populated by `FollowedGrokTopicsQueryHydrator` for new users when `EnableNewUserTopicRetrieval` or `EnableNewUserTopicFiltering` is on, the request is not a topic request, and `in_network_only` is false.
| Source | `enable` |
|---|---|
| `ThunderSource` | `!has_cached_posts` |
| `TweetMixerSource` | `!in_network_only && !has_cached_posts` |
| `PhoenixSource` | `(!is_topic_request \|\| is_bulk_topic_request) && (!EnableNewUserTopicRetrieval \|\| !has_new_user_topic_ids) && !in_network_only && !has_cached_posts` |
| `PhoenixTopicsSource` | `((is_topic_request && !is_bulk_topic_request) \|\| (EnableNewUserTopicRetrieval && has_new_user_topic_ids)) && !in_network_only && !has_cached_posts` |
| `PhoenixMOESource` | `EnablePhoenixMOESource && (!is_topic_request \|\| is_bulk_topic_request) && !in_network_only && !has_cached_posts` |
| `CachedPostsSource` | `has_cached_posts` |
`PhoenixSource` and `PhoenixTopicsSource` are mutually exclusive for a given request. `PhoenixMOESource` can run beside `PhoenixSource` on non-topic and bulk-topic requests. When `has_cached_posts` is true, only `CachedPostsSource` runs on this pipeline.
`ThunderSource` stays enabled under `in_network_only`. Out-of-network sources do not. Thunder stamps `RankedFollowing` in that mode instead of `ForYouInNetwork`.
## Cluster resolution
### PhoenixSource
`PhoenixSource::resolve_cluster`:
1. Parse `PhoenixRetrievalInferenceClusterId` to `PhoenixRetrievalCluster`.
2. If `PhoenixRetrievalNewUserHistoryThreshold > 0` and `retrieval_sequence.metadata.length` (else `0`) is below that threshold, return `PhoenixRetrievalCluster::parse(PhoenixRetrievalNewUserInferenceClusterId)`.
3. Otherwise, if a `Decider` is present:
- configured `Experiment1Lap7` and `enable_phoenix_retrieval_lap7_to_fou` → `Experiment1Fou`
- configured `Experiment1Fou` and `enable_phoenix_retrieval_fou_to_lap7` → `Experiment1Lap7`
4. Otherwise use the configured cluster.
This source is the only Phoenix retrieval source that also builds `client_context` and `user_context` via unpublished `crate::util::phoenix_request`.
### PhoenixTopicsSource
Cluster is `PhoenixRetrievalCluster::parse(PhoenixRetrievalTopicInferenceClusterId)`. No new-user threshold and no Lap7/Fou decider swap.
Topic IDs sent to retrieval:
- Topic request: `query.topic_ids`
- New-user topic retrieval: `query.new_user_topic_ids`
Each ID is passed through `TopicIdExpansion::resolve_first`, which currently returns the ID unchanged. `TopicFilteringId` is parsed to `TopicFilteringExperiment`; `TopicFilteringOverrides` (`topic_id=ExperimentId` comma list) can override the first matching topic. The resolved experiment is sent as proto mode:
| `TopicFilteringExperiment` | proto mode |
|---|---|
| `Unfiltered` (also unknown strings) | `0` |
| `CuratedV0` | `1` |
| `CuratedV0V1` | `2` |
| `PostBased90Pct` | `3` |
| `PostBased75Pct` | `4` |
| `PostBased50Pct` | `5` |
### PhoenixMOESource
Cluster is `PhoenixRetrievalCluster::parse(PhoenixRetrievalMOEInferenceClusterId)`. Max results is `PhoenixMOEMaxResults`, not `PhoenixMaxResults`. Topic IDs and filter mode are empty / `None`. Client and user context are `None`.
### ThunderSource
```text
configured = ThunderCluster::parse(ThunderClusterId)
cluster = ThunderCluster::resolve(configured, query.decider)
channel = thunder_client.get_random_channel(cluster)
```
`ThunderCluster` lives in the unpublished candidate-pipeline client crate. Missing channel returns `"ThunderSource: no available channel"`.
## Retrieval requests
All three Phoenix sources require `query.retrieval_sequence`. That field is filled by `RetrievalSequenceQueryHydrator` from the user-action aggregation service. Absence fails the source with `": missing retrieval_sequence"`.
| Field | `PhoenixSource` | `PhoenixTopicsSource` | `PhoenixMOESource` |
|---|---|---|---|
| Cluster | `resolve_cluster` | `PhoenixRetrievalTopicInferenceClusterId` | `PhoenixRetrievalMOEInferenceClusterId` |
| Max results | `PhoenixMaxResults` | `PhoenixMaxResults` | `PhoenixMOEMaxResults` |
| Topic entity IDs | `[]` | expanded effective topic IDs | `[]` |
| Topic filter mode | `None` | `Some(mode)` | `None` |
| Client / user context | built | `None` | `None` |
| Sequence | `retrieval_sequence` + `columnar_retrieval_sequence` | same | same |
Candidates are taken from `response.top_k_candidates[*].candidates[*].candidate`. `in_reply_to_tweet_id` is always `Some(...)` from the retrieval tweet info. `retweeted_tweet_id` is set only when the retrieval value is non-zero.
`ThunderSource` builds `GetInNetworkPostsRequest`:
Viewer ID.
`query.user_features.followed_user_ids`.
`ThunderMaxResults`.
`query.seen_ids`.
`ThunderAlgorithm`.
Hard-coded `false`.
Thunder also writes `query.in_network_replies` from posts that have `in_reply_to_post_id`. Each candidate gets `ancestors = [in_reply_to_tweet_id, conversation_id?]` when those IDs differ.
`TweetMixerSource` requests `Product::HOME_RECOMMENDED_TWEETS` with `TweetMixerMaxResults` and `seen_ids` as `excluded_tweet_ids`. It drops tweets whose snowflake age is greater than `Duration::from_secs(MAX_POST_AGE)` (same constant `AgeFilter` uses). `retweeted_tweet_id` is always `None` here.
## Cached posts path
`CachedPostsQueryHydrator` (gated by `EnableCachedPosts`) GETs Redis key `cached_posts_key(user_id, topic_ids, in_network_only, exclude_videos)` with a 300 ms timeout, then zstd-decompresses JSON into `Vec`.
| Hydrator outcome | `has_cached_posts` | Sources that run |
|---|---|---|
| Empty payload, timeout, or Redis error | `false` | Live Thunder / Tweet Mixer / Phoenix set |
| Fewer than 500 posts | `false` (posts still stored on the query) | Live set |
| `cached_posts.len() >= 500` | `true` | `CachedPostsSource` only |
`CachedPostsSource` returns `query.cached_posts.clone()`. It does not rewrite `served_type`.
## served_type assignment
Sources stamp `PostCandidate.served_type` at fetch time. `ScoredPostsServer` copies `served_type as i32` onto `ScoredPost` (`0` if unset). Stats treat `ForYouPhoenixRetrieval` as `PhoenixRetrievalTweets` and `ForYouPhoenixRetrievalMoe` as `PhoenixRetrievalMoeTweets`.
| Origin | `ServedType` |
|---|---|
| Thunder, `!in_network_only` | `ForYouInNetwork` |
| Thunder, `in_network_only` | `RankedFollowing` |
| `PhoenixSource` | `ForYouPhoenixRetrieval` |
| `PhoenixTopicsSource` | `ForYouPhoenixRetrieval` |
| `PhoenixMOESource` | `ForYouPhoenixRetrievalMoe` |
| `TweetMixerSource` | `ForYouTweetMixer` |
| `CachedPostsSource` | Unchanged from Redis JSON |
| `PushToHomeSource` | `ForYouPushToHome` |
`PhoenixSource` and `PhoenixTopicsSource` share `ForYouPhoenixRetrieval`. Downstream stats keyed only on that enum cannot distinguish general retrieval from topic retrieval.
## ForYouCandidatePipeline sources
`ForYouCandidatePipeline::build` registers:
1. `ScoredPostsSource`
2. `AdsSource`
3. `WhoToFollowSource`
4. `PromptsSource`
5. `PushToHomeSource`
This pipeline has no hydrators, filters, or scorers. `BlenderSelector` merges the `FeedItem` streams.
| Source | `enable` | `FeedItem` variant |
|---|---|---|
| `ScoredPostsSource` | default `true` | `Item::Post(ScoredPost)` |
| `AdsSource` | `EnableAdsSource && !is_preview` | `Item::Ad` |
| `WhoToFollowSource` | `EnableWhoToFollowModule && who_to_follow_eligible` | `Item::WhoToFollow` |
| `PromptsSource` | `EnablePrompts` | `Item::Prompt` |
| `PushToHomeSource` | `push_to_home_post_id.is_some()` | `Item::PushToHome` |
### ScoredPostsSource
Calls `ScoredPostsServer::run_pipeline(query.clone())`. If `params::TEST_USER_IDS` contains `query.user_id`, that path returns an empty `PipelineOutput` without executing Phoenix sources. Each selected `ScoredPost` becomes a `FeedItem` at `position: 0`; the blender assigns display positions later.
### AdsSource
Builds `AdIndexRequest` with `ProductSurface::HomeTimelineRanking` and a `ClientContext` from `user_id`, `client_app_id`, country / language, IP, user agent, roles, and device IDs. Preview requests skip ads.
### WhoToFollowSource
`who_to_follow_eligible` starts `false` and is set by `ServedHistoryQueryHydrator` when `EnableUrtMigrationComponents` is on. Eligibility is fatigue-based: no `EntityIdType::WHO_TO_FOLLOW` history, or last serve is at least `WhoToFollowFatigueHours` old.
The request is `Product::HomeWhoToFollow` with up to 200 excluded user IDs taken from served-history WTF entries. The source keeps at most 3 recommendations and returns a single module item, or `[]` if the mixer returns none.
If `EnableUrtMigrationComponents` is off, `who_to_follow_eligible` stays `false` and `WhoToFollowSource` does not run even when `EnableWhoToFollowModule` is on.
### PromptsSource
Requests `DisplayLocation::HOME_TIMELINE` for `INLINE_PROMPT`, `FULL_COVER`, `HALF_COVER`, and `RELEVANCE_PROMPT`. Each injection is Thrift-serialized into `Prompt.injection`. Serialization failure fails that source.
### PushToHomeSource
Requires `query.push_to_home_post_id`. TES miss or `Ok(None)` returns `[]`. TES `Err` fails the source. Root tweets (no `in_reply_to_tweet_id`) request up to 3 reply-mixer author IDs excluding the original author; reply-mixer errors log and leave `facepile_user_ids` empty. Reply tweets skip the facepile.
## Feature switches and request fields
Phoenix / Thunder / Tweet Mixer:
Default cluster string for `PhoenixSource`.
Override cluster when history length is below threshold.
`0` disables the new-user cluster override.
Cluster for `PhoenixTopicsSource`.
Cluster for `PhoenixMOESource`.
Max results for `PhoenixSource` and `PhoenixTopicsSource`.
Max results for `PhoenixMOESource`.
Master gate for `PhoenixMOESource`.
Routes new-user followed topics to `PhoenixTopicsSource` and disables `PhoenixSource`.
Thunder cluster string, algorithm, and result cap.
Tweet Mixer result cap.
Enables `CachedPostsQueryHydrator`.
For You:
Ads intake, still blocked when `is_preview`.
WTF intake, still requires `who_to_follow_eligible`.
Prompts intake.
Non-zero proto field after `HomeMixerServer` mapping. Enables `PushToHomeSource`.
## Errors and empty results
| Condition | Source result | Pipeline effect |
|---|---|---|
| Missing `retrieval_sequence` | `Err("… missing retrieval_sequence")` | That source dropped |
| Thunder has no channel | `Err("ThunderSource: no available channel")` | Thunder dropped |
| Phoenix / Tweet Mixer / ads / WTF / prompts RPC error | `Err(": …")` | That source dropped |
| Tweet Mixer tweet older than `MAX_POST_AGE` | omitted | Not a hard error |
| Cached Redis timeout / empty / decode failure | hydrator error or empty | `has_cached_posts` stays false |
| `TEST_USER_IDS` viewer | empty `PipelineOutput` | No Phoenix sources run |
| TES miss for push-to-home | `Ok([])` | No PTH item |
| TES error for push-to-home | `Err` | PTH dropped |
| Empty WTF recommendations | `Ok([])` | No module |
These sources are production Home Mixer code. They depend on unpublished clients (`PhoenixRetrievalClient`, `ThunderClient`, Tweet Mixer, Ad Index, prompts, WTF, TES, Reply Mixer). This checkout can inspect the predicates and stamps; it cannot execute the live sources. The local Phoenix path is `phoenix/run_pipeline.py` against extracted artifacts, which does not run these Rust sources.
## Next
How `execute` hydrates the query, fans out sources, then hydrates, filters, and scores.
Thunder versus Phoenix retrieval and how `in_network_only` changes sourcing.
RPC fields, PostStore timelines, and following-list fallback behind `ThunderSource`.
How `AdsSource` items are positioned after intake.
`Source` contract, length-match rules, and where a new source is registered.
Query hydrators that populate `has_cached_posts`, topic IDs, and sequences before sources run.
Query fields that drive enable predicates and `TEST_USER_IDS` empty feeds.
`missing retrieval_sequence`, Thunder `RESOURCE_EXHAUSTED`, and related source failures.
---
## 20. Filters and hydrators
> Pre-score and post-selection filters, query hydrators that fill ScoredPostsQuery, and candidate hydrators that must preserve order and length.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/20-filters-and-hydrators.md
- Generated: 2026-08-13T19:49:23.364Z
### Source Files
- `home-mixer/filters/mod.rs`
- `home-mixer/filters/age_filter.rs`
- `home-mixer/filters/vf_filter.rs`
- `home-mixer/query_hydrators/mod.rs`
- `home-mixer/query_hydrators/scoring_sequence_query_hydrator.rs`
- `home-mixer/candidate_hydrators/mod.rs`
- `home-mixer/candidate_hydrators/core_data_candidate_hydrator.rs`
- `candidate-pipeline/filter.rs`
---
title: "Filters and hydrators"
description: "Pre-score and post-selection filters, query hydrators that fill ScoredPostsQuery, and candidate hydrators that must preserve order and length."
---
`PhoenixCandidatePipeline` is the production Home Mixer surface that hydrates `ScoredPostsQuery`, enriches `PostCandidate`s, then partitions them with sequential filters. Query hydrators run in parallel and merge only the fields they own. Candidate hydrators also run in parallel and must return the same candidates in the same order — dropping a candidate belongs in a `Filter`, not a `Hydrator`. `ForYouCandidatePipeline` wraps scored posts plus ads and modules; it hydrates served history and request timestamps, then leaves candidate hydrators and filters empty.
These components live in `home-mixer/` and implement traits from `candidate-pipeline/`. Local Phoenix inference (`run_pipeline.py`) does not execute them. Production clients, `crate::params` feature switches, TES, VF, SocialGraph, and Strato are required at runtime.
`CandidatePipeline::execute` already has a dedicated request-lifecycle page. This page is the component contract and the Phoenix / For You registries.
## Contracts
Three traits own the behavior. `enable` defaults to `true`. Tracing spans are `query_hydrator`, `hydrator`, and `filter`, each recording `name`.
Fetch this hydrator's fields and return a new query. `update` copies only those fields onto the live query. A failed `hydrate` is logged (`Failed: …`) and skipped; the pipeline continues without that field set.
Return one result per input candidate, same order. Dropping candidates is not allowed. `Hydrator::run` replaces a length mismatch with `Err("Hydrator length_mismatch expected=N got=M")` for every slot and skips `update_all`. `update` copies only this hydrator's fields.
Partition into `kept` (next stage) and `removed` (excluded). Filters run sequentially; each filter sees only the previous `kept` set. `run` records `kept_count`, `removed_count`, and `filter_rate`, and increments `{Name}.run` under `requests.kept` / `requests.removed`.
`CachedHydrator` is a `Hydrator` adapter: per-candidate cache lookup, then `hydrate_from_client` for misses. A client length mismatch fails the **entire** candidate list. Cache hits and misses increment `{Name}.cache` under `requests.cache_hit` / `requests.cache_miss`.
```rust
// candidate-pipeline/hydrator.rs — length is a hard contract
async fn hydrate(&self, query: &Q, candidates: &[C]) -> Vec>;
// The returned vector must have the same candidates in the same order as the input.
// Dropping candidates in a hydrator is not allowed - use a filter stage instead.
```
## Execute order
`CandidatePipeline::execute` applies these stages in order. Query hydrators are two waves: `query_hydrators` then `dependent_query_hydrators`. Neither Phoenix nor For You overrides the second wave (default empty).
```mermaid
flowchart TB
subgraph queryLayer [Query layer — parallel]
QH[query_hydrators]
DQH[dependent_query_hydrators]
QH --> DQH
end
subgraph sourceLayer [Sources — parallel]
SRC[Thunder / Phoenix / TweetMixer / cache]
end
subgraph candLayer [Candidate layer — parallel]
CH[hydrators]
end
subgraph filterLayer [Pre-score filters — sequential]
F[filters]
end
subgraph scoreLayer [Score and select]
SC[scorers]
SEL[selector]
SC --> SEL
end
subgraph postLayer [Post-selection]
PSH[post_selection_hydrators — parallel]
PSF[post_selection_filters — sequential]
PSH --> PSF
end
queryLayer --> sourceLayer --> candLayer --> filterLayer --> scoreLayer --> postLayer
```
Removed candidates from both filter stages accumulate in `PipelineResult.filtered_candidates`. After post-selection filters, `execute` truncates to `result_size()` (`params::RESULT_SIZE` on Phoenix, `params::FOR_YOU_MAX_RESULT_SIZE` on For You).
## Query hydrators
Query hydrators fill `ScoredPostsQuery` before sources run. Each `hydrate` returns `ScoredPostsQuery { …owned fields, ..Default::default() }` and `update` copies only those fields so parallel merges do not clobber siblings.
### Phoenix registry
Wired in `PhoenixCandidatePipeline::build_with_clients`, in this order (execution is still parallel):
| Hydrator | Writes | Enable |
|---|---|---|
| `ScoringSequenceQueryHydrator` | `scoring_sequence`, `columnar_scoring_sequence` | always |
| `RetrievalSequenceQueryHydrator` | `retrieval_sequence`, `columnar_retrieval_sequence` | always |
| `BlockedUserIdsQueryHydrator` | `user_features.blocked_user_ids` | always |
| `MutedUserIdsQueryHydrator` | `user_features.muted_user_ids` | always |
| `FollowedUserIdsQueryHydrator` | `user_features.followed_user_ids` | always |
| `SubscribedUserIdsQueryHydrator` | `user_features.subscribed_user_ids` | always |
| `CachedPostsQueryHydrator` | `cached_posts`, `has_cached_posts` | `EnableCachedPosts` |
| `MutualFollowQueryHydrator` | `viewer_minhash` | `EnableMutualFollowJaccardHydration` |
| `UserDemographicsQueryHydrator` | `user_demographics` | `EnableContextFeatures` or `is_shadow_traffic` |
| `FollowedGrokTopicsQueryHydrator` | `followed_grok_topics`, maybe `new_user_topic_ids` | `EnableContextFeatures`, shadow traffic, `EnableNewUserTopicRetrieval`, or `EnableNewUserTopicFiltering` |
| `FollowedStarterPacksQueryHydrator` | `followed_starter_packs` | `EnableContextFeatures` or `is_shadow_traffic` |
| `InferredGrokTopicsQueryHydrator` | `inferred_grok_topics` | `EnableGrokTopicsHydration` |
| `ImpressionBloomFilterQueryHydrator` | `bloom_filter_entries` | always |
| `IpQueryHydrator` | `ip_location` | `EnableIpFeature` and non-empty `ip_address` |
| `UserInferredGenderQueryHydrator` | `user_inferred_gender`, `user_inferred_gender_score` | `EnableInferredGenderHydration` or `is_shadow_traffic` |
Social-graph hydrators call `SocialGraphClientOps` (`get_blocked_user_ids`, `get_muted_user_ids`, `get_followed_user_ids`, `get_subscribed_user_ids`) and write into `UserFeatures`. Downstream filters read those lists; they are not request-body fields.
### Sequence hydrators
Both sequence hydrators call `UserActionAggregationClient::fetch_aggregated_sequence` with `UAS_WINDOW_TIME_MS`, `UasSourceDataType` (default `Arrow`), `UseXdsForUas`, and optional realtime actions.
| | Scoring | Retrieval |
|---|---|---|
| Length param | `MaxSeqLengthScoring` | `MaxSeqLengthRetrieval` |
| Aggregation default | `DenseWithNotInterestedIn` (`PhoenixAggregationType`) | `Dense` (`PhoenixRetrievalAggregationType`) |
| `prediction_id` | `Some(query.prediction_id as i64)` | `None` |
| Response | `ResponseFormat::Arrow` | `ResponseFormat::Arrow` |
A missing `retrieval_sequence` later fails Phoenix retrieval. Sequence fields are `#[serde(skip)]` on `ScoredPostsQuery`.
### Cached posts
`CachedPostsQueryHydrator` GETs Redis key `cached_posts_key(user_id, topic_ids, in_network_only, exclude_videos)` with a 300 ms timeout, zstd-decompresses, and JSON-decodes `Vec`. `has_cached_posts` is `cached_posts.len() >= 500`. An empty payload leaves defaults. When `has_cached_posts` is true, most TES / SocialGraph candidate hydrators disable themselves so cached fields are not overwritten.
`ImpressedPostsQueryHydrator` is constructed in `build_with_clients` as `_impressed_posts_hydrator` and is **not** pushed into `query_hydrators`. `PreviouslySeenPostsBackupFilter` therefore only sees `impressed_post_ids` if some other path populated them. The hydrator itself writes `impressed_post_ids` from `ImpressedPostsClient::get(user_id)`.
### For You registry
`ForYouCandidatePipeline` registers two query hydrators, both gated on `EnableUrtMigrationComponents`:
| Hydrator | Writes |
|---|---|
| `ServedHistoryQueryHydrator` | `served_history`, `served_ids` (recent tweet / source IDs within `ExcludeServedTweetIdsDuration`, capped by `ExcludeServedTweetIdsNumber`), `who_to_follow_eligible` (`WhoToFollowFatigueHours`) |
| `PastRequestTimestampsQueryHydrator` | `non_polling_timestamps` |
### Present on disk, not on Phoenix
`UserActionSeqQueryHydrator` and `UserFeaturesQueryHydrator` implement `QueryHydrator` against the older `home-mixer/candidate_pipeline/query.rs` `ScoredPostsQuery` (`user_action_sequence`, `user_id: i64`). They are not in `query_hydrators/mod.rs` and are not registered on `PhoenixCandidatePipeline`. The live query type is `home-mixer/models/query.rs`.
## Candidate hydrators
Candidate hydrators run after sources and before pre-score filters. Most TES-backed hydrators implement `CachedHydrator` with a Moka cache and disable when `query.has_cached_posts`.
### Pre-score (`hydrators`)
| Hydrator | Writes | Enable |
|---|---|---|
| `InNetworkCandidateHydrator` | `in_network` | `!has_cached_posts` |
| `CoreDataCandidateHydrator` | `retweeted_user_id`, `retweeted_tweet_id`, `in_reply_to_tweet_id`, `tweet_text` | `!has_cached_posts` |
| `QuoteHydrator` | `quoted_tweet_id`, `quoted_user_id`, `quoted_author_blocks_viewer`, `quoted_video_duration_ms` | `!has_cached_posts` |
| `VideoDurationCandidateHydrator` | `min_video_duration_ms` | `!has_cached_posts` |
| `HasMediaHydrator` | `has_media` | (`EnableHasMediaHydration` or shadow traffic) and `!has_cached_posts` |
| `SubscriptionHydrator` | `subscription_author_id` | `!has_cached_posts` |
| `GizmoduckCandidateHydrator` | `author_followers_count`, `author_screen_name`, `retweeted_screen_name` | `!has_cached_posts` |
| `BlockedByHydrator` | `author_blocks_viewer` | `!has_cached_posts` |
| `FilteredTopicsHydrator` | `filtered_topic_ids`, `unfiltered_topic_ids` | topic request, excluded topics, or (`EnableNewUserTopicFiltering` and `has_new_user_topic_ids`) |
| `LanguageCodeHydrator` | `language_code` | `!has_cached_posts` |
`InNetworkCandidateHydrator` sets `in_network` when `author_id == query.user_id` or `author_id` is in `user_features.followed_user_ids`.
`CoreDataCandidateHydrator` loads TES `get_tweet_core_datas`. A miss still returns `Ok(PostCandidate::default())` so length is preserved; found/missing counts increment `{Name}.hydrate` under `hydration.found` / `hydration.missing`. `update` does **not** copy `author_id`. Sources (`ThunderSource`, `PhoenixSource`, `PhoenixTopicsSource`, `PhoenixMOESource`, `TweetMixerSource`) already set `author_id`. `CoreDataHydrationFilter` then drops `author_id == 0`.
`EngagementCountsHydrator` exists under `candidate_hydrators/` and writes `fav_count` / `reply_count` / `repost_count` / `quote_count`, but it is not in `mod.rs` and is not registered.
### Post-selection (`post_selection_hydrators`)
These run on the selector's `selected` set only.
| Hydrator | Writes | Enable |
|---|---|---|
| `VFCandidateHydrator` | `visibility_reason`, `drop_ancillary_posts` | always |
| `AdsBrandSafetyHydrator` | `brand_safety_verdict`, `safety_labels` | `EnableAdsBrandSafetyHydrator` and decider `vf_brand_safety_dark_traffic` is **off** |
| `AdsBrandSafetyVfHydrator` | same fields | `EnableAdsBrandSafetyHydrator` and that decider is **on** |
| `TweetTypeMetricsHydrator` | `tweet_type_metrics` | always |
| `FollowingRepliedUsersHydrator` | `following_replied_user_ids` | `EnableFollowingRepliedUsersFacepile` and `follower_count >= 1000` |
| `MutualFollowJaccardHydrator` | `mutual_follow_jaccard` | `EnableMutualFollowJaccardHydration` and `viewer_minhash` is `Some` |
`VFCandidateHydrator` splits IDs by network:
- In-network tweet IDs and retweet source IDs → VF safety level `TimelineHome`
- Out-of-network tweet IDs, ancestors, and quoted IDs → `TimelineHomeRecommendations`
`drop_ancillary_posts` is true when an ancestor, quoted tweet, or retweeted tweet has a drop-class VF reason. Primary tweet VF is stored in `visibility_reason`.
## Pre-score filters
`PhoenixCandidatePipeline` runs these sequentially after candidate hydration and **before** scorers. Order is the drop-priority order.
| Filter | Drops when | Enable |
|---|---|---|
| `DropDuplicatesFilter` | duplicate `tweet_id` (keeps first) | always |
| `CoreDataHydrationFilter` | `author_id == 0` | always |
| `AgeFilter` | snowflake age of `tweet_id` > `Duration::from_secs(params::MAX_POST_AGE)`, or age unknown | always |
| `SelfTweetFilter` | `author_id == query.user_id` | always |
| `RetweetDeduplicationFilter` | duplicate `retweeted_tweet_id.unwrap_or(tweet_id)` (keeps first) | always |
| `IneligibleSubscriptionFilter` | `subscription_author_id` is `Some` and not in `user_features.subscribed_user_ids` | always |
| `PreviouslySeenPostsFilter` | any related post ID is in `seen_ids` or any impression bloom filter | always |
| `PreviouslySeenPostsBackupFilter` | any related post ID is in `impressed_post_ids`; no-op if that vec is empty | always |
| `PreviouslyServedPostsFilter` | any related post ID is in `served_ids` | `EnableServedFilterAllRequests`, or (`is_bottom_request` and request context is not `ForegroundTruncate`) |
| `MutedKeywordFilter` | tokenized `tweet_text` matches `user_features.muted_keywords`; no-op if keywords empty | always |
| `AuthorSocialgraphFilter` | author muted/blocked, author blocks viewer, quoted author blocks viewer, viewer blocks quoted author, or viewer blocks retweeted user | always |
| `VideoFilter` | `min_video_duration_ms` is `Some` | `query.exclude_videos` |
| `TopicIdsFilter` | topic-request / excluded-topic mismatch (see below) | `is_topic_request()` or `has_excluded_topics()` |
| `NewUserTopicIdsFilter` | not in-network and no expanded `new_user_topic_ids` overlap | `EnableNewUserTopicFiltering` and `has_new_user_topic_ids()` and not a topic request |
Related-post checks use `get_related_post_ids` (`crate::util::candidates_util`, unpublished helper used by the seen / served filters).
`MutedKeywordFilter` tokenizes with `TweetTokenizer` / `UserMutes` / `MatchTweetGroup` inside `tokio::task::block_in_place`.
### Topic filtering
`TopicIdsFilter` uses `TopicIdExpansion` (static category / supertopic maps). `ScoredPostsQuery::is_topic_request` is `!topic_ids.is_empty()`. `is_bulk_topic_request` is `topic_ids.len() > 6`.
- **Single / small topic request:** keep if expanded or unfiltered topic IDs match the requested IDs (including supertopic expansion).
- **Bulk topic request:** keep unless every `filtered_topic_id` is in the complement of the expanded request set; empty topic lists are kept.
- **Excluded topics:** after the keep pass, drop any candidate whose `filtered_topic_ids` intersect the expanded excluded set. Candidates with empty `filtered_topic_ids` are dropped when exclusions are present. Each excluded ID increments `TopicIdsFilter.excluded_topic_id`.
`FollowedGrokTopicsQueryHydrator` fills `new_user_topic_ids` only when new-user topic params are on, the request is not a topic request, `in_network_only` is false, and snowflake age of `user_id` is under `NewUserTopicAgeThresholdSecs`. `update` copies `new_user_topic_ids` only when the hydrated vec is non-empty.
## Post-selection filters
These run after `TopKScoreSelector` and the post-selection hydrators.
| Filter | Drops when |
|---|---|
| `VFFilter` | `visibility_reason` is `SafetyResult` with `Action::Drop(_)`, or any non-`SafetyResult` reason. `None` is kept. |
| `AncillaryVFFilter` | `drop_ancillary_posts == Some(true)` |
| `DedupConversationFilter` | not the highest `score` in the conversation. Conversation id is `ancestors.iter().min()` or `tweet_id`. |
`ForYouCandidatePipeline` sets `filters()`, `hydrators()`, `post_selection_hydrators()`, and `post_selection_filters()` to empty slices. Organic visibility filtering happens inside Phoenix before `ScoredPostsSource` returns.
## Layout
:::files
home-mixer/
query_hydrators/ # QueryHydrator<ScoredPostsQuery>
candidate_hydrators/ # Hydrator / CachedHydrator<ScoredPostsQuery, PostCandidate>
filters/ # Filter<ScoredPostsQuery, PostCandidate>
candidate_pipeline/
phoenix_candidate_pipeline.rs # registries
for_you_candidate_pipeline.rs # served-history query hydrators only
models/query.rs # live ScoredPostsQuery
candidate-pipeline/
query_hydrator.rs
hydrator.rs
filter.rs
candidate_pipeline.rs # execute + run_hydrators / run_filters
:::
## Failure modes
| Signal | Meaning | What continues |
|---|---|---|
| `query_hydrator` log `Failed: …` | `hydrate` returned `Err` | Query keeps prior / default field; other hydrators still merge |
| `Skipped: length_mismatch expected=N got=M` on a hydrator | Returned vec length ≠ input | That hydrator's fields are not applied; candidates stay in place |
| `CachedHydrator length_mismatch` | Miss-path client returned the wrong length | Every candidate gets `Err` for that hydrator |
| `CoreDataHydrationFilter` removals | `author_id == 0` after sources + TES | Candidate never reaches scoring |
| `has_cached_posts == true` | Redis cache ≥ 500 posts | TES / SocialGraph / in-network hydrators skip; cached fields stand |
| `PreviouslySeenPostsBackupFilter` removes nothing | `impressed_post_ids` empty | Expected unless another writer filled the field |
| `Aggregation service call failed: …` | UAS fetch failed | Scoring or retrieval sequence stays `None` |
| `CachedPostsQueryHydrator redis GET timed out` / `GET error` | 300 ms timeout or Redis error | `cached_posts` stays empty; live TES hydration runs |
| `FollowedGrokTopics MH get timed out` | 300 ms Manhattan timeout | Followed-topic / new-user topic fields stay unset |
| Empty For You / Scored Posts feed | Test-user short-circuit or failed query build | See troubleshooting; not a filter miss |
`params::*` identifiers (`MAX_POST_AGE`, `EnableCachedPosts`, `RESULT_SIZE`, …) come from unpublished `crate::params`. This checkout imports them but does not ship the module; do not assume numeric defaults from this tree.
## Add a component
Query-level data → `QueryHydrator`. Per-candidate enrichment that must not drop rows → `Hydrator` / `CachedHydrator`. Drop / keep → `Filter`. Post-score VF-style work → `post_selection_hydrators` then `post_selection_filters`.
`hydrate` returns a default struct plus owned fields. `update` copies only those fields. Do not assign `user_features = hydrated.user_features` unless you own the whole struct (the live Phoenix hydrators copy one list at a time).
Candidate hydrators: one `Result` per input, same index. Use `Ok(Default::default())` for misses. Use a filter to drop. Length mismatches are skipped, not retried.
Push into the matching `vec!` in `PhoenixCandidatePipeline::build_with_clients` (or For You's `build`). Declare `pub mod` in the crate `mod.rs`. `enable` should read `query.params` / `query.decider` the same way siblings do.
`CandidatePipeline::components()` lists every registered name by `PipelineStage`. Filter spans expose `removed_per_filter [Name=N,…]`. Hydrator skips show `length_mismatch` warnings.
## Next
`CandidatePipeline::execute` stages from query hydration through side effects.
Source, Hydrator, Filter, Scorer, Selector, QueryHydrator, and SideEffect contracts.
Query fields, QueryBuilder defaults, and response mapping.
What runs after query hydration and before these hydrators.
What scores the `kept` set after pre-score filters.
What this checkout can run locally versus production Home Mixer.
---
## 21. Scorers and weights
> PhoenixScorer cluster and egress fallback, WeightedScorer and RankingScorer formulas, AuthorDiversityScorer decay, OON and VM rankers, TopKScoreSelector.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/21-scorers-and-weights.md
- Generated: 2026-08-13T19:48:22.973Z
### Source Files
- `home-mixer/scorers/mod.rs`
- `home-mixer/scorers/phoenix_scorer.rs`
- `home-mixer/scorers/weighted_scorer.rs`
- `home-mixer/scorers/ranking_scorer.rs`
- `home-mixer/scorers/author_diversity_scorer.rs`
- `home-mixer/scorers/oon_scorer.rs`
- `home-mixer/scorers/vm_ranker.rs`
- `home-mixer/selectors/top_k_score_selector.rs`
---
title: "Scorers and weights"
description: "PhoenixScorer cluster and egress fallback, WeightedScorer and RankingScorer formulas, AuthorDiversityScorer decay, OON and VM rankers, TopKScoreSelector."
---
`PhoenixCandidatePipeline` scores filtered `PostCandidate`s sequentially with `PhoenixScorer`, then `RankingScorer`, then `VMRanker` when `EnableVMRanker` is on. `TopKScoreSelector` sorts on `candidate.score` and keeps `TOP_K_CANDIDATES_TO_SELECT`. `ForYouCandidatePipeline` has an empty scorer list and uses `BlenderSelector` after `ScoredPostsSource` already ran this path.
Numeric defaults live in unpublished `crate::params` and feature-switch `Params`. This checkout exposes the formulas, param keys, and enable predicates, not the production weight table.
The root README still lists `WeightedScorer`, `AuthorDiversityScorer`, and `OONScorer` as sequential stages. `home-mixer/scorers/mod.rs` exports only `phoenix_scorer`, `ranking_scorer`, and `vm_ranker`. `RankingScorer` inlines the weighted sum, author-diversity decay, and out-of-network factor.
## Scoring stage
`CandidatePipeline::execute` hydrates, sources, hydrates candidates, then filters before `score()`. Enabled scorers run in vector order. Each `Scorer::run` must return one result per input candidate in the same order. A length mismatch becomes `Scorer length_mismatch expected=… got=…` for every slot; `update_all` copies only `Ok` results, so failed slots keep prior fields.
```text
filtered PostCandidate[]
│
▼
PhoenixScorer writes phoenix_scores, prediction_request_id, last_scored_at_ms
│ skipped when has_cached_posts
▼
RankingScorer writes weighted_score, then score (diversity + OON)
│
▼
VMRanker overwrites score when EnableVMRanker
│ skipped when the flag is off
▼
TopKScoreSelector sort by score desc, take TOP_K_CANDIDATES_TO_SELECT
│
▼
execute() truncate result_size() == params::RESULT_SIZE
```
```mermaid
flowchart TB
subgraph pipeline ["PhoenixCandidatePipeline"]
PS["PhoenixScorer"]
RS["RankingScorer"]
VM["VMRanker"]
TK["TopKScoreSelector"]
PS --> RS --> VM --> TK
end
subgraph unpublished ["Unpublished in this checkout"]
P["crate::params / feature switches"]
U["candidates_util::vqv_weight normalize_score"]
end
subgraph remote ["Remote inference"]
PC["PhoenixPredictionClient"]
EG["EgressPhoenixPredictionClient"]
VR["VMRankerClient"]
end
PS -->|"UseEgressSidecar then fallback"| EG
PS --> PC
RS --> P
RS --> U
VM --> VR
VM --> P
TK --> P
```
Set by `CachedPostsQueryHydrator` when Redis returns at least `MIN_CACHED_POSTS_THRESHOLD` (`500`) posts. Disables `PhoenixScorer` (and most retrieval hydrators). `RankingScorer.enable` is always `true`, so cached candidates are re-combined from stored `phoenix_scores`.
Filled by `ScoringSequenceQueryHydrator`. If `None`, `PhoenixScorer` returns `Ok(PostCandidate::default())` for every candidate and does not call predict.
## PhoenixScorer
`PhoenixScorer` holds two `PhoenixPredictionClient`s: `phoenix_client` and `egress_client` (`EgressPhoenixPredictionClient` in prod). It is the only scorer that calls the ranker model.
### Enable and product surface
| Condition | Behavior |
|---|---|
| `query.has_cached_posts` | `enable` is `false`; no predict |
| `query.scoring_sequence.is_none()` | No RPC; defaults for every candidate |
| `query.in_network_only` | `ProductSurface::HomeTimelineRankedFollowing` |
| otherwise | `ProductSurface::HomeTimelineRanking` |
The request is `build_prediction_request(query, candidates, product_surface)` (unpublished util). Response scores are looked up by `CandidateHelpers::get_original_tweet_id()` (`retweeted_tweet_id` or `tweet_id`).
### Cluster resolution
`resolve_cluster` starts from `PhoenixInferenceClusterId`, then may replace it:
1. If `PhoenixRankerNewUserHistoryThreshold > 0` and `scoring_sequence.metadata.length` (or `0` if missing) is below that threshold, use `PhoenixRankerNewUserInferenceClusterId`.
2. Else if a decider is present:
- `PhoenixCluster::Experiment1Fou` + `override_qf_use_lap7` → `Experiment1Lap7`
- `PhoenixCluster::Experiment1Lap7` + `override_qf_use_fou` → `Experiment1Fou`
3. Otherwise keep the configured cluster.
### Egress fallback
When true, `predict` goes to `egress_client` first. On error, logs `Egress predict failed, falling back` and retries the same `cluster` + request on `phoenix_client`. Direct Phoenix failures are not retried.
A failed predict (after fallback) returns `Err("Phoenix prediction failed: …")` for every candidate. Successful predict writes:
- `phoenix_scores` from `predictions.candidate_scores`
- `prediction_request_id` = `query.prediction_id`
- `last_scored_at_ms` = `current_timestamp_millis()`
`PhoenixExperimentsSideEffect` (shadow traffic only) fans out `PhoenixCluster::VARIANTS` that report `is_shadow_eligible()` on the same egress/phoenix choice, and does not fall back.
## RankingScorer
Always enabled. Loads `ScoringWeights` from `query.params`, then for each candidate:
1. `raw = compute_weighted_score`
2. `weighted_score = normalize_score(candidate, raw)` (unpublished)
3. Author-diversity multiply on the normalized vector
4. If `in_network == Some(false)`, multiply by `effective_oon_weight`
5. Write `weighted_score` (step 2) and `score` (step 4)
`TopKScoreSelector` reads `score`, not `weighted_score`.
### Weighted sum
Missing `Option` scores contribute `0.0`. VQV weights can be zeroed by unpublished `candidates_util::vqv_weight` / `quoted_vqv_weight` using `MinVideoDurationMs`, `VqvWeight`, `QuotedVqvWeight`, and `EnableQuotedVqvDurationCheck`. `VideoDurationCandidateHydrator` fills `min_video_duration_ms`; `QuoteHydrator` fills `quoted_video_duration_ms`.
| `PhoenixScores` field | Weight param | Role in `ScoringWeights` |
|---|---|---|
| `favorite_score` | `FavoriteWeight` | positive sum |
| `reply_score` | `ReplyWeight` | positive sum |
| `retweet_score` | `RetweetWeight` | positive sum |
| `photo_expand_score` | `PhotoExpandWeight` | positive sum |
| `click_score` | `ClickWeight` | positive sum |
| `profile_click_score` | `ProfileClickWeight` | positive sum |
| `vqv_score` | `VqvWeight` (gated) | positive sum |
| `share_score` | `ShareWeight` | positive sum |
| `share_via_dm_score` | `ShareViaDmWeight` | positive sum |
| `share_via_copy_link_score` | `ShareViaCopyLinkWeight` | positive sum |
| `dwell_score` | `DwellWeight` | positive sum |
| `quote_score` | `QuoteWeight` | positive sum |
| `quoted_click_score` | `QuotedClickWeight` | positive sum |
| `quoted_vqv_score` | `QuotedVqvWeight` (gated) | positive sum |
| `follow_author_score` | `FollowAuthorWeight` | positive sum |
| `dwell_time` | `ContDwellTimeWeight` | applied, not in `total_sum` |
| `click_dwell_time` | `ContClickDwellTimeWeight` | applied, not in `total_sum` |
| `not_interested_score` | `NotInterestedWeight` | negative sum |
| `block_author_score` | `BlockAuthorWeight` | negative sum |
| `mute_author_score` | `MuteAuthorWeight` | negative sum |
| `report_score` | `ReportWeight` | negative sum |
| `not_dwelled_score` | `NotDwelledWeight` | negative sum |
```text
positive_sum = favorite + reply + retweet + photo_expand + click + profile_click
+ vqv + share + share_via_dm + share_via_copy_link
+ dwell + quote + quoted_click + quoted_vqv + follow_author
negative_sum = -(not_interested + block_author + mute_author + report + not_dwelled)
total_sum = positive_sum + negative_sum
offset(combined):
total_sum == 0 → combined.max(0)
combined < 0 → (combined + negative_sum) / total_sum * NEGATIVE_SCORES_OFFSET
else → combined + NEGATIVE_SCORES_OFFSET
```
`NEGATIVE_SCORES_OFFSET` is an unpublished crate constant, not a per-request param.
### Author diversity
After normalization, candidates are sorted by weighted score descending. Per `author_id`, the *n*th appearance (`n` starting at 0) is multiplied by:
```text
(1 - AuthorDiversityFloor) * AuthorDiversityDecay^n + AuthorDiversityFloor
```
The first post from an author keeps a multiplier of `1.0`. Later posts decay toward `AuthorDiversityFloor`.
### Out-of-network factor
Applied only when `candidate.in_network == Some(false)` (`None` and `Some(true)` are unchanged):
| Query state | Factor |
|---|---|
| `topic_ids` non-empty | `TopicOonWeightFactor` |
| user snowflake age `< NewUserAgeThresholdSecs` and `followed_user_ids.len() >= NEW_USER_MIN_FOLLOWING` | `NEW_USER_OON_WEIGHT_FACTOR` |
| otherwise | `OonWeightFactor` |
Age uses unpublished `duration_since_creation_opt(query.user_id)`. `NEW_USER_MIN_FOLLOWING` and `NEW_USER_OON_WEIGHT_FACTOR` are crate constants.
## Unwired files in `home-mixer/scorers/`
`weighted_scorer.rs`, `author_diversity_scorer.rs`, and `oon_scorer.rs` are not modules in `scorers/mod.rs` and are not boxed into `PhoenixCandidatePipeline`. They import the older `crate::candidate_pipeline::candidate` types (that module also is not compiled). Treat them as a snapshot of the decomposition `RankingScorer` now owns.
### WeightedScorer
Writes only `weighted_score`. Same `apply(score, weight)` pattern with crate constants (`FAVORITE_WEIGHT`, …) instead of feature switches. Differences from `RankingScorer`:
- No `quoted_vqv_score`, `click_dwell_time`, or `not_dwelled_score`
- VQV weight is `VQV_WEIGHT` only when `video_duration_ms > MIN_VIDEO_DURATION_MS`, else `0.0`
- Offset uses `WEIGHTS_SUM` / `NEGATIVE_WEIGHTS_SUM` constants
### AuthorDiversityScorer
Constructed with `AUTHOR_DIVERSITY_DECAY` and `AUTHOR_DIVERSITY_FLOOR`. Sorts by `weighted_score` (missing sorts as `-∞`) and writes `score = weighted_score * multiplier` using the same `(1 - floor) * decay^position + floor` formula. Does not apply an OON factor.
### OONScorer
`score = base_score * OON_WEIGHT_FACTOR` when `in_network == Some(false)`; otherwise leaves `score` unchanged. No topic or new-user branch.
## VMRanker
`VMRanker.enable`. When false, `RankingScorer`'s `score` is what `TopKScoreSelector` sees.
On enable, `VMRankerCluster::parse(VMRankerClusterId)` and `client.rank` run. gRPC failure returns `Err("VMRanker gRPC call failed: …")` for every candidate. On success, `score` is the response value for `tweet_id`, or the previous `c.score` if that id is missing.
`RankRequest` includes:
| Field | Source |
|---|---|
| `viewer_id` | `query.user_id` |
| `request_timestamp_ms` | `query.request_time_ms` |
| `value_model_id` | `VMRankerValueModelId` |
| `viewer_following_count` | `followed_user_ids.len()` |
| `new_user_age_threshold_secs` | `NewUserAgeThresholdSecs` |
| `dpp_params` | `Some` only if `VMRankerDppTheta > 0` or `VMRankerDppMaxSelectedRank > 0` |
Each `RankCandidate` copies Phoenix action scores (including `not_dwelled_score`, `dwell_time`, `click_dwell_time`; not `quoted_vqv_score`), plus `in_network` (default `false`), `is_retweet` / `is_reply`, follower count, and `vqv_ineligible` when `vqv_weight == 0.0`.
## TopKScoreSelector
```rust
fn score(&self, candidate: &PostCandidate) -> f64 {
candidate.score.unwrap_or(f64::NEG_INFINITY)
}
fn size(&self) -> Option {
Some(params::TOP_K_CANDIDATES_TO_SELECT)
}
```
`Selector::select` sorts descending, then `split_off` at `size`. Missing `score` sorts last. After selection, `execute` still truncates the selected list to `PhoenixCandidatePipeline::result_size()` (`params::RESULT_SIZE`) before side effects.
## Local demo weights
`phoenix/run_pipeline.py` is the only runnable weighted sum in this checkout. After sigmoid on ranker logits it uses a four-term demo mix, not `RankingScorer`:
```python
weighted = (
all_probs[:, IDX_FAV] * 1.0
+ all_probs[:, IDX_REPLY] * 0.5
+ all_probs[:, IDX_RT] * 0.3
+ all_probs[:, IDX_DWELL] * 0.2
)
```
`IDX_FAV = 1`, `IDX_REPLY = 4`, `IDX_RT = 6`, `IDX_DWELL = 11`. `IDX_VQV = 13` is printed, not mixed. Production `runners.ACTIONS` labels (`favorite_score`, `reply_score`, `repost_score`, …) align with Phoenix score keys; Home Mixer uses `retweet_score` rather than `repost_score`.
## Failure and skip matrix
| Event | Candidates after the scorer |
|---|---|
| `PhoenixScorer` disabled (`has_cached_posts`) | Prior `phoenix_scores` unchanged |
| Missing `scoring_sequence` | Defaults; `RankingScorer` still runs |
| Egress predict fails, sidecar on | Retry on `phoenix_client` |
| Phoenix predict fails | All `Err`; no `phoenix_scores` update |
| Scorer length mismatch | All `Err`; prior fields kept |
| `EnableVMRanker` false | `RankingScorer.score` used for TopK |
| VM gRPC error | All `Err`; `RankingScorer.score` kept |
| VM response missing a `tweet_id` | Keep existing `score` |
`crate::params`, `crate::util::score_normalizer`, `crate::util::candidates_util`, `PhoenixPredictionClient`, and `PhoenixScores` (re-exported from `xai_candidate_pipeline`) are not buildable from this snapshot. Local verification of the *formulas* is `run_pipeline.py`; verification of Home Mixer scoring requires the unpublished crates.
## Next
Per-action logits, the demo weighted sum, and how production weight tables differ.
`ActionName` indices, `IDX_*` constants, `runners.ACTIONS`, and scorer weight keys.
Where the scorer vector sits in `CandidatePipeline.execute`.
`Scorer` length-match and `update` contracts.
What this checkout can run versus unpublished Home Mixer params and clients.
How `in_network` and `in_network_only` change sourcing and the OON factor.
---
## 22. Thunder GetInNetworkPosts
> InNetworkPostsService RPC fields, PostStore timelines and retention, Kafka ingest, semaphore capacity, and Strato following-list fallback.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/22-thunder-getinnetworkposts.md
- Generated: 2026-08-13T19:49:50.864Z
### Source Files
- `thunder/thunder_service.rs`
- `thunder/posts/post_store.rs`
- `thunder/main.rs`
- `thunder/kafka_utils.rs`
- `thunder/kafka/tweet_events_listener.rs`
- `home-mixer/sources/thunder_source.rs`
- `thunder/lib.rs`
---
title: "Thunder GetInNetworkPosts"
description: "InNetworkPostsService RPC fields, PostStore timelines and retention, Kafka ingest, semaphore capacity, and Strato following-list fallback."
---
`InNetworkPostsService.get_in_network_posts` is Thunder's in-network candidate RPC. `ThunderServiceImpl` acquires a process-wide `tokio::sync::Semaphore` with `try_acquire`, hydrates an empty following list from `StratoClient` only when `debug` is also true, scans in-memory `PostStore` timelines on a blocking thread, then returns the newest `LightPost`s up to `max_results`. Home Mixer calls this through `ThunderSource`. The `thunder/` tree in this checkout is a production snapshot: `args`, `config`, `strato_client`, `metrics`, `schema`, `o2`, `xai_thunder_proto`, `xai_kafka`, and `xai_http_server` are not published here, so the binary does not build or serve locally.
This checkout cannot start Thunder. Topic and destination strings are empty, SASL env-var names are empty, and numeric caps such as `MAX_POSTS_TO_RETURN` live in the unpublished `thunder/config` module. Use this page as the production contract, not as a local runbook.
## Architecture
```mermaid
flowchart LR
subgraph Feeder["Thunder feeder — is_serving = false"]
TE["Thrift TweetEvent Kafka"] --> L1["tweet_events_listener"]
L1 --> PR["InNetworkEvent producer"]
end
subgraph Serving["Thunder serving — is_serving = true"]
PR --> V2["tweet_events_listener_v2"]
V2 --> PS["PostStore DashMaps"]
STR["StratoClient"] -.->|"empty following AND debug"| SVC
PS --> SVC["ThunderServiceImpl"]
SVC --> SEM["request_semaphore"]
end
subgraph HomeMixer["Home Mixer"]
FUH["FollowedUserIdsQueryHydrator"] --> TS["ThunderSource"]
TS -->|"GetInNetworkPostsRequest debug = false"| SVC
SVC --> TS
TS --> PC["PostCandidate + InNetworkReply"]
end
```
Two processes share the same crate. The feeder (`!args.is_serving`) consumes Thrift `TweetEvent`s and publishes protobuf `InNetworkEvent`s. Serving Thunder (`args.is_serving`) consumes those events into `PostStore`, then exposes `InNetworkPostsService` over HTTP/gRPC via `xai_http_server`.
## Startup
`thunder/main.rs` always constructs `PostStore`, `StratoClient`, and `ThunderServiceImpl`, then starts Kafka. Serving mode blocks readiness on Kafka catch-up.
`PostStore::new(args.post_retention_seconds, args.request_timeout_ms)`. `ThunderServiceImpl::new` takes that store, a `StratoClient`, and `args.max_concurrent_requests` as the semaphore permit count. The gRPC server is `InNetworkPostsServiceServer` with Zstd accept and send compression.
`kafka_utils::start_kafka` opens an `mpsc` channel of size `args.kafka_num_threads`. Serving mode starts `start_tweet_event_processing_v2`. Feeder mode starts `start_tweet_event_processing` plus an `InNetworkEvent` producer.
Serving Thunder waits for one channel message per Kafka thread, then `post_store.finalize_init()` (sort timelines, trim, re-drop tombstoned IDs). It starts the 5-second stats logger and a 2-minute auto-trim. Only then does `http_server.set_readiness(true)`.
`PostStore::default()` is 2 days of retention and a 0 ms request timeout (no timeout). Production serving uses the CLI values, not that default.
## RPC: GetInNetworkPosts
:::endpoint POST InNetworkPostsService/GetInNetworkPosts Recent posts from followed authors
Thunder implements `InNetworkPostsService`. Home Mixer builds `GetInNetworkPostsRequest` in `ThunderSource`; the handler is `ThunderServiceImpl::get_in_network_posts`.
Viewer ID. Used for Strato fallback, request-timeout logs, and dropping retweets whose `source_user_id` is the viewer.
Author IDs to scan. Home Mixer copies `query.user_features.followed_user_ids`. Truncated to the first `MAX_INPUT_LIST_SIZE` entries.
Post IDs to skip. Home Mixer sends `query.seen_ids`. Truncated to `MAX_INPUT_LIST_SIZE`.
Cap after recency sort. `0` means `MAX_VIDEOS_TO_RETURN` when `is_video_request`, otherwise `MAX_POSTS_TO_RETURN`. Those constants are unpublished.
Home Mixer sets `query.params.get(ThunderAlgorithm)`. The handler never reads this field.
Enables request/response logs. Also required for Strato fallback. `ThunderSource` always sends `false`.
Selects `get_videos_by_users` instead of `get_all_posts_by_users`. `ThunderSource` always sends `false`.
Recency-sorted posts after store filters and `score_recent`.
:::
```text
GetInNetworkPostsRequest # Home Mixer ThunderSource
user_id: query.user_id
following_user_ids: query.user_features.followed_user_ids as u64
max_results: query.params.get(ThunderMaxResults)
exclude_tweet_ids: query.seen_ids
algorithm: query.params.get(ThunderAlgorithm)
debug: false
is_video_request: false
```
```text
GetInNetworkPostsResponse
posts[] LightPost:
post_id, author_id, created_at
in_reply_to_post_id, in_reply_to_user_id
is_retweet, is_reply
source_post_id, source_user_id
has_video, conversation_id
```
Handler sequence:
1. `request_semaphore.try_acquire()`. Failure increments `REJECTED_REQUESTS` and returns immediately.
2. If `following_user_ids` is empty **and** `debug`, call `strato_client.fetch_following_list(user_id, MAX_INPUT_LIST_SIZE)`.
3. Truncate following and exclude lists. Resolve `max_results`.
4. `tokio::task::spawn_blocking`: `get_videos_by_users` or `get_all_posts_by_users`, then `score_recent`.
5. Return `GetInNetworkPostsResponse { posts }`.
`score_recent` sorts by `created_at` descending (`sort_unstable_by_key(Reverse)`) and `take(max_results)`. There is no engagement score inside Thunder.
```mermaid
sequenceDiagram
participant HM as ThunderSource
participant S as ThunderServiceImpl
participant Sem as request_semaphore
participant St as StratoClient
participant PS as PostStore
HM->>S: GetInNetworkPostsRequest
S->>Sem: try_acquire
alt no permit
Sem-->>S: RESOURCE_EXHAUSTED
S-->>HM: Status resource_exhausted
else permit
opt following empty AND debug
S->>St: fetch_following_list
St-->>S: user IDs or INTERNAL
end
S->>PS: get_all_posts_by_users / get_videos_by_users
PS-->>S: LightPost[]
S-->>HM: GetInNetworkPostsResponse
end
```
## Semaphore capacity
The permit count is `args.max_concurrent_requests`. The handler never waits.
| Condition | Metric | gRPC status | Message |
|-----------|--------|-------------|---------|
| `try_acquire` fails | `REJECTED_REQUESTS` | `RESOURCE_EXHAUSTED` | `Server at capacity, please retry` |
| Permit granted | `IN_FLIGHT_REQUESTS` +1, Drop guard −1 | continues | — |
| Strato fetch fails | — | `INTERNAL` | `Failed to fetch following list: …` |
| `spawn_blocking` join fails | — | `INTERNAL` | `Failed to process posts: …` |
Home Mixer maps any Thunder status to `Err(format!("ThunderSource: {}", e))`. A capacity reject surfaces as that string, not as a retry inside `ThunderSource`.
A second semaphore exists only on the serving ingest path: `Semaphore::new(3)` in `tweet_events_listener_v2`. After Kafka catch-up, each insert/delete batch acquires one of those three permits so ingest does not starve RPC CPU.
## Strato following-list fallback
```text
if req.following_user_ids.is_empty() && req.debug:
strato_client.fetch_following_list(user_id as i64, MAX_INPUT_LIST_SIZE as i32)
```
| Caller | `following_user_ids` | `debug` | Strato? |
|--------|----------------------|---------|---------|
| `ThunderSource` | `FollowedUserIdsQueryHydrator` / SocialGraph | `false` | Never |
| Debug RPC with a non-empty list | provided | `true` | No |
| Debug RPC with an empty list | empty | `true` | Yes; failure is `INTERNAL` |
| Non-debug RPC with an empty list | empty | `false` | No; store scan runs with zero authors |
Home Mixer therefore never uses Thunder's Strato path. An empty SocialGraph following list yields an empty Thunder response, not a fallback fetch.
## PostStore timelines and retention
`PostStore` keeps full `LightPost`s in `posts: DashMap` and three per-author `VecDeque` timelines (`post_id` + `created_at` only).
| Map | Membership | Per-author take |
|-----|------------|-----------------|
| `original_posts_by_user` | `!is_reply && !is_retweet` | `MAX_ORIGINAL_POSTS_PER_AUTHOR` |
| `secondary_posts_by_user` | replies and retweets | `MAX_REPLY_POSTS_PER_AUTHOR` |
| `video_posts_by_user` | video-eligible posts | `MAX_VIDEO_POSTS_PER_AUTHOR` |
| `deleted_posts` | tombstones | not scanned as a timeline |
**Insert.** `insert_posts` drops posts with `created_at >= now` or age `> retention_seconds`, sorts remaining by `created_at`, then `insert_posts_internal`. Already-tombstoned IDs and already-stored IDs are skipped.
**Video eligibility** (store insert):
- Replies are never video-eligible.
- Otherwise `has_video`, or a retweet whose `source_post_id` is a non-reply `LightPost` with `has_video`.
Feeder ingest sets `has_video` only when the first media entity is `VideoInfo` and `duration_millis >= MIN_VIDEO_DURATION_MS`.
**Deletes.** `mark_as_deleted` removes the `LightPost`, inserts a tombstone, and appends a `TinyPost` under `DELETE_EVENT_KEY` on `original_posts_by_user` so tombstones expire with the same trim loop.
**Lookup.** `get_posts_from_map` walks authors in request order. Per author it:
1. Breaks the whole request if `request_timeout` is non-zero and elapsed (`POST_STORE_REQUEST_TIMEOUTS`).
2. Scans the deque newest-first, skips `exclude_tweet_ids`, and stops after `MAX_TINY_POSTS_PER_USER_SCAN`.
3. Resolves `LightPost`, drops tombstones, drops retweets with `source_user_id == request_user_id`.
4. For the secondary map only (`following_users` non-empty), keeps a reply when:
- `in_reply_to_post_id` is unset, or
- the replied-to post is an original (not reply, not retweet), or
- it is a reply-to-reply whose `in_reply_to_post_id` equals `conversation_id` **and** `in_reply_to_user_id` is in the following set.
Missing parent posts drop the reply.
5. Takes at most `max_per_user`.
`get_all_posts_by_users` concatenates original + secondary. `get_videos_by_users` reads only `video_posts_by_user`.
**Retention.** Serving auto-trim runs every 2 minutes. `trim_old_posts` pops deque fronts older than `retention_seconds`, removes matching `posts` entries, and expires `DELETE_EVENT_KEY` tombstones. `finalize_init` sorts every deque by `created_at` ascending, trims, then removes any ID still listed in `deleted_posts` so out-of-order create/delete during catch-up cannot resurrect a post.
## Kafka ingest
Topic and dest constants in `kafka_utils.rs` are empty strings in this snapshot. The mode split is still in code.
| Mode | `args.is_serving` | Consumer | Writer |
|------|-------------------|----------|--------|
| Feeder | `false` | Thrift `TweetEvent` (`deserialize_tweet_event`) | `InNetworkEvent` producer; does **not** write `PostStore` |
| Serving | `true` | Protobuf `InNetworkEvent` (`deserialize_tweet_event_v2`) | `PostStore::insert_posts` / `mark_as_deleted` |
**Feeder filters**
- Skip `TweetCreateEvent` when `core_data.nullcast` is true.
- Skip `TweetDeleteEvent` when `now - created_at_secs > post_retention_seconds`.
- Treat `QuotedTweetDeleteEvent` as a delete of `quoting_tweet_id`.
- Other event variants are logged and ignored.
**Serving consumer**
- `group_id` is `{kafka_group_id}-{uuid}`.
- `max_partition_fetch_bytes` is 100 MiB (feeder uses 10 MiB).
- Threads split `args.kafka_tweet_events_v2_num_partitions`.
- Catch-up: when `sum(partition lag) < lags.len() * kafka_batch_size`, the thread sends on the init channel. `main` waits for `kafka_num_threads` sends.
- After catch-up, insert batches hold one of three ingest permits.
- v2 marks `is_reply` if the flag is set **or** `in_reply_to_post_id` / `in_reply_to_user_id` is present.
Both modes spawn `start_partition_lag_monitor` on `args.lag_monitor_interval_secs`. Poll errors increment `KAFKA_POLL_ERRORS` and sleep 100 ms.
## Home Mixer caller
`PhoenixCandidatePipeline` always includes `ThunderSource` next to Phoenix, Tweet Mixer, and `CachedPostsSource`.
| Item | Behavior |
|------|----------|
| `enable` | `!query.has_cached_posts` (cache hit of ≥ 500 Redis posts disables Thunder) |
| Channel | `ThunderCluster::parse(ThunderClusterId)` then `ThunderCluster::resolve(..., decider)`; missing channel is `"ThunderSource: no available channel"` |
| Following | `query.user_features.followed_user_ids` from `FollowedUserIdsQueryHydrator` |
| Exclude | `query.seen_ids` |
| `served_type` | `ForYouInNetwork` unless `query.in_network_only`, then `RankedFollowing` |
| Replies | Each post with `in_reply_to_post_id` is stored on `query.in_network_replies` |
| Ancestors | `[in_reply_to_post_id]` plus `conversation_id` when it differs |
`InNetworkCandidateHydrator` later sets `candidate.in_network` from the same following list (or self). That flag is independent of Thunder's store filters.
## Args used by this snapshot
`thunder/args` is unpublished. These fields are read from `Args`:
| Field | Used for |
|-------|----------|
| `post_retention_seconds` | Store retention, insert filter, feeder delete skip, log days |
| `request_timeout_ms` | Mid-scan abort in `get_posts_from_map` |
| `max_concurrent_requests` | RPC semaphore permits |
| `grpc_port`, `http_port` | `HttpServer` + `GrpcConfig` |
| `enable_profiling` | `xai_profiling::spawn_server(3000, …)` |
| `is_serving` | Feeder vs serving Kafka path and catch-up |
| `kafka_num_threads` | Thread count and catch-up barrier |
| `kafka_batch_size` | Batch threshold and catch-up lag compare |
| `kafka_group_id` | Consumer group prefix |
| `kafka_tweet_events_v2_num_partitions` | Serving partition range |
| `tweet_events_num_partitions` | Feeder partition range |
| `in_network_events_consumer_dest` | Serving consumer dest |
| `auto_offset_reset`, `skip_to_latest`, `fetch_timeout_ms` | Consumer config |
| `lag_monitor_interval_secs` | Partition-lag scrape |
| `security_protocol`, SASL user/mechanism/password (consumer and producer) | Kafka SSL |
Optional profiling is independent of readiness. Termination is `http_server.wait_for_termination()`.
## Unpublished config identifiers
Referenced but not defined in this checkout:
`MAX_INPUT_LIST_SIZE`, `MAX_POSTS_TO_RETURN`, `MAX_VIDEOS_TO_RETURN`, `MAX_ORIGINAL_POSTS_PER_AUTHOR`, `MAX_REPLY_POSTS_PER_AUTHOR`, `MAX_VIDEO_POSTS_PER_AUTHOR`, `MAX_TINY_POSTS_PER_USER_SCAN`, `DELETE_EVENT_KEY`, `MIN_VIDEO_DURATION_MS`.
Do not invent values. The only numeric defaults present in-repo are `PostStore::default()` (2 days, 0 ms timeout), ingest semaphore `3`, auto-trim interval `2` minutes, stats interval `5` seconds, and cached-posts disable threshold `500`.
## Errors and verification
| Symptom | Cause | Check |
|---------|--------|--------|
| `ThunderSource: status: ResourceExhausted, message: "Server at capacity, please retry"` | RPC semaphore exhausted | Lower `max_concurrent_requests` pressure or retry; Thunder does not queue |
| `ThunderSource: no available channel` | `ThunderClient.get_random_channel` returned `None` | Cluster id / decider, not PostStore |
| Empty in-network set with a populated following list | Retention miss, excludes, timeout mid-scan, or reply/retweet filters | `POST_STORE_REQUEST_TIMEOUTS`, `POST_STORE_POSTS_RETURNED` |
| Empty set and empty following | SocialGraph returned no IDs; Strato does not run because `debug` is false | `FollowedUserIdsQueryHydrator` |
| Thunder never called | `has_cached_posts` | Redis cache ≥ 500 candidates |
| Binary will not build | Unpublished crates and empty Kafka dests | Expected in this checkout |
Serving logs `Kafka init took …` then `HTTP/gRPC server is ready`. Stats lines look like `PostStore Stats: N users, M total posts, D deleted posts`.
## Next
Thunder versus Phoenix enable predicates and how `in_network_only` changes `served_type`.
Where `ThunderSource` sits in `PhoenixCandidatePipeline` next to Phoenix and cached posts.
What this checkout can execute versus unpublished Thunder and Home Mixer crates.
`RESOURCE_EXHAUSTED` and other For You failure signals.
---
## 23. Grox plans and tasks
> PlanMaster.ALL_PLANS, TaskEligibility values, DAG TASK_DEPENDENCIES, ContentClassifier, and TaskResult merge rules.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/23-grox-plans-and-tasks.md
- Generated: 2026-08-13T19:49:23.731Z
### Source Files
- `grox/plans/plan_master.py`
- `grox/plans/plan.py`
- `grox/schedules/types.py`
- `grox/dispatcher.py`
- `grox/engine.py`
- `grox/tasks/task.py`
- `grox/classifiers/content/classifier.py`
- `grox/plans/plan_post_safety.py`
---
title: "Grox plans and tasks"
description: "PlanMaster.ALL_PLANS, TaskEligibility values, DAG TASK_DEPENDENCIES, ContentClassifier, and TaskResult merge rules."
---
`PlanMaster.exec` fans every inbound `TaskPayload` across `ALL_PLANS` in parallel. Each `Plan` runs only when its `REQUIRED_ELIGIBILITY` is in `payload.eligibilities`, then executes a named `TASKS` / `TASK_DEPENDENCIES` DAG. Non-`None` plan `TaskResult`s are folded by `PlanMaster.merge_results` before `Engine` puts the merged result on `resp_queue`.
Startup, process layout, and Kafka generator wiring live on [Execute Grox content plans](/execute-grox-plans). This page is the plan/task contract: eligibility values, DAGs, classifiers, and merge rules.
`grox/main.py` imports unpublished modules (`grox.service.GrpcServer`, `grox.config.config`, `grox.data_loaders.data_types`, `grok_sampler`, `monitor`, `strato_http`). The plan graph is readable in this checkout; the service does not run from the published tree alone.
## Architecture
```mermaid
flowchart TB
subgraph generators [Stream generators]
SG["StreamTaskGenerator.ELIGIBILITIES_TO_INJECT"]
end
subgraph runtime [Dispatcher and Engine]
TQ["task_queue: TaskPayload"]
EN["Engine._process_task"]
RQ["resp_queue: TaskResult"]
end
subgraph master [PlanMaster]
ALL["asyncio.gather ALL_PLANS"]
MR["merge_results"]
end
subgraph planLayer [Plan]
EL["_eligible(REQUIRED_ELIGIBILITY)"]
DAG["TASK_DEPENDENCIES futures"]
end
subgraph taskLayer [Task]
SK["should_skip / DISABLE_RULES"]
EX["_exec → TaskContext"]
end
SG --> TQ --> EN --> ALL
ALL --> EL
EL -->|ineligible: None| MR
EL -->|eligible| DAG --> SK --> EX
EX --> MR --> RQ
```
`Dispatcher` writes `TaskPayload`s. `Engine._process_task` is the only caller of `PlanMaster.exec`. `Plan.execute` always returns a `TaskResult` when eligible; ineligible plans return `None` and are dropped before merge.
## TaskEligibility
`TaskEligibility` is a `str` enum on `TaskPayload.eligibilities`. A plan runs iff `REQUIRED_ELIGIBILITY in payload.eligibilities` — membership, not equality. One payload can activate multiple plans.
| Member | Value | Plan |
|---|---|---|
| `SPAM_COMMENT` | `spam_comment` | `PlanSpamComment` |
| `BANGER_INITIAL_SCREEN` | `banger_initial_screen` | `PlanInitialBanger` |
| `POST_EMBEDDING_WITH_SUMMARY` | `post_embedding_with_summary` | `PlanPostEmbeddingWithSummary` |
| `POST_EMBEDDING_WITH_SUMMARY_FOR_REPLY` | `post_embedding_with_summary_for_reply` | `PlanPostEmbeddingWithSummaryForReply` |
| `MM_EMB_V5` | `mm_emb_v5` | `PlanPostEmbeddingV5` |
| `MM_EMB_V5_FOR_REPLY` | `mm_emb_v5_for_reply` | `PlanPostEmbeddingV5ForReply` |
| `REPLY_RANKING` | `reply_ranking` | `PlanReplyRanking` |
| `SAFETY_PTOS` | `safety_ptos` | `PlanSafetyPtos` |
| `POST_SAFETY` | `post_safety` | `PlanPostSafety` |
| `MM_EMB_V4` | `mm_emb_v4` | **No plan.** `TaskMultimodalPostEmbeddingRecsysV4` exists but is not registered in any `TASKS` map. |
### Who injects which eligibility
`StreamTaskGenerator._poll` copies `ELIGIBILITIES_TO_INJECT` onto every payload.
| Generator | `TaskGeneratorType` | Injected eligibilities |
|---|---|---|
| `PostStreamTaskGenerator` | `POST_STREAM` | `SPAM_COMMENT` **and** `REPLY_RANKING` |
| `MinTractionPostStreamForGroxTaskGenerator` | `POST_MIN_TRACTION_STREAM_FOR_GROX` | `BANGER_INITIAL_SCREEN` |
| `PostStreamRecoveryTaskGenerator` | `POST_STREAM_RECOVERY` | `BANGER_INITIAL_SCREEN` |
| `PostStreamTestTaskGenerator` | `POST_STREAM_TEST` | `BANGER_INITIAL_SCREEN` |
| `PostSafetyStreamTaskGenerator` | `POST_SAFETY_STREAM` | `POST_SAFETY` |
| `PlanSpamComment` / reply-ranking recovery | `REPLY_RANKING_RECOVERY` | `REPLY_RANKING` |
| `PostEmbeddingRequestWithSummaryStreamTaskGenerator` | `POST_EMBEDDING_REQUEST_STREAM_WITH_SUMMARY` | `POST_EMBEDDING_WITH_SUMMARY` |
| `PostEmbeddingRequestWithSummaryRecoveryStreamTaskGenerator` | `POST_EMBEDDING_REQUEST_STREAM_WITH_SUMMARY_RECOVERY` | `POST_EMBEDDING_WITH_SUMMARY` |
| `MinTractionPostStreamForGroxMultiModalTaskGenerator` | `POST_MIN_TRACTION_STREAM_FOR_GROX_MULTI_MODAL` | `POST_EMBEDDING_WITH_SUMMARY_FOR_REPLY` |
| `PostEmbeddingRequestWithSummaryForReplyRecoveryStreamTaskGenerator` | `POST_EMBEDDING_REQUEST_STREAM_WITH_SUMMARY_FOR_REPLY_RECOVERY` | `POST_EMBEDDING_WITH_SUMMARY_FOR_REPLY` |
| `PostEmbeddingV5StreamTaskGenerator` | `POST_EMBEDDING_V5_STREAM` | `MM_EMB_V5` |
| `PostEmbeddingV5ForReplyStreamTaskGenerator` | `POST_EMBEDDING_V5_FOR_REPLY_STREAM` | `MM_EMB_V5_FOR_REPLY` |
| `MinTractionPostStreamForGroxPtosTaskGenerator` | `POST_MIN_TRACTION_STREAM_FOR_GROX_PTOS` | `SAFETY_PTOS` |
| `SafetyPtosRecoveryStreamTaskGenerator` | `SAFETY_PTOS_RECOVERY` | `SAFETY_PTOS` |
| `SafetyPtosDeluxeStreamTaskGenerator` | `SAFETY_PTOS_DELUXE` | `SAFETY_PTOS` |
| `PostStreamDelayedTaskGenerator` | `POST_STREAM_DELAYED` | `{}` — no plan is eligible |
`POST_STREAM` is the only generator that injects two eligibilities. `PlanMaster` then runs `PlanSpamComment` and `PlanReplyRanking` concurrently on the same payload. Deluxe PTOS reuses `SAFETY_PTOS`; deluxe vs standard is selected later by `payload.task_type == TaskGeneratorType.SAFETY_PTOS_DELUXE`.
## PlanMaster.ALL_PLANS
Registration order in `grox/plans/plan_master.py` is also merge order for the first non-`None` embedding:
1. `PlanInitialBanger()`
2. `PlanPostSafety()`
3. `PlanSpamComment()`
4. `PlanPostEmbeddingWithSummary()`
5. `PlanPostEmbeddingWithSummaryForReply()`
6. `PlanPostEmbeddingV5()`
7. `PlanPostEmbeddingV5ForReply()`
8. `PlanReplyRanking()`
9. `PlanSafetyPtos()`
```python
results = await asyncio.gather(*[p.execute(task) for p in cls.ALL_PLANS])
result = cls.merge_results(task, [r for r in results if r is not None])
```
`Plan.get_name()` is `camel_to_snake(class_name)` and is the `plan_name` metric attribute (`plan_initial_banger`, `plan_post_safety`, …). Construction fails if any `TASK_DEPENDENCIES` key or edge is missing from `TASKS`.
## DAG execution
`Plan.execute` creates one `asyncio.Future` per task that appears as a dependency (`self.deps`). Every `TASKS` key is scheduled concurrently; a task `await`s its upstream futures, then calls `task.exec(ctx)`.
| Upstream result | Downstream |
|---|---|
| Any dep returns `TaskResultCategory.SKIPPED` | Downstream is marked `SKIPPED` and does **not** run |
| Any dep raises | Future is set with the exception; the plan `gather` fails |
| All deps `SUCCESS` (or no deps) | `task.exec(ctx)` runs |
Plan-level outcome:
- Exceptions are caught, appended to `ctx.errors`, and do **not** escape `Plan.execute`.
- Remaining incomplete futures are cancelled in `finally`.
- Returned `TaskResult.success` is `len(ctx.errors) == 0`.
- Metrics: `plan.execute.count`, `.success.count`, `.failed.count`, `plan.execute.duration`.
```text
filter ──► rate_limit ──► media ──► classify/embed ──┬──► sink / Manhattan
└──► Kafka / UPA
```
Most plans follow that spine. A skipped filter or rate-limit (`TaskStopExecution`) short-circuits the rest of the DAG without failing the plan.
## Plan DAG catalog
| Task key | Class |
|---|---|
| `task_initial_banger_filter` | `TaskInitialBangerFilter` |
| `task_banger_annotation_rate_limit` | `TaskRateLimitBangerAnnotationWithPost` |
| `task_media_hydration` | `TaskMediaHydrationBanger` |
| `task_banger_screen_initial` | `TaskBangerScreen` |
| `task_grok_upa_action_with_labels` | `TaskGrokUpaActionWithLabels` |
| `task_publish_unified_post_annotations_manhattan` | `TaskPublishUnifiedPostAnnotationsManhattan` |
| `task_publish_kafka` | `TaskPublishKafka` |
```text
task_initial_banger_filter
→ task_banger_annotation_rate_limit
→ task_media_hydration
→ task_banger_screen_initial
├─→ task_grok_upa_action_with_labels
└─→ task_publish_unified_post_annotations_manhattan
→ task_publish_kafka
```
Filter drops replies and protected authors. Screen writes `ContentCategoryType.BANGER_INITIAL_SCREEN` and `GROK_RANKER` (positive when `quality_score >= 0.4`).
| Task key | Class |
|---|---|
| `task_post_safety_deluxe_filter` | `TaskPostSafetyDeluxeFilter` |
| `task_post_safety_annotation_rate_limit` | `TaskRateLimitPostSafetyAnnotationWithPost` |
| `task_media_hydration` | `TaskMediaHydrationBanger` |
| `task_post_safety_screen_deluxe` | `TaskPostSafetyScreenDeluxe` |
| `task_grok_upa_action_with_labels` | `TaskGrokUpaActionWithLabels` |
| `task_upsert_tweet_bool_metadata_to_unified_post_annotations_manhattan` | `TaskUpsertTweetBoolMetadataToUnifiedPostAnnotation` |
```text
task_post_safety_deluxe_filter
→ task_post_safety_annotation_rate_limit
→ task_media_hydration
→ task_post_safety_screen_deluxe
├─→ task_grok_upa_action_with_labels
└─→ task_upsert_tweet_bool_metadata_to_unified_post_annotations_manhattan
```
Filter drops replies and protected authors. Classifier always emits `POST_SAFETY_SCREEN` with `positive=False`, `score=0.0`; labels live on `tweet_bool_metadata`.
| Task key | Class |
|---|---|
| `task_spam_filter` | `TaskSpamFilter` |
| `task_reply_spam_annotation_rate_limit` | `TaskRateLimitReplySpamAnnotationWithPost` |
| `task_media_hydration` | `TaskMediaHydration` |
| `task_spam_detection` | `TaskSpamDetection` |
| `task_publish_reply_spam_mh` | `TaskWriteReplySpamManhattan` |
| `task_publish_kafka` | `TaskPublishKafka` |
```text
task_spam_filter
→ task_reply_spam_annotation_rate_limit
→ task_media_hydration
→ task_spam_detection
├─→ task_publish_reply_spam_mh
└─→ task_publish_kafka
```
Filter requires a reply with a distinct author from both the parent and the root. Detection writes `ContentCategoryType.SPAM_COMMENT`.
| Task key | Class |
|---|---|
| `task_reply_ranking_filter` | `TaskReplyRankingFilter` |
| `task_reply_ranking_annotation_rate_limit` | `TaskRateLimitReplyRankingAnnotationWithPost` |
| `task_media_hydration` | `TaskMediaHydration` |
| `task_rank_replies` | `TaskRankReplies` |
| `task_write_reply_ranking_manhattan` | `TaskWriteReplyRankingManhattan` |
```text
task_reply_ranking_filter
→ task_reply_ranking_annotation_rate_limit
→ task_media_hydration
→ task_rank_replies
→ task_write_reply_ranking_manhattan
```
Scores go to `ctx.reply_ranking_results`, not `content_categories`. `ReplyScorer` is not a `ContentClassifier`.
| Task key | Class |
|---|---|
| `task_safety_ptos_filter` | `TaskSafetyPtosFilter` |
| `task_safety_ptos_annotation_rate_limit` | `TaskRateLimitSafetyPtosAnnotationWithPost` |
| `task_media_hydration` | `TaskMediaHydration` |
| `task_safety_ptos_category_detection` | `TaskSafetyPtosCategoryDetection` |
| `task_safety_ptos_policy_detection` | `TaskSafetyPtosPolicyDetection` |
| `task_write_safety_post_annotations_result_sink` | `TaskWriteSafetyPostAnnotationsResultSink` |
```text
task_safety_ptos_filter
→ task_safety_ptos_annotation_rate_limit
→ task_media_hydration
→ task_safety_ptos_category_detection
→ task_safety_ptos_policy_detection
→ task_write_safety_post_annotations_result_sink
```
Category detection writes `ctx.safety_annotations`. Policy detection fills `violatedPolicies[].safetyPolicy`. Deluxe (`task_type == SAFETY_PTOS_DELUXE`) uses `VLM_PRIMARY_CRITICAL` plus an AdultContent recheck injected at score `50` and removed if the policy comes back `NoViolation`. The filter root uses `{}` (empty dict) rather than `set()`; iteration still yields no deps.
**`PlanPostEmbeddingWithSummary`** (`POST_EMBEDDING_WITH_SUMMARY`) — originals only:
```text
task_post_embedding_rate_limit_summary
→ task_post_embedding_with_summary_filter
→ task_media_hydration
→ task_post_embedding_summarizer
→ task_multimodal_post_embedding_with_summary # dict["v3"]
→ task_write_post_embedding_sink_v3
```
**`PlanPostEmbeddingWithSummaryForReply`** (`POST_EMBEDDING_WITH_SUMMARY_FOR_REPLY`) — same spine, reply-only filter and a distinct rate-limit cache.
**`PlanPostEmbeddingV5`** (`MM_EMB_V5`) — no content filter:
```text
task_post_embedding_rate_limit
→ task_media_hydration
→ task_asr_transcription
→ task_multimodal_post_embedding_v5 # dict["v5_1"]
→ task_write_post_embedding_sink_v5 # TaskWriteMMEmbeddingSinkV5SkipKafkaForReplies
```
**`PlanPostEmbeddingV5ForReply`** (`MM_EMB_V5_FOR_REPLY`) inserts `TaskPostEmbeddingWithSummaryForReplyFilter` after the v5-for-reply rate limit, then the same ASR → embed → `TaskWriteMMEmbeddingSinkV5` chain.
## Task contract
`Task.exec` is a classmethod with `@retry(stop=stop_after_attempt(2), wait=wait_fixed(1))`. Several classifier/sink tasks override `exec` with `Task.exec.__wrapped__` and skip that retry.
Empty by default. First matching `rule.should_disable(ctx)` returns `SKIPPED` before `_exec`.
| Rule | Skips when |
|---|---|
| `DisableTaskForLocal` | `is_local` |
| `DisableTaskForDev` | `is_dev` |
| `DisableTaskForNonProd` | not `is_prod` |
| `DisableTaskForNonMmEmbProd` | not `is_mm_emb_prod` (embedding sinks / Kafka pub) |
| `DisableTaskForNonPtosProd` | not `is_ptos_prod` (PTOS sink) |
Return / exception map from `Task.exec`:
| Signal | `TaskResultCategory` | Plan DAG |
|---|---|---|
| `should_skip` or `TaskStopExecution` | `SKIPPED` | dependents skip |
| `_exec` completes | `SUCCESS` | dependents run |
| any other exception | re-raised | plan records `ctx.errors` |
Typed wrappers stop with `TaskStopExecution` when the payload is missing the required object:
- `TaskWithPost` — `payload.post`
- `TaskWithUser` — `payload.user`
- `TaskWithUserContext` — `payload.user_context`
- `TaskWithContentAnalysis` — `payload.grox_content_analysis`
`TaskFilter` / `TaskRateLimit` also raise `TaskStopExecution` when `_eligible` is false. Rate-limit caches are process-local `TTLCache(maxsize=10_000, ttl=60)` keyed by post id.
## TaskContext vs TaskResult
`TaskContext` is the per-plan scratch pad. Only a subset is copied onto `TaskResult`.
Inbound payload. Copied onto `TaskResult.task`.
Copied onto the plan result, then concatenated across plans at merge.
Copied onto `TaskResult`. Embedding tasks write `multimodal_post_embedding_dict` (`"v3"`, `"v4"`, `"v5_1"`) instead, so this field stays `None` unless something else sets it.
Copied onto `TaskResult.reason`.
`success = len(errors) == 0`; `error` is `"\\n".join(str(e) for e in errors)`.
PTOS only. Consumed by the PTOS sink inside the plan. **Not** a `TaskResult` field.
Reply ranking only. Consumed by the Manhattan write task. **Not** a `TaskResult` field.
Used by Kafka publish (`TaskPublishKafka`). **Not** a `TaskResult` field.
## ContentClassifier
`ContentClassifier.classify(post)` increments request/intake metrics, calls `_classify`, then records success/error and latency. `_classify` is `_to_convo` → `_sample` → `_parse`.
| Classifier | Categories | Sampler | Parse |
|---|---|---|---|
| `BangerInitialScreenClassifier` | `BANGER_INITIAL_SCREEN`, `GROK_RANKER` | `VisionSampler` / `VLM_PRIMARY`, temperature `0.000001` | `…` → `BangerInitialScreenResult`; `positive = quality_score >= 0.4` |
| `PostSafetyDeluxeClassifier` | `POST_SAFETY_SCREEN` | `VLM_PRIMARY_CRITICAL` | `…` → `tweet_bool_metadata`; `positive=False` |
| `SpamEapiLowFollowerClassifier` | `SPAM_COMMENT` | `VLM_PRIMARY` (overrideable) | JSON `SpamSampleResult`; keeps only `SPAM_COMMENT` and asserts exactly one |
| `SafetyPtosCategoryClassifier` | `SAFETY_PTOS` | `VLM_SAFETY` or deluxe `VLM_PRIMARY_CRITICAL` | Tasks call `classify_post` → `SafetyPostAnnotations`, not `classify` |
| `SafetyPtosPolicyClassifier` | `SAFETY_PTOS` | `VLM_PRIMARY_CRITICAL` (+ EAPI samplers when deluxe) | Per-violation policy prompts: ViolentMedia, AdultContent, Spam, IllegalAndRegulatedBehaviors, HateOrAbuse, ViolentSpeech, SuicideOrSelfHarm |
`ReplyScorer` (`TaskRankReplies`) uses the same VLM stack and `ReplyScoringSystem` prompt but does not subclass `ContentClassifier`.
Banger `classify` optionally takes cached Grok topics from `StratoGrokTopics` (`CACHE_TTL_SECONDS = 3600` on `TaskBangerScreen`).
## TaskResult merge rules
`PlanMaster.merge_results(task, results)` assumes `results` is the non-`None` subset.
| Field | Rule |
|---|---|
| `task` | Original inbound `TaskPayload` |
| `content_categories` | Flatten `r.content_categories` across plans, each `c.model_copy()` |
| `task_started_at` | `min(r.task_started_at)` |
| `task_finished_at` | `max(r.task_finished_at)` |
| `multimodal_post_embedding` | First non-`None` in `ALL_PLANS` order; else `None` |
| `reason` | `"\\n".join` of non-empty `r.reason` |
| `success` | `all(r.success for r in results)` |
| `error` | `"\\n".join` of `r.error or "unknown error"` for every `not r.success` |
If every plan returns `None` (no matching eligibility — including `POST_STREAM_DELAYED`), `merge_results` calls `min`/`max` on an empty sequence and raises. `Engine._run_task` catches that, puts `TaskResult(success=False, error=str(e))`, and increments `engine.task.failed.count`.
Engine failure envelope when `PlanMaster.exec` itself raises:
```python
TaskResult(
task=task,
success=False,
error=str(e),
task_finished_at=start, # perf_counter at entry
task_started_at=time.perf_counter(), # after the exception
)
```
Dispatcher retry: on `result.success is False` and `task.attempt < dispatcher.max_attempts`, `attempt` is incremented and the same payload is re-queued. Final failure acks the originating generator when `identify_task_origin` resolves.
## Payload fields
Generator message id (`payload.mid`). Dispatcher inflight set key.
Default empty. Copied onto `TaskContext.eligibilities`.
Default `0`. Incremented by Dispatcher on retry.
Selects deluxe PTOS classifiers/caches when `SAFETY_PTOS_DELUXE`.
Defined in unpublished `grox.data_loaders.data_types`. Missing objects skip typed tasks via `TaskStopExecution`.
Copied from the stream loader; plans do not gate on it.
## Constraints
- Adding a plan requires an `ALL_PLANS` entry, a unique `REQUIRED_ELIGIBILITY`, and a generator that injects that eligibility.
- `TASK_DEPENDENCIES` must be a closed graph over `TASKS` keys or `Plan.__init__` raises `ValueError`.
- Skip is contagious along outgoing edges; a failed task fails the plan but does not skip siblings already running.
- Intra-plan side effects (Manhattan, Kafka, Strato sinks) read `TaskContext`. Cross-plan merge only carries categories, timestamps, the unused embedding scalar, reason, and success/error.
- `TaskPublishKafka` is disabled in local and dev; UPA / reply-spam / reply-ranking Manhattan writes are disabled outside prod; embedding sinks require `is_mm_emb_prod`; PTOS sink requires `is_ptos_prod`.
## Related pages
Engine, Dispatcher, and GrpcServer startup, fill/result loops, and ack/retry.
What this checkout can execute versus unpublished Grox crates and production snapshots.
Unpublished Grox modules and other local-run failure modes.
---
## 24. Test Phoenix
> uv run pytest targets, attention-mask and retrieval assertions, and the success criteria encoded in the Phoenix test modules.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/24-test-phoenix.md
- Generated: 2026-08-13T19:50:54.647Z
### Source Files
- `phoenix/test_recsys_model.py`
- `phoenix/test_recsys_retrieval_model.py`
- `phoenix/pyproject.toml`
- `phoenix/README.md`
- `phoenix/grok.py`
- `phoenix/recsys_model.py`
---
title: "Test Phoenix"
description: "uv run pytest targets, attention-mask and retrieval assertions, and the success criteria encoded in the Phoenix test modules."
---
Phoenix unit tests live in `phoenix/test_recsys_model.py` and `phoenix/test_recsys_retrieval_model.py`. From `phoenix/`, the documented target is `uv run pytest` on those two files. The suite asserts `make_recsys_attn_mask` candidate isolation, `right_anchored_rope_positions`, `compute_post_age_bucket`, `normalize_continuous_value`, `CandidateTower` L2 outputs, and synthetic retrieval `RetrievalOutput` shapes. It initializes Haiku params on random `create_example_batch` / `create_example_corpus` tensors and does **not** load `oss-phoenix-artifacts`.
These tests are not the inference-pipeline success signal. A ranked table from `run_pipeline.py` requires extracted checkpoints. A green pytest run only means the JAX helpers and a randomly initialized retrieval model still satisfy the contracts below.
## Prerequisites
`>=3.11` from `phoenix/pyproject.toml`.
`dm-haiku>=0.0.13`, `jax==0.8.1`, `numpy>=1.26.4`. `pyright` is also a project dependency; the test modules do not invoke it.
`pytest` is listed under `[dependency-groups] dev`, not under `[project] dependencies`.
`[tool.uv] environments` is restricted to `sys_platform == 'darwin'` and `sys_platform == 'linux'`.
There is no `pytest.ini` and no `[tool.pytest]` table. There is no CI workflow in this checkout that runs these files. Modules are flat (`from grok import ...`, `from recsys_model import ...`), so the working directory must be `phoenix/` or that directory must be on `PYTHONPATH`.
:::files
phoenix/
test_recsys_model.py pytest-style classes: mask, RoPE, age, norms
test_recsys_retrieval_model.py unittest.TestCase: tower, model, runner
grok.py make_recsys_attn_mask, right_anchored_rope_positions
recsys_model.py compute_post_age_bucket, NormConfig, HashConfig
recsys_retrieval_model.py CandidateTower, PhoenixRetrievalModel, RetrievalOutput
runners.py create_example_batch, create_example_corpus, RecsysRetrievalInferenceRunner
pyproject.toml pytest in [dependency-groups] dev
:::
## Run the suite
```bash
cd phoenix
uv sync
```
`uv sync` installs the project plus the default `dev` group, which provides `pytest`.
```bash
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```
That is the command in `phoenix/README.md`. From the repo root, prefix the path: `uv run --directory phoenix pytest test_recsys_model.py test_recsys_retrieval_model.py` only works if imports still resolve; prefer `cd phoenix`.
Success is pytest exit code `0`. The two files encode **22** pytest methods in `test_recsys_model.py` and **12** `unittest.TestCase` methods in `test_recsys_retrieval_model.py`. Failures are assertion messages such as `Position i should NOT attend to future position j` or a shape mismatch on `user_representation`.
```bash title="Documented pytest target"
cd phoenix
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```
```bash title="Verbose / one class"
cd phoenix
uv run pytest test_recsys_model.py::TestMakeRecsysAttnMask -v
uv run pytest test_recsys_retrieval_model.py::TestPhoenixRetrievalModel::test_retrieve_top_k -v
```
```bash title="File __main__ runners"
cd phoenix
uv run python test_recsys_model.py
uv run python test_recsys_retrieval_model.py
```
`test_recsys_model.py` ends with `pytest.main([__file__, "-v"])`. `test_recsys_retrieval_model.py` ends with `unittest.main()`. pytest collects both styles when you pass the files.
```bash
cd phoenix && uv run pytest test_recsys_model.py test_recsys_retrieval_model.py -q
```
```text
# Expected: process exit 0, every collected method in the two files passed.
# Not expected: artifact-path errors, sports_corpus.npz loads, or a ranked feed table.
```
## What the suite covers
| File | Style | Classes | Surface under test |
|---|---|---|---|
| `test_recsys_model.py` | pytest classes | `TestMakeRecsysAttnMask`, `TestRightAnchoredRopePositions`, `TestComputePostAgeBucket`, `TestNormalizeContinuousValue` | Ranking-path helpers in `grok.py` and `recsys_model.py` |
| `test_recsys_retrieval_model.py` | `unittest.TestCase` | `TestCandidateTower`, `TestPhoenixRetrievalModel`, `TestRetrievalInferenceRunner` | `CandidateTower`, `PhoenixRetrievalModelConfig.make()`, `RecsysRetrievalInferenceRunner` |
The retrieval user tower concatenates user + history and calls the transformer with `candidate_start_offset=None`. Isolation-mask tests therefore apply to ranking (`make_recsys_attn_mask`), not to retrieval encoding.
## Attention-mask success criteria
`make_recsys_attn_mask(seq_len, candidate_start_offset, dtype=jnp.float32)` returns a `[1, 1, seq_len, seq_len]` array. `1` means attend; `0` means do not. Implementation starts from `jnp.tril`, zeros the candidate–candidate block, then restores the candidate diagonal.
Total length: user prefix + history + candidates.
First candidate index. Positions `[0, candidate_start_offset)` are user + history.
Default `jnp.float32`. `test_dtype_preserved` also constructs `jnp.float16` and asserts the dtype is kept.
`TestMakeRecsysAttnMask` locks these rules:
| Method | Pass condition |
|---|---|
| `test_output_shape` | `mask.shape == (1, 1, seq_len, seq_len)` |
| `test_user_history_has_causal_attention` | For `i, j < candidate_start_offset`: `1` iff `j <= i` |
| `test_candidates_attend_to_user_history` | Every candidate row is `1` on all user/history keys |
| `test_candidates_attend_to_themselves` | Candidate diagonal is `1` |
| `test_candidates_do_not_attend_to_other_candidates` | Off-diagonal candidate–candidate cells are `0` |
| `test_full_mask_structure` | Exact matrix for `[user, h1, h2, c1, c2, c3]` |
| `test_single_candidate` | Causal prefix plus one self-attending candidate |
| `test_all_candidates` | Offset `1`: only position `0` is the prefix; each later token attends to user + self |
Canonical fixture (`seq_len=6`, `candidate_start_offset=3`):
```text
# keys →
# u h1 h2 c1 c2 c3
user [ 1, 0, 0, 0, 0, 0 ]
h1 [ 1, 1, 0, 0, 0, 0 ]
h2 [ 1, 1, 1, 0, 0, 0 ]
c1 [ 1, 1, 1, 1, 0, 0 ]
c2 [ 1, 1, 1, 0, 1, 0 ]
c3 [ 1, 1, 1, 0, 0, 1 ]
```
User + history attention in this function is **causal** (`jnp.tril`), not bidirectional. `test_user_history_has_causal_attention` fails if a history token attends to a later history token. That is the contract to keep when editing `make_recsys_attn_mask`.
The all-candidates edge (`candidate_start_offset=1`) is the isolation rule with no history: each candidate attends to the user token and itself only.
## RoPE, age buckets, and continuous norms
### `right_anchored_rope_positions`
Signature: `(padding_mask, history_seq_len, num_user_prefix_tokens) -> [B, T]`.
| Method | Pass condition |
|---|---|
| `test_output_shape` | Shape `[B, T]` |
| `test_prefix_positions_preserved` | Prefix slots are `0 .. num_user_prefix_tokens-1` |
| `test_candidates_share_position` | Every candidate index equals `history_end = num_prefix + history_seq_len` |
| `test_padding_gets_zero` | `padding_mask == False` positions are `0.0` |
Newest history is right-anchored so the last valid history token keeps a fixed RoPE index. Tokens after `history_end` (candidates) share that `history_end` index.
### `compute_post_age_bucket`
`POST_AGE_MAX_MINUTES = 4800`. Default `granularity_mins=60`. Formula: `bucket = (age_minutes // granularity_mins) + 1`, clipped to the overflow bucket `4800 // granularity_mins + 1`, then forced to `0` when age is negative or either timestamp is `0`.
| Method | Input (minutes old, 60-min buckets) | Expected bucket |
|---|---|---|
| `test_basic_bucketing` | 30 | `1` (0–59) |
| `test_two_hour_post` | 120 | `3` (120–179) |
| `test_missing_timestamp_zero` | `impr_ts == 0` or `post_ts == 0` | `0` |
| `test_negative_age_maps_to_zero` | post created after impression | `0` |
| `test_overflow_bucket` | 5000 | `81` (`4800 // 60 + 1`) |
| `test_batch_processing` | `[30, 120, missing]` | `[1, 3, 0]`, shape `(1, 3)` |
### `normalize_continuous_value`
`NormConfig(norm_scale=30.0, use_log=False)` by default. Values clip to `[0, norm_scale]` first.
| Method | Config | Input | Expected |
|---|---|---|---|
| `test_linear_normalization` | `norm_scale=30`, `use_log=False` | `[0, 15, 30, 60]` | `[0.0, 0.5, 1.0, 1.0]` |
| `test_log_normalization` | `norm_scale=30`, `use_log=True` | `[0, 30]` | `[0.0, 1.0]` (`log1p(x) / log1p(scale)`) |
| `test_clamping` | `norm_scale=10`, linear | `[-5, 0, 5, 15]` | `[0.0, 0.0, 0.5, 1.0]` |
Linear path: `clip(x, 0, scale) / scale`. Log path: `log1p(clip(x, 0, scale)) / log1p(scale)`.
## Retrieval success criteria
Retrieval tests construct a **tiny** `PhoenixRetrievalModelConfig` (not the published mini checkpoint): `emb_size=64`, `history_seq_len=16`, `candidate_seq_len=8`, `batch_size=2`, `num_actions=19`, `corpus_size=100`, `top_k=10`, `HashConfig(num_user_hashes=2, num_item_hashes=2, num_author_hashes=2)`, `product_surface_vocab_size=16`, and a 1-layer `TransformerConfig` (`widening_factor=2`, `key_size=32`, `num_q_heads=2`, `num_kv_heads=2`, `attn_output_multiplier=0.125`). Forward graphs use `hk.without_apply_rng(hk.transform(...))` and `jax.random.PRNGKey(0)`.
### `CandidateTower`
Input shape in the tests: `[B, C, num_hashes, D]` with `B=4`, `C=8`, `num_hashes=4`, `D=64`.
| Method | `enable_linear_proj` | Pass condition |
|---|---|---|
| `test_candidate_tower_output_shape` | `True` | Output `[B, C, D]` |
| `test_candidate_tower_normalized` | `True` | Per-vector L2 norm `≈ 1` (`decimal=5`) |
| `test_candidate_tower_mean_pooling` | `False` | Same shape and unit L2 |
| `test_mean_pooling_has_no_params` | `False` | `sum(p.size for p in jax.tree.leaves(params)) == 0` |
Linear mode is a two-layer SiLU MLP (`candidate_tower_projection_1`, `candidate_tower_projection_2`) then L2. Mean-pool mode averages on the hash axis, then L2.
### `PhoenixRetrievalModel`
`__call__(batch, embeddings, corpus_embeddings, top_k)` returns `RetrievalOutput`.
`[B, D]`. `test_user_representation_normalized` requires unit L2 per row.
`[B, K]`. Every index in `[0, corpus_size)`.
`[B, K]`. Per-row scores are nonincreasing (`scores[i] >= scores[i+1]`).
| Method | Pass condition |
|---|---|
| `test_model_forward` | The three `RetrievalOutput` shapes above |
| `test_candidate_representation_normalized` | `build_candidate_representation` → `[B, C, D]` unit L2 |
| `test_retrieve_top_k` | In-range indices and descending scores |
| `test_mean_pooling_model_forward` | Same user / top-k shapes with `enable_linear_proj=False` |
Scoring is `user_representation @ corpus_embeddings.T` then `jax.lax.top_k`. Corpus rows from `create_example_corpus` are already L2-normalized.
### `RecsysRetrievalInferenceRunner`
`initialize()` builds dummy batch/embeddings from the config, a dummy corpus of shape `(10, emb_size)`, and `dummy_top_k=5`, then stores `runner.params`. Tests pass `RetrievalModelRunner(..., bs_per_device=0.125)` (`BaseModelRunner` default is `2.0`).
| Method | Pass condition |
|---|---|
| `test_runner_initialization` | `runner.params is not None` after `initialize()` |
| `test_runner_encode_user` | `encode_user` → `[B, D]` |
| `test_runner_retrieve` | After `set_corpus(corpus_embeddings, corpus_post_ids)`, `retrieve(..., top_k=10)` matches `RetrievalOutput` shapes |
## Synthetic fixtures
`create_example_batch` (seed `42`) and `create_example_corpus` (seed `123`) are the only data sources. Hash `0` is reserved for padding; generated hashes are in `[1, num_*_embeddings)`. History rows are randomly truncated to `[history_len // 2, history_len]`. Actions are Bernoulli with `p=0.3` (`> 0.7` kept). Corpus embeddings are Gaussian then L2-normalized; `corpus_post_ids` is `arange(corpus_size)`.
| Knob | Test value | Notes |
|---|---|---|
| `emb_size` | `64` | Not the published mini-model width |
| `history_seq_len` | `16` | Published mini config uses `127` |
| `candidate_seq_len` | `8` | Published mini config uses `64` |
| `num_actions` | `19` | Matches `BaseInferenceRunner._get_num_actions` fallback |
| `corpus_size` | `100` | Not `sports_corpus.npz` |
| `top_k` | `10` | Not `--top_k_retrieval` |
| Transformer | 1 layer, 2 heads | Not the exported ranker/retrieval depth |
Changing isolation, RoPE candidate sharing, overflow bucket `81`, unit-norm towers, or descending `top_k` without updating these assertions is a failed contract, not a style change.
## What these tests do not cover
| Surface | Status |
|---|---|
| `run_pipeline.py` / `run_ranker.py` / `run_retrieval.py` | Not imported |
| `oss-phoenix-artifacts` (`model_params.npz`, `embedding_tables.npz`, `config.json`, `sports_corpus.npz`, `example_sequence.json`) | Not loaded |
| Ranking `PhoenixModelConfig` forward / multi-action logits | No test module |
| Hash-table lookup against the 1M-row published tables | Random embeddings only |
| Home Mixer, Thunder, Grox | Out of this directory |
| pyright | Dependency only |
A green unit suite plus a failed `run_pipeline.py` is consistent: tests never touch LFS artifacts.
## Troubleshooting
Run from `phoenix/`. The tree is not an installed package; pytest adds the current directory to `sys.path`.
`pytest` is a `dev` group extra. Re-run `uv sync` in `phoenix/`, then `uv run pytest ...` (not a bare `pytest` from a non-project venv).
`jax` is pinned to `0.8.1`. uv environments list only Darwin and Linux. Windows is not an advertised target.
Those errors come from the inference CLI or Home Mixer, not these tests. See [Troubleshooting](/troubleshooting) and [Run the inference pipeline](/run-inference-pipeline).
## Next
Mask rules these tests freeze: candidates attend to user, history, and self only.
`PhoenixRetrievalModelConfig`, `TransformerConfig`, `RecsysBatch`, and published `config.json` keys versus the tiny test config.
Checkpoint load, `sports_corpus.npz` retrieval, and the ranked table — a different success signal.
Python 3.11+, uv, and Git LFS for artifacts (needed for inference, not for pytest).
LFS, artifact paths, `viewer_id`, and other runtime failures outside the unit suite.
---
## 25. Troubleshooting
> LFS and artifact-path failures, viewer_id must be specified, TEST_USER_IDS empty feeds, Thunder RESOURCE_EXHAUSTED, missing retrieval_sequence, and unpublished Grox modules.
- Page Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/pages/25-troubleshooting.md
- Generated: 2026-08-13T19:52:23.478Z
### Source Files
- `phoenix/run_pipeline.py`
- `phoenix/README.md`
- `home-mixer/server.rs`
- `home-mixer/scored_posts_server.rs`
- `thunder/thunder_service.rs`
- `home-mixer/sources/phoenix_source.rs`
- `grox/main.py`
- `candidate-pipeline/scorer.rs`
---
title: "Troubleshooting"
description: "LFS and artifact-path failures, viewer_id must be specified, TEST_USER_IDS empty feeds, Thunder RESOURCE_EXHAUSTED, missing retrieval_sequence, and unpublished Grox modules."
---
Local Phoenix inference fails on a Git LFS pointer or a wrong `--artifacts_dir`. Home Mixer and Thunder fail later, on `viewer_id`, silent `TEST_USER_IDS` empties, `RESOURCE_EXHAUSTED`, or a missing `retrieval_sequence`. Grox and the Rust workspace do not start from this checkout: required packages and crates are unpublished.
This tree has no `Cargo.toml`, no `grox` install metadata, and no `home-mixer/params` or `thunder/args` modules. The only packaged, locally executable surface is `phoenix/` after the LFS archive is materialized. See [Runtime boundaries](/runtime-boundaries).
## Symptom index
| Symptom | Surface | Typical cause |
|---|---|---|
| `oss-phoenix-artifacts.zip` is ~135 bytes of ASCII (`version https://git-lfs.github.com/spec/v1`) | `phoenix/artifacts/` | Git LFS not installed or not pulled |
| `unzip: End-of-central-directory signature not found` | `unzip artifacts/oss-phoenix-artifacts.zip` | Unzipping the LFS pointer instead of the ~2.9 GB object |
| `FileNotFoundError: .../retrieval/config.json` (or `ranker/`, `sports_corpus.npz`, `example_sequence.json`) | `run_pipeline.py` | Archive not extracted, or `--artifacts_dir` still the default `./artifacts` |
| `INVALID_ARGUMENT: viewer_id must be specified` | `QueryBuilder.build` | `ScoredPostsQuery.viewer_id` is `0` or omitted |
| `INVALID_ARGUMENT: query must be specified` | `ForYouFeedService` | `ForYouFeedQuery.query` is missing |
| `200` with `scored_posts: []` or `items: []` and no pipeline work | `ScoredPostsServer` / `ForYouFeedServer` | `query.user_id` is in unpublished `params::TEST_USER_IDS` |
| `RESOURCE_EXHAUSTED: Server at capacity, please retry` | `ThunderServiceImpl.get_in_network_posts` | Request semaphore is full; `try_acquire` rejects immediately |
| `PhoenixSource: missing retrieval_sequence` (also `PhoenixMOESource`, `PhoenixTopicsSource`) | Phoenix candidate sources | `RetrievalSequenceQueryHydrator` failed or was skipped; `retrieval_sequence` stays `None` |
| `ThunderSource: no available channel` / `ThunderSource: ...` | `ThunderSource` | No Thunder channel, or the gRPC status (including `RESOURCE_EXHAUSTED`) is mapped into a source `Err` |
| `ModuleNotFoundError: grox.config` / `grox.service` / `monitor.logging` | `python grox/main.py` | Unpublished Grox packages |
```mermaid
flowchart TD
req["ScoredPostsQuery / ForYouFeedQuery"] --> vid{"viewer_id == 0?"}
vid -->|yes| e400["INVALID_ARGUMENT: viewer_id must be specified"]
vid -->|no| test{"user_id in TEST_USER_IDS?"}
test -->|yes| empty["OK empty scored_posts / items"]
test -->|no| hyd["RetrievalSequenceQueryHydrator"]
hyd -->|"UAS fetch fails"| none["retrieval_sequence stays None"]
hyd -->|ok| seq["sequence populated"]
none --> phx["Phoenix*Source: missing retrieval_sequence"]
phx --> drop["source Err dropped by fetch_candidates"]
seq --> src["ThunderSource + enabled Phoenix sources"]
src --> cap{"Thunder semaphore full?"}
cap -->|yes| rex["RESOURCE_EXHAUSTED: Server at capacity, please retry"]
rex --> drop
cap -->|no| posts["candidates collected"]
drop --> maybe["empty or Thunder-only feed"]
```
`CandidatePipeline.execute` does not fail the RPC when a hydrator or source returns `Err`. Failed query hydrators are skipped; failed sources are dropped via `flatten`. An empty feed can therefore be a success response.
## Phoenix artifacts and Git LFS
`phoenix/artifacts/oss-phoenix-artifacts.zip` is tracked by Git LFS (`.gitattributes` applies `filter=lfs` to `*.zip` and `*.npz`). A complete object is **2903518802** bytes (about 2.70 GiB). The published pointer is:
```text
version https://git-lfs.github.com/spec/v1
oid sha256:fbc6017d00588754e22e0c7eb2f786a008a74d309c03c8085fa2fad418a83dac
size 2903518802
```
`run_pipeline.py` opens these paths under `--artifacts_dir` with no extra existence check:
| Relative path | Role |
|---|---|
| `retrieval/config.json` | Retrieval hash + transformer config |
| `retrieval/model_params.npz` | Retrieval weights |
| `retrieval/embedding_tables.npz` | Retrieval hash embeddings |
| `ranker/config.json` | Ranker hash + transformer config |
| `ranker/model_params.npz` | Ranker weights |
| `ranker/embedding_tables.npz` | Ranker hash embeddings |
| `sports_corpus.npz` | Default corpus (`--corpus_file` override) |
| `example_sequence.json` | Default user sequence (`--sequence_file` override) |
Directory that must already contain `retrieval/` and `ranker/` as siblings, plus the corpus and sequence files unless you override those flags. The README extract layout is `artifacts/oss-phoenix-artifacts`, not `./artifacts`.
User action sequence JSON. Missing file raises `FileNotFoundError`.
Precomputed candidate representations. Missing file raises `FileNotFoundError`.
`file artifacts/oss-phoenix-artifacts.zip` prints `ASCII text` when LFS did not materialize the object. `unzip` then fails with `End-of-central-directory signature not found`.
Check size. A pointer is about 135 bytes. A real archive is 2903518802 bytes.
```bash
git lfs install
git lfs pull
```
Then re-check: `git lfs ls-files` should list `phoenix/artifacts/oss-phoenix-artifacts.zip`, and `file` should report a Zip archive, not ASCII.
```bash
cd phoenix
unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/
```
This creates `artifacts/oss-phoenix-artifacts/` with `retrieval/`, `ranker/`, `sports_corpus.npz`, and `example_sequence.json`.
`run_pipeline.py` defaults `--artifacts_dir` to `./artifacts`. The extract step writes `artifacts/oss-phoenix-artifacts/`. Running from `phoenix/` without the flag looks for `phoenix/artifacts/retrieval/config.json`, which does not exist.
```bash title="Correct — extracted layout"
cd phoenix
uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts
```
```bash title="Wrong — default path"
cd phoenix
uv run run_pipeline.py
# FileNotFoundError: artifacts/retrieval/config.json
```
If you point `--corpus_file` or a model NPZ at a leftover LFS pointer (ASCII starting with `version https://git-lfs.github.com/spec/v1`), `numpy.load` / zip extraction fails. Pull LFS, then use files from the extracted `oss-phoenix-artifacts/` tree, not the `.zip` path itself.
Success signal for the local pipeline is the ranked table (`PIPELINE RESULTS — User …`) printed by `run_pipeline.py`. Phoenix unit tests do not need the archive:
```bash
cd phoenix
uv sync
uv run pytest test_recsys_model.py test_recsys_retrieval_model.py
```
`run_ranker.py` and `run_retrieval.py` initialize random demo weights. They do not load `oss-phoenix-artifacts` and cannot diagnose LFS or extract-path failures.
## viewer_id must be specified
`QueryBuilder.build` rejects a proto `ScoredPostsQuery` whose `viewer_id` is `0` (protobuf default for an omitted `uint64`) before Gizmoduck, feature switches, or the pipeline run:
```text
INVALID_ARGUMENT: viewer_id must be specified
```
This gate is shared by:
| RPC | Extra request check |
|---|---|
| `ScoredPostsService.GetScoredPosts` | None beyond `viewer_id` |
| `ScoredPostsService.GetDebugScoredPosts` | Uses `DebugScoredPostsQuery.query` (default empty proto if unset) — still requires non-zero `viewer_id` |
| `ForYouFeedService.GetForYouFeed` | `ForYouFeedQuery.query` must be present or the status is `query must be specified` |
| `ForYouFeedService.GetForYouFeedUrt` | Same nested `query` requirement |
```text title="Rejected scored-posts request"
viewer_id: 0
# or viewer_id omitted
```
```text title="QueryBuilder.build"
INVALID_ARGUMENT: viewer_id must be specified
```
A non-zero `viewer_id` is copied onto `ScoredPostsQuery.user_id`. Gizmoduck `get_viewer_data` is then called with a **200 ms** timeout. Timeout or client error becomes `ViewerData::default()` and does **not** fail the request. `in_network_only` becomes true only when the proto flag is set or `allow_for_you_recommendations == Some(false)`.
Home Mixer process flags (`home-mixer/main.rs`) do not supply a viewer. They only bind the server:
gRPC listen port.Metrics port.Negative disables shard coordinates.Feature-switch custom string `datacenter`.
`XServiceBuilder` still needs unpublished `params::STRINGCENTER_BUNDLE_PATH`, `params::FS_PATH`, decider paths, and mTLS from the environment. A missing `viewer_id` is a request bug, not a process-flag bug.
## TEST_USER_IDS empty feeds
If `params::TEST_USER_IDS` contains `query.user_id`, both product servers return an empty success and skip `CandidatePipeline.execute`:
| Server | Return value |
|---|---|
| `ScoredPostsServer.run_pipeline` | `PipelineOutput { scored_posts: vec![], pipeline_result: PipelineResult::empty() }` |
| `ForYouFeedServer.get_for_you_feed` | `ForYouFeedOutput { items: vec![] }` |
`PipelineResult::empty()` is documented in the pipeline crate as the short-circuit for test users. `GetForYouFeedUrt` still serializes that empty item list into a URT timeline.
This is not an error status. Logs such as `Scored Posts response - … N posts` never appear. Treat a silent empty feed for a known test account as the `TEST_USER_IDS` gate, not as Phoenix or Thunder outage.
`params::TEST_USER_IDS` and `params::TRACE_USER_IDS` live in the unpublished `home-mixer/params` module. `TRACE_USER_IDS` only calls `b3_info.force_sample()`; it does not empty the feed.
## Thunder RESOURCE_EXHAUSTED
`ThunderServiceImpl` constructs `Semaphore::new(max_concurrent_requests)` from unpublished `thunder/args`. `GetInNetworkPosts` uses `try_acquire`, not a blocking wait:
| Semaphore result | Metrics | gRPC status |
|---|---|---|
| permit acquired | `IN_FLIGHT_REQUESTS.inc()` (decremented on drop) | request proceeds |
| at capacity | `REJECTED_REQUESTS.inc()` | `RESOURCE_EXHAUSTED` / `Server at capacity, please retry` |
Home Mixer `ThunderSource` always sends `debug: false`. It maps any tonic error, including this one, to `ThunderSource: {status}`. `fetch_candidates` then drops that `Err`, so the scored-posts RPC can still return `OK` with only out-of-network (or zero) candidates.
Strato following-list fallback in Thunder runs only when `following_user_ids` is empty **and** `req.debug` is true. The Home Mixer client never sets `debug`, so an empty `user_features.followed_user_ids` (failed `FollowedUserIdsQueryHydrator`, or no follows) yields an empty in-network set without calling Strato.
Related Thunder source strings:
- `ThunderSource: no available channel` — `ThunderClient.get_random_channel` returned `None` for the resolved `ThunderCluster`.
- `Failed to fetch following list: …` — debug-only Strato path.
- `Failed to process posts: …` — `spawn_blocking` join error after PostStore lookup.
Raise `max_concurrent_requests` on the Thunder process, reduce caller concurrency, or retry. There is no in-process queue.
## Missing retrieval_sequence
`ScoredPostsQuery::new` sets `retrieval_sequence` and `columnar_retrieval_sequence` to `None`. `RetrievalSequenceQueryHydrator` is the writer: it calls `UserActionAggregationClient.fetch_aggregated_sequence` and, on success, copies `result.sequence` and `result.columnar_bytes`.
`hydrate_query` runs enabled hydrators in parallel and **only applies `Ok` results**. A failed UAS call is logged (`Aggregation service call failed: …`) and left unset.
These sources then hard-fail if the field is still `None`:
| Source | Error string | Enable predicate (must all hold) |
|---|---|---|
| `PhoenixSource` | `PhoenixSource: missing retrieval_sequence` | Not a non-bulk topic request; new-user topic retrieval not forcing topics-only; `!in_network_only`; `!has_cached_posts` |
| `PhoenixMOESource` | `PhoenixMOESource: missing retrieval_sequence` | `EnablePhoenixMOESource`; same topic / cache / in-network guards |
| `PhoenixTopicsSource` | `PhoenixTopicsSource: missing retrieval_sequence` | Topic request (non-bulk) or new-user topic IDs; `!in_network_only`; `!has_cached_posts` |
Because `fetch_candidates` flattens `Result`, those errors remove Phoenix candidates instead of failing `GetScoredPosts`. Thunder (and cache / TweetMixer when enabled) can still populate the feed.
If the request is `in_network_only` or `has_cached_posts`, Phoenix sources are disabled and this string never appears. Diagnose an empty out-of-network slice by checking UAS hydration logs, then `query.retrieval_sequence`, then source enable predicates.
`ScoringSequenceQueryHydrator` is a sibling hydrator for ranking, not retrieval. A missing scoring sequence is a later scorer concern and is not the `missing retrieval_sequence` string.
## Unpublished Grox modules
`grox/main.py` cannot start in this checkout. `serve()` imports modules that are not in the tree:
```python
from grox.engine import Engine
from grox.service import GrpcServer
from grox.dispatcher import Dispatcher
from grox.config.config import grox_config
```
Present top-level packages: `classifiers`, `data_loaders` (partial), `embedder`, `engine.py`, `dispatcher.py`, `generators`, `lib`, `plans`, `schedules`, `summarizer`, `tasks`.
Missing imports that fail first:
| Import | Used by |
|---|---|
| `grox.config.config` (`grox_config`, `ModelName`, `KafkaTopicName`, …) | `main.py`, `engine.py`, `dispatcher.py`, classifiers, loaders, summarizers |
| `grox.service` (`GrpcServer`) | `main.py` |
| `grox.data_loaders.data_types` | classifiers, loaders, tasks |
| `grox.data_loaders.media_processor` / `media_loader` / `media_description_loader` | `engine.py`, classifiers |
| `grox.data_loaders.mappers.post_mapper` | `strato_loader.py` |
| `grox.lm.*` (`convo`, `post`, `post_v5`, `thread`, `user`) | classifiers, summarizer |
| `grox.prompts.template` | classifiers |
| `grox.classifiers.content.classifier_data_collection` | classifier graph |
| `monitor.logging` / `monitor.metrics` | `schedules/init.py`, `engine.py`, `dispatcher.py` |
There is no Grox `pyproject.toml` or requirements file. External names such as `setproctitle` and `tenacity` are also imported from published files but not pinned here.
Treat `python grox/main.py` `ModuleNotFoundError` as expected for this snapshot. The published tree is the plan/task graph, not a runnable service.
## Unpublished Rust workspace
The same boundary applies to Home Mixer, Thunder, and the candidate-pipeline crate:
| Missing in-tree module | Imported as |
|---|---|
| `home-mixer/params.rs` | `crate::params` / `params::TEST_USER_IDS`, feature params, `STRINGCENTER_BUNDLE_PATH` |
| `home-mixer/clients/` | Gizmoduck, UAS, TES, Kafka, … |
| `home-mixer/util.rs` | URT helpers |
| `thunder/args.rs` | `max_concurrent_requests`, Kafka and port flags |
| `thunder/config.rs` | `MAX_INPUT_LIST_SIZE`, `MAX_POSTS_TO_RETURN`, `MAX_VIDEOS_TO_RETURN` |
| `thunder/metrics.rs`, `o2.rs`, `schema.rs`, `strato_client.rs` | service internals |
Published files also `use` unpublished crates (`xai_home_mixer_proto`, `xai_thunder_proto`, `xai_x_service_builder`, `xai_http_server`, `xai_candidate_pipeline::component_library`, …). There is no workspace `Cargo.toml`. `cargo build` is not a supported local action.
## Adjacent pipeline errors
These are not the six named failures, but they produce empty or unscored feeds after the request is accepted.
`Scorer::run` and `Hydrator::run` require the output vector length to match the input. On mismatch they log `Skipped: length_mismatch expected=N got=M` and replace the batch with `Err` entries. `update_all` then leaves those candidates unchanged (no scores copied). The RPC still returns whatever the selector keeps.
`FollowedUserIdsQueryHydrator` writes `user_features.followed_user_ids`. If that hydrator fails, Thunder is called with an empty following list and `debug: false`, so it does not consult Strato. Combined with a missing `retrieval_sequence`, both major sources contribute nothing.
## Next
Python 3.11+, uv or pip, Git LFS, and the extract layout required before inference.
Extract `oss-phoenix-artifacts` and recognize the ranked table as success.
What this checkout can execute versus Home Mixer, Thunder, and Grox snapshots.
`--artifacts_dir`, corpus, and sequence flags after LFS is healthy.
`QueryBuilder.viewer_id` validation and CLI flags.
`TEST_USER_IDS` empty responses and proto mapping.
Semaphore capacity, Kafka ingest, and Strato fallback.
Engine / Dispatcher / GrpcServer startup that depends on unpublished modules.
`uv run pytest` targets that do not need the artifact archive.
---