# Overview > Public surfaces of the For You stack, who can run what from this checkout, and the first docs routes to follow. - Repository: xai-org/x-algorithm - GitHub: https://github.com/xai-org/x-algorithm - Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c - Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt ## Source Files - `README.md` - `phoenix/README.md` - `phoenix/run_pipeline.py` - `home-mixer/lib.rs` - `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs` - `candidate-pipeline/candidate_pipeline.rs` - `thunder/thunder_service.rs` --- --- title: "Overview" description: "Public surfaces of the For You stack, who can run what from this checkout, and the first docs routes to follow." --- This repository is a source snapshot of X's For You recommendation stack: Home Mixer orchestrates feed assembly, Thunder serves in-network posts, Phoenix retrieves and ranks candidates, Grox runs async content-understanding plans, and `candidate-pipeline` defines the shared `CandidatePipeline` trait. The only executable surface in this checkout is the Phoenix JAX inference tree under `phoenix/`. Home Mixer, Thunder, Grox, and the pipeline crate are published as production source without a Cargo/Python workspace that can start those servers. Do not expect `cargo run` or `python grox/main.py` to start production services from this tree. There is no `Cargo.toml`, and Home Mixer, Thunder, and Grox import unpublished crates and modules (`xai_*`, `grox.service`, `grox.config`, Thunder `args`/`config`/`strato_client`). ## Checkout layout :::files x-algorithm/ ├── phoenix/ # Runnable JAX retrieval + ranking │ ├── run_pipeline.py # End-to-end entry (retrieval → rank) │ ├── run_retrieval.py # Standalone retrieval toy runner │ ├── run_ranker.py # Standalone ranking toy runner │ ├── recsys_model.py # PhoenixModelConfig + ranker │ ├── recsys_retrieval_model.py │ ├── artifacts/oss-phoenix-artifacts.zip # Git LFS, ~3 GB │ └── test_recsys_*.py ├── home-mixer/ # For You + Scored Posts gRPC snapshot │ ├── main.rs # HomeMixerServer CLI │ ├── server.rs # QueryBuilder + service registration │ └── candidate_pipeline/ # ForYouCandidatePipeline, PhoenixCandidatePipeline ├── thunder/ # InNetworkPostsService snapshot │ ├── main.rs │ └── thunder_service.rs # GetInNetworkPosts ├── grox/ # Content-understanding engine snapshot │ ├── main.py # Engine + Dispatcher + GrpcServer │ └── plans/plan_master.py └── candidate-pipeline/ # Source / Hydrator / Filter / Scorer traits ::: ## Who can run what | Surface | Path | Runnable here? | What this checkout actually provides | |---|---|---|---| | Phoenix inference | `phoenix/run_pipeline.py` | Yes, after LFS extract | Loads published checkpoints + `sports_corpus.npz`, prints a ranked table | | Phoenix unit tests | `phoenix/test_recsys_*.py` | Yes | Attention-mask and retrieval assertions via `uv run pytest` | | Phoenix toy runners | `run_ranker.py`, `run_retrieval.py` | Partial | Hardcoded 128-dim toy configs; they do **not** load the LFS archive | | Home Mixer gRPC | `home-mixer/main.rs` | No | Source for `HomeMixerServer`, `QueryBuilder`, both feed services | | Thunder gRPC | `thunder/main.rs` | No | Source for `ThunderServiceImpl::get_in_network_posts` | | Grox server | `grox/main.py` | No | Source for `Engine` / `Dispatcher` / `PlanMaster`; `grox.service` and `grox.config` are unpublished | | Pipeline framework | `candidate-pipeline/` | No (library snapshot) | Trait contracts only; Home Mixer imports extra unpublished `xai_candidate_pipeline::component_library` | `phoenix/pyproject.toml` requires Python `>=3.11` and pins `jax==0.8.1` plus `dm-haiku`. Install with `uv sync` in `phoenix/`, or `pip install jax jaxlib dm-haiku numpy`. The zip is a Git LFS pointer until `git lfs pull`; extract to `phoenix/artifacts/oss-phoenix-artifacts/` before inference. ## Production request path A live For You request does not call Phoenix Python. `HomeMixerServer` builds `PhoenixCandidatePipeline`, wraps it in `ScoredPostsServer`, then wraps that server as `ScoredPostsSource` inside `ForYouCandidatePipeline`. ```mermaid flowchart TB subgraph grpc [Home Mixer gRPC] FY[ForYouFeedService] SP[ScoredPostsService] end subgraph fyPipe [ForYouCandidatePipeline] QH1[ServedHistory / PastRequestTimestamps] SRC1[ScoredPostsSource] SRC2[AdsSource / WhoToFollow / Prompts / PushToHome] BLEND[BlenderSelector] end subgraph phxPipe [PhoenixCandidatePipeline] QH2[Action sequences, follows, topics, bloom, IP] TH[ThunderSource] PX[PhoenixSource / Topics / MoE] TM[TweetMixerSource] CACHE[CachedPostsSource] HYD[Candidate hydrators] FIL[Pre-score filters] SC[PhoenixScorer → RankingScorer → VMRanker] TOPK[TopKScoreSelector] PSF[VF + conversation dedup] end subgraph unpublished [Not startable from this checkout] THSVC[Thunder InNetworkPostsService] PXCLUS[Phoenix retrieval clusters] GROX[Grox Engine / PlanMaster] end FY --> QH1 --> SRC1 QH1 --> SRC2 --> BLEND SRC1 --> SP SP --> QH2 --> TH & PX & TM & CACHE TH & PX & TM & CACHE --> HYD --> FIL --> SC --> TOPK --> PSF TH -.-> THSVC PX -.-> PXCLUS ``` `ForYouCandidatePipeline` has empty hydrator, filter, and scorer slices. Organic ranking happens inside `PhoenixCandidatePipeline`; For You only hydrates served-history context, fans out extra item types, and blends with `BlenderSelector`. `CandidatePipeline::execute` is the shared stage order: 1. Query hydrators, then dependent query hydrators 2. Sources (parallel) 3. Candidate hydrators (length and order must match; drops are illegal) 4. Pre-score filters 5. Scorers 6. Selector 7. Post-selection hydrators and filters 8. Truncate to `result_size` 9. `finalize`, then fire-and-forget side effects `PipelineResult::empty()` short-circuits `TEST_USER_IDS` without running those stages. ## Public gRPC and CLI surfaces These identifiers exist in source. Binding them requires the unpublished proto crates and production clients. ### Home Mixer `HomeMixerServer` registers two services with gzip/zstd and `params::MAX_GRPC_MESSAGE_SIZE`: | Service | RPC | Input | Output | |---|---|---|---| | `ScoredPostsService` | `GetScoredPosts` | `ScoredPostsQuery` | `ScoredPostsResponse` (`ScoredPost` list) | | `ScoredPostsService` | `GetDebugScoredPosts` | `DebugScoredPostsQuery` | scored posts + `debug_json` | | `ForYouFeedService` | `GetForYouFeed` | `ForYouFeedQuery` (must include `query`) | `ForYouFeedResponse` (`FeedItem` list) | | `ForYouFeedService` | `GetForYouFeedUrt` | `ForYouFeedQuery` | `ForYouFeedUrtResponse` (serialized URT) | `QueryBuilder::build` rejects `viewer_id == 0` with `INVALID_ARGUMENT: viewer_id must be specified`. `in_network_only` is true when the proto flag is set **or** Gizmoduck `allow_for_you_recommendations == Some(false)`. Both `ScoredPostsServer::run_pipeline` and `ForYouFeedServer::get_for_you_feed` return empty payloads when `params::TEST_USER_IDS` contains the viewer. `home-mixer/main.rs` CLI: gRPC listen port. Metrics port. Shard ordinal. Negative disables `ShardCoordinate`. Shard ring size when `shard_coordinate >= 0`. Passed into prod clients and feature-switch recipient. OpenTelemetry endpoint. ### Thunder `ThunderServiceImpl` implements `InNetworkPostsService.GetInNetworkPosts`. Request fields used in this snapshot: `user_id`, `following_user_ids`, `exclude_tweet_ids`, `max_results`, `is_video_request`, `debug`. An empty following list is fetched from Strato only when `debug` is set. At semaphore capacity the RPC returns `RESOURCE_EXHAUSTED` (`"Server at capacity, please retry"`). `thunder/lib.rs` also declares `args`, `config`, `metrics`, `o2`, `schema`, and `strato_client`, which are not present in the tree. ### Grox `grox/main.py` starts `Engine`, `Dispatcher`, and `GrpcServer` from a shared schedule context. `Engine` fans every `TaskPayload` through `PlanMaster.ALL_PLANS` (banger, post safety, spam, embeddings, reply ranking, PTOS). Startup imports `grox.service` and `grox.config.config`, which are not in this checkout. ## Local Phoenix inference The published path is retrieve-then-rank against exported artifacts, not Home Mixer. `phoenix/artifacts/oss-phoenix-artifacts.zip` is a Git LFS object (`.gitattributes` tracks `*.zip` and `*.npz`). After `git lfs pull`: ```bash cd phoenix unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/ ``` That must produce `artifacts/oss-phoenix-artifacts/{retrieval,ranker,sports_corpus.npz,example_sequence.json}`. The script default `--artifacts_dir ./artifacts` does **not** match the extract layout; pass the nested directory. ```bash uv cd phoenix uv sync uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts ``` ```bash pip cd phoenix pip install jax jaxlib dm-haiku numpy python run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts ``` Success is a `PIPELINE RESULTS` table, not a gRPC response. Columns: `Rank`, `Score`, `Ret`, `Fav`, `Reply`, `RT`, `Dwell`, `VQV`, `Topics`, Post URL (`https://x.com/a/status/{post_id}`). Directory that contains `retrieval/`, `ranker/`, and the corpus/sequence files. User history JSON. Default: `{artifacts_dir}/example_sequence.json`. Corpus NPZ. Default: `{artifacts_dir}/sports_corpus.npz`. Dot-product cutoff against the corpus. Rows printed after ranking. Local ranking uses a demo weighted sum, not Home Mixer's `RankingScorer` feature-switch table: ```text score = 1.0·P(fav) + 0.5·P(reply) + 0.3·P(rt) + 0.2·P(dwell) ``` Action indices in that sum: `IDX_FAV=1`, `IDX_REPLY=4`, `IDX_QUOTE=5`, `IDX_RT=6`, `IDX_DWELL=11`, `IDX_VQV=13`. Production `PhoenixCandidatePipeline` scorers are `PhoenixScorer` (cluster + egress fallback), `RankingScorer`, then `VMRanker`. `WeightedScorer`, `AuthorDiversityScorer`, and `OONScorer` exist as files but are not in `home-mixer/scorers/mod.rs` and are not wired into `build_with_clients`. The Phoenix README documents the published mini checkpoint as 128-dim, 4 layers, 4 heads, `history_seq_len=127`, `candidate_seq_len=64`, 1M user/item/author vocabs, 2 hashes per entity, 19 actions. Live weights come from each `config.json` after extract. The sports corpus is a frozen ~537K-post Sports-topic slice, not the production ANN index. Tests (no LFS required): ```bash cd phoenix uv run pytest test_recsys_model.py test_recsys_retrieval_model.py ``` ## Candidate sources in this snapshot **`PhoenixCandidatePipeline`** (`PostCandidate`): `ThunderSource`, `TweetMixerSource`, `PhoenixSource`, `PhoenixTopicsSource`, `PhoenixMOESource`, `CachedPostsSource`. Thunder is in-network (followed-author `GetInNetworkPosts`). Phoenix sources are out-of-network retrieval. **`ForYouCandidatePipeline`** (`FeedItem`): `ScoredPostsSource` (in-process `ScoredPostsServer::run_pipeline`), `AdsSource`, `WhoToFollowSource`, `PromptsSource`, `PushToHomeSource`. ## Design constraints that leak into every later page - **Candidate isolation.** Ranker attention lets each candidate see user + history + self, never other candidates. Scores are batch-independent. - **Hash embeddings.** User/item/author IDs go through linear-congruential hashes into pad-offset unified tables (`pad=65`) before the transformer. - **Hydrator contract.** Returned vectors must keep input length and order. A length mismatch is skipped and replaced with errors; use a `Filter` to drop. - **Apache 2.0.** See `LICENSE`. Transformer blocks are adapted from the Grok-1 open-source release. ## Next Python 3.11+, uv or pip, Git LFS, and the extract layout required before inference. Extract `oss-phoenix-artifacts`, run `run_pipeline.py`, and recognize the ranked table. What this checkout can execute versus Home Mixer, Thunder, and Grox snapshots. `CandidatePipeline.execute` stages and how For You wraps Phoenix.