# Candidate sources

> PhoenixCandidatePipeline and ForYouCandidatePipeline sources, enable predicates, cluster resolution, and served_type assignment.

- Repository: xai-org/x-algorithm
- GitHub: https://github.com/xai-org/x-algorithm
- Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c
- Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt

## Source Files

- `home-mixer/sources/mod.rs`
- `home-mixer/sources/thunder_source.rs`
- `home-mixer/sources/phoenix_source.rs`
- `home-mixer/sources/phoenix_moe_source.rs`
- `home-mixer/sources/phoenix_topics_source.rs`
- `home-mixer/sources/ads_source.rs`
- `home-mixer/sources/cached_posts_source.rs`
- `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs`

---

---
title: "Candidate sources"
description: "PhoenixCandidatePipeline and ForYouCandidatePipeline sources, enable predicates, cluster resolution, and served_type assignment."
---

`PhoenixCandidatePipeline` and `ForYouCandidatePipeline` each own a `Source` list. After query hydration, `CandidatePipeline::fetch_candidates` keeps sources whose `enable(&query)` is true, runs those `source` methods in parallel with `join_all`, and concatenates successful `Ok` vectors. `Err` results are logged by `Source::run` and dropped. Phoenix sources emit `PostCandidate`. For You sources emit `FeedItem`. Organic scored posts enter For You only through `ScoredPostsSource`, which executes `PhoenixCandidatePipeline` via `ScoredPostsServer::run_pipeline`.

Feature-switch identifiers such as `PhoenixMaxResults` live in unpublished `crate::params`. This checkout shows the names and how they are read, not the default values.

```mermaid
flowchart TB
  subgraph FY["ForYouCandidatePipeline — FeedItem"]
    SPS[ScoredPostsSource]
    ADS[AdsSource]
    WTF[WhoToFollowSource]
    PRM[PromptsSource]
    PTH[PushToHomeSource]
    BS[BlenderSelector]
    SPS --> BS
    ADS --> BS
    WTF --> BS
    PRM --> BS
    PTH --> BS
  end

  subgraph PHX["PhoenixCandidatePipeline — PostCandidate"]
    TH[ThunderSource]
    TM[TweetMixerSource]
    PX[PhoenixSource]
    PT[PhoenixTopicsSource]
    MOE[PhoenixMOESource]
    CP[CachedPostsSource]
    TK[TopKScoreSelector]
    TH --> TK
    TM --> TK
    PX --> TK
    PT --> TK
    MOE --> TK
    CP --> TK
  end

  SPS -->|run_pipeline| PHX
  QH[CachedPostsQueryHydrator / FollowedGrokTopicsQueryHydrator / RetrievalSequenceQueryHydrator] --> PHX
  SH[ServedHistoryQueryHydrator] --> FY
```

## Source contract

`xai_candidate_pipeline::source::Source<Q, C>`:

| Method | Default | Role |
|---|---|---|
| `enable(&Q) -> bool` | `true` | Gate before `run` |
| `source(&Q) -> Result<Vec<C>, String>` | required | Fetch candidates |
| `run(&Q)` | wraps `source` | Logs count or error, then returns the result |
| `name()` | short type name | Tracing / stats |

`fetch_candidates` records enabled versus disabled names, then flattens only `Ok` vectors. One failed source does not fail the request.

:::files
home-mixer/sources/
├── thunder_source.rs
├── tweet_mixer_source.rs
├── phoenix_source.rs
├── phoenix_topics_source.rs
├── phoenix_moe_source.rs
├── cached_posts_source.rs
├── scored_posts_source.rs
├── ads_source.rs
├── who_to_follow_source.rs
├── prompts_source.rs
└── push_to_home_source.rs
:::

## PhoenixCandidatePipeline sources

`build_with_clients` registers sources in this order:

1. `ThunderSource`
2. `TweetMixerSource`
3. `PhoenixSource`
4. `PhoenixTopicsSource`
5. `PhoenixMOESource`
6. `CachedPostsSource`

Order does not serialize execution. Enabled sources run concurrently; results are appended in that registration order.

| Source | Candidate type | Backend | `served_type` |
|---|---|---|---|
| `ThunderSource` | `PostCandidate` | Thunder `GetInNetworkPosts` | `ForYouInNetwork` or `RankedFollowing` |
| `TweetMixerSource` | `PostCandidate` | Tweet Mixer `HOME_RECOMMENDED_TWEETS` | `ForYouTweetMixer` |
| `PhoenixSource` | `PostCandidate` | Phoenix retrieval | `ForYouPhoenixRetrieval` |
| `PhoenixTopicsSource` | `PostCandidate` | Phoenix retrieval with topic IDs | `ForYouPhoenixRetrieval` |
| `PhoenixMOESource` | `PostCandidate` | Phoenix retrieval | `ForYouPhoenixRetrievalMoe` |
| `CachedPostsSource` | `PostCandidate` | `query.cached_posts` | Preserved from cache |

Prod retrieval client construction pins `PhoenixRetrievalCluster::Experiment1Fou` and `PhoenixRetrievalCluster::Experiment1Lap7`. Cluster choice per request is still resolved from params and deciders.

## Enable predicates

Shared query flags used by almost every Phoenix-side gate:

<ParamField body="in_network_only" type="bool">
True when the proto sets `in_network_only` or viewer `allow_for_you_recommendations == Some(false)`. Disables every out-of-network source.
</ParamField>

<ParamField body="has_cached_posts" type="bool">
True when `CachedPostsQueryHydrator` loads at least 500 cached `PostCandidate`s. Disables Thunder, Tweet Mixer, and all Phoenix retrieval sources; enables `CachedPostsSource`.
</ParamField>

<ParamField body="is_topic_request()" type="bool">
`!topic_ids.is_empty()`.
</ParamField>

<ParamField body="is_bulk_topic_request()" type="bool">
`topic_ids.len() > 6`. Bulk topic requests take the general Phoenix path, not `PhoenixTopicsSource`.
</ParamField>

<ParamField body="has_new_user_topic_ids()" type="bool">
`!new_user_topic_ids.is_empty()`. Populated by `FollowedGrokTopicsQueryHydrator` for new users when `EnableNewUserTopicRetrieval` or `EnableNewUserTopicFiltering` is on, the request is not a topic request, and `in_network_only` is false.
</ParamField>

| Source | `enable` |
|---|---|
| `ThunderSource` | `!has_cached_posts` |
| `TweetMixerSource` | `!in_network_only && !has_cached_posts` |
| `PhoenixSource` | `(!is_topic_request \|\| is_bulk_topic_request) && (!EnableNewUserTopicRetrieval \|\| !has_new_user_topic_ids) && !in_network_only && !has_cached_posts` |
| `PhoenixTopicsSource` | `((is_topic_request && !is_bulk_topic_request) \|\| (EnableNewUserTopicRetrieval && has_new_user_topic_ids)) && !in_network_only && !has_cached_posts` |
| `PhoenixMOESource` | `EnablePhoenixMOESource && (!is_topic_request \|\| is_bulk_topic_request) && !in_network_only && !has_cached_posts` |
| `CachedPostsSource` | `has_cached_posts` |

`PhoenixSource` and `PhoenixTopicsSource` are mutually exclusive for a given request. `PhoenixMOESource` can run beside `PhoenixSource` on non-topic and bulk-topic requests. When `has_cached_posts` is true, only `CachedPostsSource` runs on this pipeline.

<Warning>
`ThunderSource` stays enabled under `in_network_only`. Out-of-network sources do not. Thunder stamps `RankedFollowing` in that mode instead of `ForYouInNetwork`.
</Warning>

## Cluster resolution

### PhoenixSource

`PhoenixSource::resolve_cluster`:

1. Parse `PhoenixRetrievalInferenceClusterId` to `PhoenixRetrievalCluster`.
2. If `PhoenixRetrievalNewUserHistoryThreshold > 0` and `retrieval_sequence.metadata.length` (else `0`) is below that threshold, return `PhoenixRetrievalCluster::parse(PhoenixRetrievalNewUserInferenceClusterId)`.
3. Otherwise, if a `Decider` is present:
   - configured `Experiment1Lap7` and `enable_phoenix_retrieval_lap7_to_fou` → `Experiment1Fou`
   - configured `Experiment1Fou` and `enable_phoenix_retrieval_fou_to_lap7` → `Experiment1Lap7`
4. Otherwise use the configured cluster.

This source is the only Phoenix retrieval source that also builds `client_context` and `user_context` via unpublished `crate::util::phoenix_request`.

### PhoenixTopicsSource

Cluster is `PhoenixRetrievalCluster::parse(PhoenixRetrievalTopicInferenceClusterId)`. No new-user threshold and no Lap7/Fou decider swap.

Topic IDs sent to retrieval:

- Topic request: `query.topic_ids`
- New-user topic retrieval: `query.new_user_topic_ids`

Each ID is passed through `TopicIdExpansion::resolve_first`, which currently returns the ID unchanged. `TopicFilteringId` is parsed to `TopicFilteringExperiment`; `TopicFilteringOverrides` (`topic_id=ExperimentId` comma list) can override the first matching topic. The resolved experiment is sent as proto mode:

| `TopicFilteringExperiment` | proto mode |
|---|---|
| `Unfiltered` (also unknown strings) | `0` |
| `CuratedV0` | `1` |
| `CuratedV0V1` | `2` |
| `PostBased90Pct` | `3` |
| `PostBased75Pct` | `4` |
| `PostBased50Pct` | `5` |

### PhoenixMOESource

Cluster is `PhoenixRetrievalCluster::parse(PhoenixRetrievalMOEInferenceClusterId)`. Max results is `PhoenixMOEMaxResults`, not `PhoenixMaxResults`. Topic IDs and filter mode are empty / `None`. Client and user context are `None`.

### ThunderSource

```text
configured = ThunderCluster::parse(ThunderClusterId)
cluster    = ThunderCluster::resolve(configured, query.decider)
channel    = thunder_client.get_random_channel(cluster)
```

`ThunderCluster` lives in the unpublished candidate-pipeline client crate. Missing channel returns `"ThunderSource: no available channel"`.

## Retrieval requests

All three Phoenix sources require `query.retrieval_sequence`. That field is filled by `RetrievalSequenceQueryHydrator` from the user-action aggregation service. Absence fails the source with `"<Source>: missing retrieval_sequence"`.

| Field | `PhoenixSource` | `PhoenixTopicsSource` | `PhoenixMOESource` |
|---|---|---|---|
| Cluster | `resolve_cluster` | `PhoenixRetrievalTopicInferenceClusterId` | `PhoenixRetrievalMOEInferenceClusterId` |
| Max results | `PhoenixMaxResults` | `PhoenixMaxResults` | `PhoenixMOEMaxResults` |
| Topic entity IDs | `[]` | expanded effective topic IDs | `[]` |
| Topic filter mode | `None` | `Some(mode)` | `None` |
| Client / user context | built | `None` | `None` |
| Sequence | `retrieval_sequence` + `columnar_retrieval_sequence` | same | same |

Candidates are taken from `response.top_k_candidates[*].candidates[*].candidate`. `in_reply_to_tweet_id` is always `Some(...)` from the retrieval tweet info. `retweeted_tweet_id` is set only when the retrieval value is non-zero.

`ThunderSource` builds `GetInNetworkPostsRequest`:

<ParamField body="user_id" type="u64" required>
Viewer ID.
</ParamField>

<ParamField body="following_user_ids" type="Vec<u64>">
`query.user_features.followed_user_ids`.
</ParamField>

<ParamField body="max_results" type="param">
`ThunderMaxResults`.
</ParamField>

<ParamField body="exclude_tweet_ids" type="Vec<u64>">
`query.seen_ids`.
</ParamField>

<ParamField body="algorithm" type="param">
`ThunderAlgorithm`.
</ParamField>

<ParamField body="debug / is_video_request" type="bool">
Hard-coded `false`.
</ParamField>

Thunder also writes `query.in_network_replies` from posts that have `in_reply_to_post_id`. Each candidate gets `ancestors = [in_reply_to_tweet_id, conversation_id?]` when those IDs differ.

`TweetMixerSource` requests `Product::HOME_RECOMMENDED_TWEETS` with `TweetMixerMaxResults` and `seen_ids` as `excluded_tweet_ids`. It drops tweets whose snowflake age is greater than `Duration::from_secs(MAX_POST_AGE)` (same constant `AgeFilter` uses). `retweeted_tweet_id` is always `None` here.

## Cached posts path

`CachedPostsQueryHydrator` (gated by `EnableCachedPosts`) GETs Redis key `cached_posts_key(user_id, topic_ids, in_network_only, exclude_videos)` with a 300 ms timeout, then zstd-decompresses JSON into `Vec<PostCandidate>`.

| Hydrator outcome | `has_cached_posts` | Sources that run |
|---|---|---|
| Empty payload, timeout, or Redis error | `false` | Live Thunder / Tweet Mixer / Phoenix set |
| Fewer than 500 posts | `false` (posts still stored on the query) | Live set |
| `cached_posts.len() >= 500` | `true` | `CachedPostsSource` only |

`CachedPostsSource` returns `query.cached_posts.clone()`. It does not rewrite `served_type`.

## served_type assignment

Sources stamp `PostCandidate.served_type` at fetch time. `ScoredPostsServer` copies `served_type as i32` onto `ScoredPost` (`0` if unset). Stats treat `ForYouPhoenixRetrieval` as `PhoenixRetrievalTweets` and `ForYouPhoenixRetrievalMoe` as `PhoenixRetrievalMoeTweets`.

| Origin | `ServedType` |
|---|---|
| Thunder, `!in_network_only` | `ForYouInNetwork` |
| Thunder, `in_network_only` | `RankedFollowing` |
| `PhoenixSource` | `ForYouPhoenixRetrieval` |
| `PhoenixTopicsSource` | `ForYouPhoenixRetrieval` |
| `PhoenixMOESource` | `ForYouPhoenixRetrievalMoe` |
| `TweetMixerSource` | `ForYouTweetMixer` |
| `CachedPostsSource` | Unchanged from Redis JSON |
| `PushToHomeSource` | `ForYouPushToHome` |

`PhoenixSource` and `PhoenixTopicsSource` share `ForYouPhoenixRetrieval`. Downstream stats keyed only on that enum cannot distinguish general retrieval from topic retrieval.

## ForYouCandidatePipeline sources

`ForYouCandidatePipeline::build` registers:

1. `ScoredPostsSource`
2. `AdsSource`
3. `WhoToFollowSource`
4. `PromptsSource`
5. `PushToHomeSource`

This pipeline has no hydrators, filters, or scorers. `BlenderSelector` merges the `FeedItem` streams.

| Source | `enable` | `FeedItem` variant |
|---|---|---|
| `ScoredPostsSource` | default `true` | `Item::Post(ScoredPost)` |
| `AdsSource` | `EnableAdsSource && !is_preview` | `Item::Ad` |
| `WhoToFollowSource` | `EnableWhoToFollowModule && who_to_follow_eligible` | `Item::WhoToFollow` |
| `PromptsSource` | `EnablePrompts` | `Item::Prompt` |
| `PushToHomeSource` | `push_to_home_post_id.is_some()` | `Item::PushToHome` |

### ScoredPostsSource

Calls `ScoredPostsServer::run_pipeline(query.clone())`. If `params::TEST_USER_IDS` contains `query.user_id`, that path returns an empty `PipelineOutput` without executing Phoenix sources. Each selected `ScoredPost` becomes a `FeedItem` at `position: 0`; the blender assigns display positions later.

### AdsSource

Builds `AdIndexRequest` with `ProductSurface::HomeTimelineRanking` and a `ClientContext` from `user_id`, `client_app_id`, country / language, IP, user agent, roles, and device IDs. Preview requests skip ads.

### WhoToFollowSource

`who_to_follow_eligible` starts `false` and is set by `ServedHistoryQueryHydrator` when `EnableUrtMigrationComponents` is on. Eligibility is fatigue-based: no `EntityIdType::WHO_TO_FOLLOW` history, or last serve is at least `WhoToFollowFatigueHours` old.

The request is `Product::HomeWhoToFollow` with up to 200 excluded user IDs taken from served-history WTF entries. The source keeps at most 3 recommendations and returns a single module item, or `[]` if the mixer returns none.

<Note>
If `EnableUrtMigrationComponents` is off, `who_to_follow_eligible` stays `false` and `WhoToFollowSource` does not run even when `EnableWhoToFollowModule` is on.
</Note>

### PromptsSource

Requests `DisplayLocation::HOME_TIMELINE` for `INLINE_PROMPT`, `FULL_COVER`, `HALF_COVER`, and `RELEVANCE_PROMPT`. Each injection is Thrift-serialized into `Prompt.injection`. Serialization failure fails that source.

### PushToHomeSource

Requires `query.push_to_home_post_id`. TES miss or `Ok(None)` returns `[]`. TES `Err` fails the source. Root tweets (no `in_reply_to_tweet_id`) request up to 3 reply-mixer author IDs excluding the original author; reply-mixer errors log and leave `facepile_user_ids` empty. Reply tweets skip the facepile.

## Feature switches and request fields

Phoenix / Thunder / Tweet Mixer:

<ParamField body="PhoenixRetrievalInferenceClusterId" type="string">
Default cluster string for `PhoenixSource`.
</ParamField>

<ParamField body="PhoenixRetrievalNewUserInferenceClusterId" type="string">
Override cluster when history length is below threshold.
</ParamField>

<ParamField body="PhoenixRetrievalNewUserHistoryThreshold" type="u64">
`0` disables the new-user cluster override.
</ParamField>

<ParamField body="PhoenixRetrievalTopicInferenceClusterId" type="string">
Cluster for `PhoenixTopicsSource`.
</ParamField>

<ParamField body="PhoenixRetrievalMOEInferenceClusterId" type="string">
Cluster for `PhoenixMOESource`.
</ParamField>

<ParamField body="PhoenixMaxResults" type="integer">
Max results for `PhoenixSource` and `PhoenixTopicsSource`.
</ParamField>

<ParamField body="PhoenixMOEMaxResults" type="integer">
Max results for `PhoenixMOESource`.
</ParamField>

<ParamField body="EnablePhoenixMOESource" type="bool">
Master gate for `PhoenixMOESource`.
</ParamField>

<ParamField body="EnableNewUserTopicRetrieval" type="bool">
Routes new-user followed topics to `PhoenixTopicsSource` and disables `PhoenixSource`.
</ParamField>

<ParamField body="ThunderClusterId / ThunderAlgorithm / ThunderMaxResults" type="param">
Thunder cluster string, algorithm, and result cap.
</ParamField>

<ParamField body="TweetMixerMaxResults" type="integer">
Tweet Mixer result cap.
</ParamField>

<ParamField body="EnableCachedPosts" type="bool">
Enables `CachedPostsQueryHydrator`.
</ParamField>

For You:

<ParamField body="EnableAdsSource" type="bool">
Ads intake, still blocked when `is_preview`.
</ParamField>

<ParamField body="EnableWhoToFollowModule" type="bool">
WTF intake, still requires `who_to_follow_eligible`.
</ParamField>

<ParamField body="EnablePrompts" type="bool">
Prompts intake.
</ParamField>

<ParamField body="push_to_home_post_id" type="Option&lt;u64&gt;">
Non-zero proto field after `HomeMixerServer` mapping. Enables `PushToHomeSource`.
</ParamField>

## Errors and empty results

| Condition | Source result | Pipeline effect |
|---|---|---|
| Missing `retrieval_sequence` | `Err("… missing retrieval_sequence")` | That source dropped |
| Thunder has no channel | `Err("ThunderSource: no available channel")` | Thunder dropped |
| Phoenix / Tweet Mixer / ads / WTF / prompts RPC error | `Err("<Source>: …")` | That source dropped |
| Tweet Mixer tweet older than `MAX_POST_AGE` | omitted | Not a hard error |
| Cached Redis timeout / empty / decode failure | hydrator error or empty | `has_cached_posts` stays false |
| `TEST_USER_IDS` viewer | empty `PipelineOutput` | No Phoenix sources run |
| TES miss for push-to-home | `Ok([])` | No PTH item |
| TES error for push-to-home | `Err` | PTH dropped |
| Empty WTF recommendations | `Ok([])` | No module |

These sources are production Home Mixer code. They depend on unpublished clients (`PhoenixRetrievalClient`, `ThunderClient`, Tweet Mixer, Ad Index, prompts, WTF, TES, Reply Mixer). This checkout can inspect the predicates and stamps; it cannot execute the live sources. The local Phoenix path is `phoenix/run_pipeline.py` against extracted artifacts, which does not run these Rust sources.

## Next

<CardGroup>
  <Card title="For You request lifecycle" href="/request-lifecycle">
    How `execute` hydrates the query, fans out sources, then hydrates, filters, and scores.
  </Card>
  <Card title="In-network and out-of-network" href="/in-network-out-of-network">
    Thunder versus Phoenix retrieval and how `in_network_only` changes sourcing.
  </Card>
  <Card title="Thunder GetInNetworkPosts" href="/thunder-in-network-posts">
    RPC fields, PostStore timelines, and following-list fallback behind `ThunderSource`.
  </Card>
  <Card title="Blend ads into the feed" href="/blend-ads">
    How `AdsSource` items are positioned after intake.
  </Card>
  <Card title="Add a pipeline component" href="/add-pipeline-component">
    `Source` contract, length-match rules, and where a new source is registered.
  </Card>
  <Card title="Filters and hydrators" href="/filters-and-hydrators">
    Query hydrators that populate `has_cached_posts`, topic IDs, and sequences before sources run.
  </Card>
  <Card title="ScoredPostsQuery and gRPC" href="/scored-posts-query">
    Query fields that drive enable predicates and `TEST_USER_IDS` empty feeds.
  </Card>
  <Card title="Troubleshooting" href="/troubleshooting">
    `missing retrieval_sequence`, Thunder `RESOURCE_EXHAUSTED`, and related source failures.
  </Card>
</CardGroup>