# For You request lifecycle > CandidatePipeline.execute stages from query hydration through side effects, and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline. - Repository: xai-org/x-algorithm - GitHub: https://github.com/xai-org/x-algorithm - Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c - Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt ## Source Files - `candidate-pipeline/candidate_pipeline.rs` - `home-mixer/candidate_pipeline/phoenix_candidate_pipeline.rs` - `home-mixer/candidate_pipeline/for_you_candidate_pipeline.rs` - `home-mixer/server.rs` - `home-mixer/scored_posts_server.rs` - `home-mixer/for_you_server.rs` --- --- title: "For You request lifecycle" description: "CandidatePipeline.execute stages from query hydration through side effects, and how ForYouCandidatePipeline wraps PhoenixCandidatePipeline." --- `CandidatePipeline::execute` is the shared driver for every Home Mixer request. `HomeMixerServer` builds one `PhoenixCandidatePipeline` and one `ForYouCandidatePipeline`. Scored Posts RPCs call `PhoenixCandidatePipeline::execute` directly. For You RPCs call `ForYouCandidatePipeline::execute`, which re-enters Phoenix through `ScoredPostsSource` → `ScoredPostsServer::run_pipeline`. Both pipelines implement `CandidatePipeline`. Phoenix’s candidate type is `PostCandidate`. For You’s candidate type is `FeedItem` (`Post`, `Ad`, `WhoToFollow`, `Prompt`, or `PushToHome`). This checkout publishes the Home Mixer and candidate-pipeline sources. Production `params` constants (`RESULT_SIZE`, `FOR_YOU_MAX_RESULT_SIZE`, `TOP_K_CANDIDATES_TO_SELECT`, `TEST_USER_IDS`) and the unpublished client crates are not in this tree. Local inference uses `phoenix/run_pipeline.py`, not these gRPC servers. ## Request entry points `HomeMixerServer::build` constructs a shared `PhoenixCandidatePipeline`, wraps it in `ScoredPostsServer`, then builds `ForYouCandidatePipeline` with that same `ScoredPostsServer`. Both gRPC services share one `QueryBuilder`. :::endpoint POST /xai.home_mixer.ScoredPostsService/GetScoredPosts Ranked PostCandidate list Runs `QueryBuilder::build` then `ScoredPostsServer::run_pipeline` → `PhoenixCandidatePipeline::execute`. Response is `ScoredPostsResponse.scored_posts`. ::: :::endpoint POST /xai.home_mixer.ScoredPostsService/GetDebugScoredPosts Ranked posts plus pipeline dump Forces B3 sampling, applies `feature_switch_overrides`, then returns `scored_posts` plus `debug_json` with query, retrieved, filtered, and selected candidates. ::: :::endpoint POST /xai.home_mixer.ForYouFeedService/GetForYouFeed Blended FeedItem list Requires `ForYouFeedQuery.query`. Runs `ForYouCandidatePipeline::execute`. Response is `ForYouFeedResponse.items`. ::: :::endpoint POST /xai.home_mixer.ForYouFeedService/GetForYouFeedUrt URT timeline bytes Same execute path as `GetForYouFeed`, then `urt::make_urt_timeline`. Decodes `cursor` into `is_bottom_request` / `is_top_request`. Copies `request_context` and `is_polling` onto `ScoredPostsQuery` after `QueryBuilder::build`. ::: ### QueryBuilder Every RPC funnels through `QueryBuilder::build` before `execute`. Must be non-zero. `0` returns `Status::invalid_argument("viewer_id must be specified")`. Set if the proto flag is true **or** Gizmoduck `allow_for_you_recommendations == Some(false)`. `ScoredPostsServer::run_pipeline` and `ForYouFeedServer::get_for_you_feed` return empty results without calling `execute`. Forces B3 sampling on the request span. Other `ScoredPostsQuery` fields set here: `seen_ids`, `served_ids`, `topic_ids`, `excluded_topic_ids`, `exclude_videos`, `is_bottom_request`, `is_preview`, `push_to_home_post_id`, feature-switch `params`, a per-user `Decider`, Gizmoduck roles / muted keywords / follower count / subscription / age, device status, `request_id`, and a new `prediction_id`. Gizmoduck viewer fetch times out at `200ms` and falls back to `ViewerData::default()`. `is_shadow_traffic` is `is_sampled(request_id, 0.5)`. ## Nested pipelines For You does **not** embed `PhoenixCandidatePipeline` as a field. The wrap is a source call: ```text ForYouFeedService / ForYouFeedUrt │ ▼ QueryBuilder.build → ScoredPostsQuery │ ▼ ForYouCandidatePipeline.execute │ query hydrators (served history, timestamps) │ sources (parallel) │ ├─ ScoredPostsSource ──► ScoredPostsServer.run_pipeline │ │ │ │ │ ▼ │ │ PhoenixCandidatePipeline.execute │ │ │ │ │ ▼ │ │ Vec → FeedItem::Post │ ├─ AdsSource │ ├─ WhoToFollowSource │ ├─ PromptsSource │ └─ PushToHomeSource │ BlenderSelector (ads / prompts / WTF / push-to-home) │ side effects (spawned, non-blocking) ▼ ForYouFeedResponse.items | URT bytes ``` ```mermaid sequenceDiagram participant Client participant FY as ForYouFeedService participant QB as QueryBuilder participant ForYou as ForYouCandidatePipeline participant SPS as ScoredPostsSource participant SPSV as ScoredPostsServer participant Phoenix as PhoenixCandidatePipeline Client->>FY: GetForYouFeed(ForYouFeedQuery) FY->>QB: build(ScoredPostsQuery) alt viewer_id == 0 QB-->>FY: Status invalid_argument else user_id in TEST_USER_IDS FY-->>Client: empty items else FY->>ForYou: execute(query) ForYou->>ForYou: hydrate_query par parallel sources ForYou->>SPS: source(query.clone()) SPS->>SPSV: run_pipeline SPSV->>Phoenix: execute Note over Phoenix: query hydrate → sources → hydrate → filter → score → select → post-select → side effects Phoenix-->>SPSV: PipelineResult PostCandidate SPSV-->>SPS: FeedItem::Post[] ForYou->>ForYou: Ads / WTF / Prompts / PushToHome end ForYou->>ForYou: BlenderSelector ForYou->>ForYou: spawn side effects ForYou-->>FY: selected FeedItems FY-->>Client: ForYouFeedResponse end ``` Direct `GetScoredPosts` skips the For You layer and calls `PhoenixCandidatePipeline::execute` on the same `ScoredPostsQuery` type. ## CandidatePipeline.execute stages `execute` is a default trait method. Neither Home Mixer pipeline overrides it. Stage order is fixed. | Order | Stage | Parallelism | Drop / fail behavior | | --- | --- | --- | --- | | 1 | `hydrate_query` | Enabled `QueryHydrator`s in parallel | `Err` is logged and skipped; only `Ok` results call `update` | | 2 | `hydrate_dependent_query` | Same, after stage 1 | Default is empty (`&[]`). Neither Home Mixer pipeline wires this | | 3 | `fetch_candidates` | Enabled `Source`s in parallel | `Result` is flattened; a failed source contributes nothing | | 4 | `hydrate` | Enabled candidate `Hydrator`s in parallel | Length must match input; mismatch skips that hydrator. Cannot drop candidates | | 5 | `filter` | Enabled `Filter`s **sequentially** | Each filter partitions `kept` / `removed` | | 6 | `score` | Enabled `Scorer`s **sequentially** | Length must match; mismatch skips that scorer. Cannot drop candidates | | 7 | `select` | Single `Selector` | If `enable` is false, all candidates stay selected | | 8 | `hydrate_post_selection` | Parallel hydrators | Same length-match / no-drop contract as stage 4 | | 9 | `filter_post_selection` | Sequential filters | Removed candidates append to `filtered_candidates` | | 10 | Truncate | `split_off(result_size())` | Surplus moves to `non_selected_candidates` | | 11 | `finalize` | Sync hook | Default no-op. Neither pipeline overrides it | | 12 | `run_side_effects` | `tokio::spawn` + `join_all` | Does not delay the returned `PipelineResult` | Every component has `enable(&query)`. Disabled names are recorded on the tracing span as `disabled`. ### Stage contracts that change execute behavior - **Query hydrators** return a fresh `Q` and copy only their fields in `update`. Failed hydrators leave those fields at `QueryBuilder` defaults (`retrieval_sequence: None`, empty follow lists, and so on). - **Sources** return `Result, String>`. `PhoenixSource` fails with `"PhoenixSource: missing retrieval_sequence"` when stage 1 did not populate the sequence; that error is dropped by `flatten()`, so the request continues with other sources. - **Hydrators and scorers** must return one result per input candidate, same order. `Hydrator::run` / `Scorer::run` replace a length mismatch with `Err("… length_mismatch …")` for every slot. - **Filters** are the only stage allowed to drop candidates before selection. - **Side effects** receive `SideEffectInput { query, selected_candidates, non_selected_candidates }`. `non_selected` is selector leftovers plus the post-truncate tail. ```rust // candidate-pipeline/candidate_pipeline.rs — execute body let hydrated_query = self.hydrate_query(query).await; let hydrated_query = self.hydrate_dependent_query(hydrated_query).await; let candidates = self.fetch_candidates(&hydrated_query).await; let hydrated_candidates = self.hydrate(&hydrated_query, candidates).await; let (kept_candidates, mut filtered_candidates) = self.filter(&hydrated_query, hydrated_candidates.clone()); let scored_candidates = self.score(&hydrated_query, kept_candidates).await; let SelectResult { selected: selected_candidates, non_selected: mut non_selected_candidates } = self.select(&hydrated_query, scored_candidates); // post-selection hydrate + filter, truncate to result_size(), finalize, spawn side effects ``` `PipelineResult.retrieved_candidates` is the **post-hydrate** set (`hydrated_candidates`), not the raw source union. `filtered_candidates` is pre-score removals plus post-selection removals. ## PhoenixCandidatePipeline `CandidatePipeline`. `result_size()` is unpublished `params::RESULT_SIZE`. Selector is `TopKScoreSelector` with size `params::TOP_K_CANDIDATES_TO_SELECT` (truncate happens again at `RESULT_SIZE` after post-selection). ### Query hydrators (parallel) `ScoringSequenceQueryHydrator`, `RetrievalSequenceQueryHydrator`, `BlockedUserIdsQueryHydrator`, `MutedUserIdsQueryHydrator`, `FollowedUserIdsQueryHydrator`, `SubscribedUserIdsQueryHydrator`, `CachedPostsQueryHydrator`, `MutualFollowQueryHydrator`, `UserDemographicsQueryHydrator`, `FollowedGrokTopicsQueryHydrator`, `FollowedStarterPacksQueryHydrator`, `InferredGrokTopicsQueryHydrator`, `ImpressionBloomFilterQueryHydrator`, `IpQueryHydrator`, `UserInferredGenderQueryHydrator`. `ImpressedPostsQueryHydrator` is constructed in `build_with_clients` and bound to `_impressed_posts_hydrator` — it is **not** inserted into the hydrator list. ### Sources (parallel) | Source | `enable` | | --- | --- | | `ThunderSource` | `!query.has_cached_posts` | | `TweetMixerSource` | `!in_network_only && !has_cached_posts` | | `PhoenixSource` | Not a small topic request; not new-user topic retrieval; `!in_network_only && !has_cached_posts` | | `PhoenixTopicsSource` | Topic or new-user topic request; `!in_network_only && !has_cached_posts` | | `PhoenixMOESource` | `EnablePhoenixMOESource` and same topic / `in_network_only` gates as Phoenix (no cache gate in `enable`) | | `CachedPostsSource` | `query.has_cached_posts` | When `CachedPostsQueryHydrator` sets `has_cached_posts`, Thunder / TweetMixer / Phoenix / Phoenix Topics disable and `CachedPostsSource` returns `query.cached_posts`. `PhoenixScorer` also disables on that flag, so cached candidates skip re-scoring. ### Pre-score hydrators and filters Hydrators (parallel): `InNetworkCandidateHydrator`, `CoreDataCandidateHydrator`, `QuoteHydrator`, `VideoDurationCandidateHydrator`, `HasMediaHydrator`, `SubscriptionHydrator`, `GizmoduckCandidateHydrator`, `BlockedByHydrator`, `FilteredTopicsHydrator`, `LanguageCodeHydrator`. Filters (sequential): `DropDuplicatesFilter`, `CoreDataHydrationFilter`, `AgeFilter(MAX_POST_AGE)`, `SelfTweetFilter`, `RetweetDeduplicationFilter`, `IneligibleSubscriptionFilter`, `PreviouslySeenPostsFilter`, `PreviouslySeenPostsBackupFilter`, `PreviouslyServedPostsFilter`, `MutedKeywordFilter`, `AuthorSocialgraphFilter`, `VideoFilter` (`exclude_videos` only), `TopicIdsFilter`, `NewUserTopicIdsFilter`. ### Scorers and selection Wired sequential scorers: `PhoenixScorer` (`enable` = `!has_cached_posts`), `RankingScorer` (always on), `VMRanker` (`EnableVMRanker`). `RankingScorer` writes the weighted multi-action score used by `TopKScoreSelector` (`candidate.score`, missing → `-inf`). Sibling modules `WeightedScorer`, `AuthorDiversityScorer`, and `OonScorer` exist in `home-mixer/scorers/` but are **not** in this pipeline’s `scorers` vec. ### Post-selection and side effects Post-selection hydrators: `VFCandidateHydrator`, `AdsBrandSafetyHydrator`, `AdsBrandSafetyVfHydrator`, `TweetTypeMetricsHydrator`, `FollowingRepliedUsersHydrator`, `MutualFollowJaccardHydrator`. Post-selection filters: `VFFilter`, `AncillaryVFFilter`, `DedupConversationFilter`. Side effects (spawned): `PhoenixExperimentsSideEffect` (`is_shadow_traffic`), `RerankingKafkaSideEffect` (prod and 5% sample), `RedisPostCandidateCacheSideEffect` (prod and `!has_cached_posts`), `ScoredStatsSideEffect`, `MutualFollowStatsSideEffect` (`EnableMutualFollowJaccardHydration`), `PhoenixRequestCacheSideEffect` (`EnablePhoenixRequestCacheSideEffect`). `CacheRequestInfoSideEffect` exists on disk and is **not** wired. `ScoredPostsServer` maps `selected_candidates` to proto `ScoredPost` (`tweet_id`, `author_id`, `score`, `in_network`, `served_type`, ancestors, VF reason, tweet-type metrics, brand-safety verdict, safety labels, text). Surface logging uses `in_network_only` → `ranked_following`, non-empty `topic_ids` → `topics`, excluded topics → `for_you_with_snoozed_topics`, else `for_you`. ## ForYouCandidatePipeline `CandidatePipeline`. Hydrators, filters, scorers, and both post-selection lists are empty slices — `execute` still walks those stages as no-ops. `result_size()` is unpublished `params::FOR_YOU_MAX_RESULT_SIZE`. ### Query hydrators | Hydrator | `enable` | Writes | | --- | --- | --- | | `ServedHistoryQueryHydrator` | `EnableUrtMigrationComponents` | `served_history`, recent `served_ids`, `who_to_follow_eligible` (fatigue vs `WhoToFollowFatigueHours`) | | `PastRequestTimestampsQueryHydrator` | `EnableUrtMigrationComponents` | `non_polling_timestamps` | These run **before** `ScoredPostsSource` clones the query into Phoenix, so Phoenix filters that read `served_ids` see For You hydration when `EnableUrtMigrationComponents` is on. ### Sources (parallel) | Source | `enable` | Output item | | --- | --- | --- | | `ScoredPostsSource` | always (default) | `FeedItem::Post` from Phoenix `run_pipeline` | | `AdsSource` | `EnableAdsSource && !is_preview` | `FeedItem::Ad` | | `WhoToFollowSource` | `EnableWhoToFollowModule && who_to_follow_eligible` | one `FeedItem::WhoToFollow` (max 3 users) | | `PromptsSource` | `EnablePrompts` | `FeedItem::Prompt` | | `PushToHomeSource` | `push_to_home_post_id.is_some()` | `FeedItem::PushToHome` | ### Selector `BlenderSelector` overrides `select` (it does not use default score-sort). It partitions items, blends ads with `SafeGapAdsBlender` when `AdsBlenderType == "safe_gap"` else `PartitionOrganicAdsBlender`, inserts prompts at `PROMPTS_POSITION`, inserts one Who-to-Follow module at `WHO_TO_FOLLOW_POSITION`, and pins push-to-home at index `0`. `score()` is unused (`0.0`). After `select`, `execute` still truncates the blended list to `FOR_YOU_MAX_RESULT_SIZE`. ### Side effects (spawned) `AdsInjectionLoggingSideEffect`, `PublishSeenIdsToKafkaSideEffect`, `ServedCandidatesKafkaSideEffect` (prod + `is_shadow_traffic` + URT flag), `ClientEventsKafkaSideEffect`, `ForYouResponseStatsSideEffect`, `UpdatePastRequestTimestampsSideEffect` (prod, URT flag, not polling, not `BackgroundFetch`), `UpdateServedHistorySideEffect`, `TruncateServedHistorySideEffect`. ## PipelineResult and responses Post-hydrate candidates. Phoenix: `PostCandidate`. For You: mixed `FeedItem`s from all enabled sources. Union of pre-score and post-selection removals. Empty on For You (no filters). Post-truncate, post-`finalize` list returned to the server. Fully hydrated `ScoredPostsQuery` after both query-hydrator stages. `GetDebugScoredPosts` serializes all four fields plus counts. `GetForYouFeed` returns only `selected_candidates`. `GetForYouFeedUrt` serializes those items into a URT timeline; cursor decode errors are logged and the cursor is ignored. ## Short-circuits and errors | Condition | Effect | | --- | --- | | `viewer_id == 0` | RPC fails before `execute` | | `user_id` in `TEST_USER_IDS` | Empty `ScoredPosts` / empty For You items; no `execute` | | Missing `ForYouFeedQuery.query` | `Status::invalid_argument("query must be specified")` | | Query hydrator `Err` | Field stays default; other hydrators still merge | | Source `Err` | That source contributes zero candidates | | `PhoenixSource: missing retrieval_sequence` | Phoenix retrieval dropped; Thunder and others can still fill the batch | | Hydrator / scorer length mismatch | That component’s updates are skipped | | URT cursor decode failure | Warning; request continues without cursor fields | | Side-effect `Err` | Isolated inside the spawned task; response already returned | Empty final Phoenix results increment `{PipelineName}.execute` with scope `requests/result_empty`. ## Related pages CLI flags, QueryBuilder validation, and For You versus Scored Posts entry points. Query fields, TEST_USER_IDS empty feeds, and ScoredPost / ForYouFeed / URT mapping. Phoenix and For You sources, enable predicates, and served_type. Pre-score vs post-selection filters and the length-match hydrator contract. PhoenixScorer, RankingScorer, VMRanker, and TopKScoreSelector. Source, Hydrator, Filter, Scorer, Selector, QueryHydrator, and SideEffect contracts. AdsSource, blender types, and prompt / who-to-follow / push-to-home insertion. viewer_id must be specified, TEST_USER_IDS, and missing retrieval_sequence.