# ScoredPostsQuery and gRPC

> ScoredPostsQuery fields, QueryBuilder defaults, TEST_USER_IDS empty responses, and ScoredPost / ForYouFeed / URT response mapping.

- Repository: xai-org/x-algorithm
- GitHub: https://github.com/xai-org/x-algorithm
- Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c
- Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt

## Source Files

- `home-mixer/models/query.rs`
- `home-mixer/server.rs`
- `home-mixer/scored_posts_server.rs`
- `home-mixer/for_you_server.rs`
- `home-mixer/main.rs`
- `home-mixer/models/candidate.rs`
- `home-mixer/models/user_features.rs`

---

---
title: "ScoredPostsQuery and gRPC"
description: "ScoredPostsQuery fields, QueryBuilder defaults, TEST_USER_IDS empty responses, and ScoredPost / ForYouFeed / URT response mapping."
---

Home Mixer serves ranked posts through two gRPC services on the same process: `ScoredPostsService` and `ForYouFeedService`. Both convert `xai_home_mixer_proto::ScoredPostsQuery` into the Rust `ScoredPostsQuery` in `home-mixer/models/query.rs` via `QueryBuilder`, then either short-circuit empty for `params::TEST_USER_IDS` or run a candidate pipeline. `xai_home_mixer_proto`, `crate::params`, Gizmoduck, and the URT serializer live in unpublished crates; this checkout shows the request and response mapping, not a runnable server.

```mermaid
sequenceDiagram
    participant Client
    participant QB as QueryBuilder
    participant SPS as ScoredPostsServer
    participant FYS as ForYouFeedServer
    participant Phoenix as PhoenixCandidatePipeline
    participant ForYou as ForYouCandidatePipeline

    Client->>QB: proto ScoredPostsQuery
    alt viewer_id == 0
        QB-->>Client: INVALID_ARGUMENT viewer_id must be specified
    else valid viewer
        QB->>QB: Gizmoduck ViewerData (200 ms) + feature switches
        QB-->>SPS: Rust ScoredPostsQuery
        QB-->>FYS: Rust ScoredPostsQuery
        alt user_id in TEST_USER_IDS
            SPS-->>Client: ScoredPostsResponse scored_posts = []
            FYS-->>Client: ForYouFeedResponse items = [] / empty URT
        else live request
            SPS->>Phoenix: execute(query)
            Phoenix-->>SPS: selected PostCandidates
            SPS-->>Client: ScoredPostsResponse
            FYS->>ForYou: execute(query)
            ForYou->>SPS: ScoredPostsSource.run_pipeline
            ForYou-->>FYS: Vec FeedItem
            FYS-->>Client: ForYouFeedResponse or ForYouFeedUrtResponse
        end
    end
```

## gRPC process

`home-mixer/main.rs` starts `HomeMixerServer` as service name `home-mixer` with mTLS from the environment, gRPC reflection from `xai_home_mixer_proto::FILE_DESCRIPTOR_SET`, feature switches at `params::FS_PATH`, and dark-traffic reject layers.

| Flag | Default | Role |
|------|---------|------|
| `--grpc-port` | `50051` | gRPC listen port |
| `--metrics-port` | `9090` | Metrics / profiling HTTP |
| `--shard-coordinate` | `-1` | Shard ordinal; `< 0` means no `ShardCoordinate` |
| `--shard-total-size` | `500` | Used only when a shard is set |
| `--datacenter` | `atla` | Feature-switch recipient + client wiring |
| `--otel-endpoint` | empty | Optional OTel export |

Both services accept and send Gzip and Zstd, and cap decode/encode size at `params::MAX_GRPC_MESSAGE_SIZE`. `HomeMixerServer::build` constructs one shared `QueryBuilder` (feature switches, decider, datacenter, prod Gizmoduck), one `PhoenixCandidatePipeline`, a `ScoredPostsServer` wrapping that pipeline, then a `ForYouCandidatePipeline` that calls back into the same `ScoredPostsServer`.

| Service | RPC | Request | Response |
|---------|-----|---------|----------|
| `ScoredPostsService` | `GetScoredPosts` | `ScoredPostsQuery` | `ScoredPostsResponse` |
| `ScoredPostsService` | `GetDebugScoredPosts` | `DebugScoredPostsQuery` | `DebugScoredPostsResponse` |
| `ForYouFeedService` | `GetForYouFeed` | `ForYouFeedQuery` | `ForYouFeedResponse` |
| `ForYouFeedService` | `GetForYouFeedUrt` | `ForYouFeedQuery` | `ForYouFeedUrtResponse` |

`GetForYouFeed` requires `ForYouFeedQuery.query`. `GetDebugScoredPosts` unwraps a missing nested query to proto default, which then fails `viewer_id` validation. Every successful RPC injects B3 trace headers on the response.

## Proto request fields `QueryBuilder` reads

The proto crate is unpublished. The following fields are the ones `QueryBuilder::build` and the For You URT handler actually consume.

<ParamField body="viewer_id" type="u64" required>
Mapped to `ScoredPostsQuery.user_id`. `0` is rejected as `INVALID_ARGUMENT` with `viewer_id must be specified`.
</ParamField>

<ParamField body="client_app_id" type="i32">
Copied onto the query and used as the feature-switch recipient `client_app_id`.
</ParamField>

<ParamField body="country_code" type="string">
Copied onto the query and the feature-switch recipient.
</ParamField>

<ParamField body="language_code" type="string">
Copied onto the query and the feature-switch recipient.
</ParamField>

<ParamField body="seen_ids" type="repeated u64">
Viewer-seen tweet IDs. Copied as-is.
</ParamField>

<ParamField body="served_ids" type="repeated u64">
Initial served IDs. For You later overwrites this from served-history hydration when that hydrator is enabled.
</ParamField>

<ParamField body="in_network_only" type="bool">
Forced `true` if Gizmoduck returns `allow_for_you_recommendations == Some(false)`.
</ParamField>

<ParamField body="is_bottom_request" type="bool">
Copied into `ScoredPostsQuery`. URT cursor decode can overwrite it.
</ParamField>

<ParamField body="is_preview" type="bool">
Copied into `ScoredPostsQuery.is_preview`.
</ParamField>

<ParamField body="exclude_videos" type="bool">
Copied into `ScoredPostsQuery.exclude_videos`.
</ParamField>

<ParamField body="topic_ids" type="repeated i64">
Non-empty marks a topics request (`is_topic_request()`). More than 6 IDs is a bulk topic request.
</ParamField>

<ParamField body="excluded_topic_ids" type="repeated i64">
Snoozed/excluded topics. Non-empty is `has_excluded_topics()`.
</ParamField>

<ParamField body="request_id" type="u64">
Passed through `resolve_request_id`. A separate `prediction_id` is always newly generated.
</ParamField>

<ParamField body="push_to_home_post_id" type="u64">
`0` becomes `None` via `non_zero`.
</ParamField>

<ParamField body="device_status" type="DeviceStatus">
Missing status becomes proto default. Fields copied: `ip_address`, `user_agent`, `time_zone`, `device_network_type`, `client_version`, `device_id`, `mobile_device_id`, `mobile_device_ad_id`. Time zone and network type go through `timezone_string_to_enum` / `network_type_string_to_enum`.
</ParamField>

For You RPCs also read these proto fields **after** `QueryBuilder::build`:

| Proto field | Applied on |
|-------------|------------|
| `request_context` | `query.request_context` |
| `is_polling` | `query.is_polling` |
| `cursor` | Decoded with `cursor_utils::decode_ordered_cursor`; sets `cursor`, `is_bottom_request`, `is_top_request` |

`GetDebugScoredPosts` also applies `feature_switch_overrides: map<string, string>` after recipient matching.

<RequestExample>
```json
{
  "viewer_id": 123456789,
  "client_app_id": 3033300,
  "country_code": "US",
  "language_code": "en",
  "seen_ids": [1, 2],
  "served_ids": [],
  "in_network_only": false,
  "is_bottom_request": false,
  "is_preview": false,
  "exclude_videos": false,
  "topic_ids": [],
  "excluded_topic_ids": [],
  "request_id": 0,
  "push_to_home_post_id": 0,
  "device_status": {
    "ip_address": "203.0.113.10",
    "user_agent": "XAndroid",
    "time_zone": "America/New_York",
    "device_network_type": "WIFI",
    "client_version": "10.0",
    "device_id": "dev",
    "mobile_device_id": "",
    "mobile_device_ad_id": ""
  }
}
```
</RequestExample>

That JSON is reconstructed from `QueryBuilder` usage, not a checked-in `.proto`.

## QueryBuilder construction

`QueryBuilder::build` is the only path from proto to the live Rust query.

<Steps>
<Step title="Reject missing viewer">
`viewer_id == 0` returns `Status::invalid_argument("viewer_id must be specified")` before any hydration.
</Step>
<Step title="Force-sample traced users">
If `viewer_id` is in `params::TRACE_USER_IDS`, B3 sampling is forced. The ID list is not in this checkout.
</Step>
<Step title="Fetch Gizmoduck viewer data">
`gizmoduck_client.get_viewer_data(viewer_id)` is awaited with a **200 ms** timeout. Error or timeout becomes `ViewerData::default()`.
</Step>
<Step title="Force ranked-following when For You is disallowed">
`in_network_only = proto.in_network_only || allow_for_you_recommendations == Some(false)`.
</Step>
<Step title="Evaluate feature switches">
Recipient fields: `user_id`, `country`, `language`, `client_app_id`, custom `datacenter`, `account_age_days` from `days_since_creation(viewer_id)`, `has_phone_number`, and `user_roles` when non-empty. Debug overrides call `override_fs` per key.
</Step>
<Step title="Allocate IDs and construct the query">
`prediction_id = generate_request_id()`, `request_id = resolve_request_id(proto.request_id)`, `is_shadow_traffic = is_sampled(request_id, 0.5)`, then `ScoredPostsQuery::new(...)`.
</Step>
</Steps>

Fields taken from `ViewerData` when the Gizmoduck call succeeds: `roles`, `muted_keywords`, `follower_count`, `subscription_level`, `age_in_years`, `has_phone_number`, `allow_for_you_recommendations`. The client struct itself is unpublished; those names are the ones `QueryBuilder` reads.

`QueryBuilder::mock()` builds empty feature switches, an empty decider store, datacenter `"mock"`, and `MockGizmoduckClient`.

## Live `ScoredPostsQuery` fields

The gRPC servers, Phoenix pipeline, and For You pipeline all use `home-mixer/models/query.rs`. `home-mixer/candidate_pipeline/query.rs` defines a narrower leftover type (`user_id: i64`, string `request_id`, no sequences) that the servers do not use.

### Set at construction

| Field | Construction default / source |
|-------|-------------------------------|
| `user_id` | proto `viewer_id` |
| `client_app_id`, `country_code`, `language_code` | proto |
| `seen_ids`, `served_ids` | proto |
| `in_network_only` | proto **or** Gizmoduck For You opt-out |
| `is_bottom_request` | proto; URT cursor may overwrite |
| `is_top_request` | `false`; URT `CursorType::TOP` sets `true` |
| `params` | feature-switch match (+ debug overrides) |
| `decider` | `Some(decider.with_recipient(viewer_id))` |
| `user_roles` | Gizmoduck roles, else empty |
| `user_features.muted_keywords` | Gizmoduck |
| `user_features.follower_count` | Gizmoduck |
| `topic_ids`, `excluded_topic_ids`, `exclude_videos` | proto |
| `request_id`, `prediction_id` | resolved / generated |
| `request_time_ms` | `current_time_ms()` |
| `ip_address`, `user_agent`, `time_zone`, `device_network_type`, `client_version`, `device_id`, `mobile_device_id`, `mobile_device_ad_id` | `device_status` |
| `subscription_level` | Gizmoduck |
| `is_shadow_traffic` | 50% sample on `request_id` |
| `is_preview` | proto |
| `user_age_in_years` | Gizmoduck `age_in_years` |
| `push_to_home_post_id` | proto, `0` → `None` |

All other struct fields start empty / `None` / `false` in `ScoredPostsQuery::new`.

Helpers on the live type:

- `is_topic_request()` — `!topic_ids.is_empty()`
- `is_bulk_topic_request()` — `topic_ids.len() > 6`
- `has_excluded_topics()` — `!excluded_topic_ids.is_empty()`
- `has_new_user_topic_ids()` — `!new_user_topic_ids.is_empty()`

`PipelineQuery` exposes `params` and `decider`. `GetTwitterContextViewer` builds a viewer with `user_id`, `client_application_id`, `request_country_code`, and `request_language_code`.

### Filled by query hydrators

Phoenix (`PhoenixCandidatePipeline`) hydrates scoring/retrieval sequences, social-graph IDs, cached posts, mutual-follow minhash, demographics, Grok topics, starter packs, impression bloom filters, IP location, and inferred gender. For You (`ForYouCandidatePipeline`) hydrates served history and non-polling timestamps, then calls Phoenix through `ScoredPostsSource`.

| Hydrator | Writes |
|----------|--------|
| `ScoringSequenceQueryHydrator` | `scoring_sequence`, `columnar_scoring_sequence` |
| `RetrievalSequenceQueryHydrator` | `retrieval_sequence`, `columnar_retrieval_sequence` |
| `BlockedUserIdsQueryHydrator` | `user_features.blocked_user_ids` |
| `MutedUserIdsQueryHydrator` | `user_features.muted_user_ids` |
| `FollowedUserIdsQueryHydrator` | `user_features.followed_user_ids` |
| `SubscribedUserIdsQueryHydrator` | `user_features.subscribed_user_ids` |
| `CachedPostsQueryHydrator` | `cached_posts`, `has_cached_posts` |
| `MutualFollowQueryHydrator` | `viewer_minhash` |
| `UserDemographicsQueryHydrator` | `user_demographics` |
| `FollowedGrokTopicsQueryHydrator` | `followed_grok_topics` (`[bool; 32]`), optionally `new_user_topic_ids` |
| `FollowedStarterPacksQueryHydrator` | `followed_starter_packs` (`[bool; 20]`) |
| `InferredGrokTopicsQueryHydrator` | `inferred_grok_topics` (`[bool; 32]`) |
| `ImpressionBloomFilterQueryHydrator` | `bloom_filter_entries` |
| `IpQueryHydrator` | `ip_location` |
| `UserInferredGenderQueryHydrator` | `user_inferred_gender`, `user_inferred_gender_score` |
| `ServedHistoryQueryHydrator` | `served_history`, `served_ids`, `who_to_follow_eligible` |
| `PastRequestTimestampsQueryHydrator` | `non_polling_timestamps` |

`UserFeatures` also stores `muted_keywords` and `follower_count` from Gizmoduck at construction. The struct is a Strato `MValCodec` with camelCase serde names.

<Warning>
`ImpressedPostsQueryHydrator` exists and writes `impressed_post_ids`, but Phoenix constructs it as `_impressed_posts_hydrator` and does not put it in the hydrator list. `impressed_post_ids` therefore stays empty on the live path.
</Warning>

`UserFeaturesQueryHydrator` and `UserActionSeqQueryHydrator` target the leftover `candidate_pipeline::query::ScoredPostsQuery` and are not registered on either live pipeline.

## TEST_USER_IDS empty responses

Both servers check `params::TEST_USER_IDS.contains(&query.user_id)` **after** query construction and **before** pipeline execute. The ID list is unpublished.

| Entry point | Short-circuit result |
|-------------|----------------------|
| `ScoredPostsServer::run_pipeline` | `PipelineOutput { scored_posts: [], pipeline_result: PipelineResult::empty() }` |
| `ForYouFeedServer::get_for_you_feed` | `ForYouFeedOutput { items: [] }` |
| `get_for_you_feed_urt` | Calls `get_for_you_feed`, so URT is serialized from an empty item list |

`PipelineResult::empty()` uses `ScoredPostsQuery::default()` as the stored query (not the incoming query) and empty retrieved/filtered/selected vectors. Debug JSON for a test user therefore does not include the real request. For You URT still runs `urt::make_urt_timeline` on the empty list.

<Info>
`TRACE_USER_IDS` only forces B3 sampling. It does not empty the feed. Empty feeds for listed test users are expected, not a scoring failure.
</Info>

## Scored Posts surface classification

`log_request_info` labels each Scored Posts request for logs and `ScoredPostsServer.product_surface`:

| Condition | Surface |
|-----------|---------|
| `in_network_only` | `ranked_following` |
| else non-empty `topic_ids` | `topics` |
| else `has_excluded_topics()` | `for_you_with_snoozed_topics` |
| else | `for_you` |

Single-topic requests also increment `ScoredPosts.topic` with `type=request` and `type=empty` when selection is empty.

## Response mapping

### `ScoredPost`

`candidates_to_scored_posts` maps each selected `PostCandidate` to `xai_home_mixer_proto::ScoredPost`. Missing optionals become `0`, `false`, empty bytes, or proto default.

| `ScoredPost` field | Source |
|--------------------|--------|
| `tweet_id` | `candidate.tweet_id` |
| `author_id` | `candidate.author_id` |
| `retweeted_tweet_id` | `retweeted_tweet_id.unwrap_or(0)` |
| `retweeted_user_id` | `retweeted_user_id.unwrap_or(0)` |
| `in_reply_to_tweet_id` | `in_reply_to_tweet_id.unwrap_or(0)` |
| `score` | `candidate.score.unwrap_or(0.0) as f32` — **not** `weighted_score` |
| `in_network` | `in_network.unwrap_or(false)` |
| `served_type` | `served_type as i32`, else `0` |
| `last_scored_timestamp_ms` | `last_scored_at_ms.unwrap_or(0)` |
| `prediction_request_id` | `prediction_request_id.unwrap_or(0)` |
| `ancestors` | cloned |
| `screen_names` | `get_screen_names()` — author and retweeted-author screen names only |
| `visibility_reason` | `FilteredReason` converted into proto, if present |
| `tweet_type_metrics` | bytes, default empty |
| `following_replied_user_ids` | cloned |
| `brand_safety_verdict` | `BrandSafetyVerdict` as `i32`; missing → `MediumRisk` (`3`) |
| `safety_label_types` | mapped `SafetyLabelType` values; unmapped types dropped |
| `tweet_text` | cloned |

`BrandSafetyVerdict` values: `Unspecified = 0`, `Safe = 1`, `LowRisk = 2`, `MediumRisk = 3`.

Mapped safety labels include NSFW / NSFA / gore / PDNA / Grok SFA-NSFA variants listed in `safety_label_to_proto`. Anything else is omitted.

`PostCandidate` fields that stay off the proto include `phoenix_scores`, `weighted_score`, quote IDs, video durations, engagement counts, topic IDs, `has_media`, `language_code`, and `mutual_follow_jaccard`.

<ResponseExample>
```json
{
  "scored_posts": [
    {
      "tweet_id": 1001,
      "author_id": 42,
      "retweeted_tweet_id": 0,
      "retweeted_user_id": 0,
      "in_reply_to_tweet_id": 0,
      "score": 1.25,
      "in_network": true,
      "served_type": 1,
      "last_scored_timestamp_ms": 0,
      "prediction_request_id": 0,
      "ancestors": [],
      "screen_names": { "42": "example" },
      "tweet_type_metrics": "",
      "following_replied_user_ids": [],
      "brand_safety_verdict": 3,
      "safety_label_types": [],
      "tweet_text": ""
    }
  ]
}
```
</ResponseExample>

`GetDebugScoredPosts` returns the same `scored_posts` plus `debug_json`: serialized `query`, `retrieved_candidates`, `filtered_candidates`, `selected_candidates`, and count stats. Serialization failure becomes `{"error": "Failed to serialize debug info: ..."}`.

### `ForYouFeed` items

`ScoredPostsSource` wraps each `ScoredPost` as:

```rust
FeedItem {
    position: 0,
    item: Some(feed_item::Item::Post(post)),
}
```

For You sources add other `feed_item::Item` variants. `BlenderSelector` partitions them, blends ads, then inserts prompts, who-to-follow, and push-to-home:

| Variant | Source | Placement |
|---------|--------|-----------|
| `Post(ScoredPost)` | `ScoredPostsSource` → Phoenix | Organic list, then ads blender |
| `Ad(AdIndexInfo)` | `AdsSource` | `SafeGapAdsBlender` or `PartitionOrganicAdsBlender` |
| `Prompt` | `PromptsSource` | Inserted at index `i` with `position = PROMPTS_POSITION` |
| `WhoToFollow` | `WhoToFollowSource` | First module only; insert index `WHO_TO_FOLLOW_POSITION - 1` |
| `PushToHome` | `PushToHomeSource` | Pinned at index `0` |

`WhoToFollowSource` enables only when `EnableWhoToFollowModule` is on **and** `who_to_follow_eligible` is true (served-history fatigue). `GetForYouFeed` returns `ForYouFeedResponse { items }` as the blender's selected `FeedItem`s.

### URT

`GetForYouFeedUrt` reuses `get_for_you_feed`, then:

1. `urt::make_urt_timeline(items, cursor, request_context, client_app_id, viewer_id, language_code, country_code)`
2. `xai_urt_thrift::serialize_binary` → `ForYouFeedUrtResponse.urt`

Empty `country_code` is passed as `None`. Cursor decode failure logs a warning and ignores the cursor (`is_bottom_request` / `is_top_request` stay at QueryBuilder values). Serialize failure is `Status::internal("failed to serialize URT: …")`. The URT builder itself is unpublished (`crate::util::urt`).

## Errors and empty-feed cases

| Symptom | Cause |
|---------|--------|
| `INVALID_ARGUMENT: viewer_id must be specified` | proto `viewer_id == 0`, including debug with a missing nested query |
| `INVALID_ARGUMENT: query must be specified` | `ForYouFeedQuery.query` absent |
| Empty `scored_posts` / `items` / URT | `user_id` in `TEST_USER_IDS`, or a live pipeline that selected nothing |
| Missing roles / muted keywords / age | Gizmoduck timeout or error (`ViewerData::default()`) |
| Forced `in_network_only` | `allow_for_you_recommendations == Some(false)` |
| URT cursor ignored | `decode_ordered_cursor` error |
| `INTERNAL: failed to serialize URT` | Thrift serialize failure after a successful pipeline |
| `impressed_post_ids` always empty | hydrator not registered |

This checkout cannot start `HomeMixerServer`: `params`, proto stubs, Gizmoduck, and URT crates are not published here.

## Next

<CardGroup>
<Card title="Assemble a Home Mixer request" href="/assemble-home-mixer-request">
CLI flags, viewer_id validation, and For You versus Scored Posts entry points.
</Card>
<Card title="For You request lifecycle" href="/request-lifecycle">
`CandidatePipeline.execute` stages and how `ForYouCandidatePipeline` wraps Phoenix.
</Card>
<Card title="Filters and hydrators" href="/filters-and-hydrators">
Query hydrators that fill `ScoredPostsQuery` and candidate hydrators that must preserve order.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
`viewer_id must be specified`, `TEST_USER_IDS` empty feeds, and other Home Mixer failures.
</Card>
</CardGroup>