# Hash embeddings

> HashConfig, linear-congruential ID hashing, pad-offset unified tables, and how RecsysBatch hashes are looked up before the transformer.

- Repository: xai-org/x-algorithm
- GitHub: https://github.com/xai-org/x-algorithm
- Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c
- Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt

## Source Files

- `phoenix/recsys_model.py`
- `phoenix/run_pipeline.py`
- `phoenix/runners.py`
- `phoenix/recsys_retrieval_model.py`
- `phoenix/README.md`

---

---
title: "Hash embeddings"
description: "HashConfig, linear-congruential ID hashing, pad-offset unified tables, and how RecsysBatch hashes are looked up before the transformer."
---

Phoenix never feeds raw `user_id`, `post_id`, or `author_id` values into the transformer. `run_pipeline.py` maps each ID through a training-matched linear-congruential hash, shifts the result into a pad-offset region of a unified table, and passes the looked-up vectors as `RecsysEmbeddings`. `PhoenixModel.build_inputs` and `PhoenixRetrievalModel.build_user_representation` only consume those pre-looked-up tensors plus the hash indices that mark padding.

<Info>
Retrieval and ranking keep **separate** `config.json` `hash_params` and **separate** `embedding_tables.npz` files. Hash a given ID with the stage that owns the table you index. Mixing retrieval hashes into the ranker table (or the reverse) is a silent lookup error.
</Info>

## Contract

The model API is two objects, not an ID list:

| Object | Owns | Does not own |
|---|---|---|
| `RecsysBatch` | Hash indices, history actions, product-surface IDs, optional timestamps / IP hashes | Embedding vectors |
| `RecsysEmbeddings` | Vectors already gathered from the hash tables | ID hashing |

Callers look up first, then call the Haiku module. `RecsysInferenceRunner.rank` and `RecsysRetrievalInferenceRunner.encode_user` both document this: hashes go in the batch, embeddings are already gathered.

```mermaid
flowchart LR
  subgraph ids [Raw IDs]
    U["user_id"]
    P["post_id"]
    A["author_id"]
  end

  subgraph hasher ["run_pipeline.build_hash_functions"]
    LCG["_hash_ids LCG"]
    OFF["pad-offset<br/>user / item / author"]
  end

  subgraph tables ["build_unified_emb_table"]
    T["pad 65 + user + item + author"]
  end

  subgraph tensors [Model inputs]
    B["RecsysBatch hashes"]
    E["RecsysEmbeddings"]
  end

  subgraph reduce ["block_*_reduce"]
    R["proj_mat_1 / 2 / 3"]
  end

  subgraph xf [Transformer]
    X["user + history + candidates"]
  end

  U --> LCG
  P --> LCG
  A --> LCG
  LCG --> OFF
  OFF --> B
  OFF --> T
  T --> E
  B --> R
  E --> R
  R --> X
```

## HashConfig

`HashConfig` is the width of the hash dimension. It is not the hash function itself. Published `config.json` supplies the counts; `PhoenixModelConfig` and `PhoenixRetrievalModelConfig` both default to a new `HashConfig()` when the field is omitted.

<ParamField body="num_user_hashes" type="int" default="2">
Hash functions per user. Shapes `user_hashes` as `[B, num_user_hashes]` and `user_embeddings` as `[B, num_user_hashes, D]`.
</ParamField>

<ParamField body="num_item_hashes" type="int" default="2">
Hash functions per post. Shapes `history_post_hashes` / `candidate_post_hashes` as `[B, S|C, num_item_hashes]`.
</ParamField>

<ParamField body="num_author_hashes" type="int" default="2">
Hash functions per author. Shapes `history_author_hashes` / `candidate_author_hashes` as `[B, S|C, num_author_hashes]`.
</ParamField>

<ParamField body="num_ip_hashes" type="int" default="0">
Optional IP hash width. When `user_ip_embeddings` is set and this value is greater than 0, `block_user_reduce` sums the IP vectors and adds them to the projected user embedding. The published pipeline leaves this at 0.
</ParamField>

`PhoenixModelConfig.use_ip_address` exists as a config flag (`False` by default) and is not read by `build_inputs`. IP injection is gated only by `num_ip_hashes` and a non-`None` `user_ip_embeddings` tensor.

The published mini checkpoint uses 2 hashes per entity and 1,000,000 rows per split table (`user` / `item` / `author`). Demo scripts `run_ranker.py` and `run_retrieval.py` hard-code the same `HashConfig(2, 2, 2)` but skip published hashing entirely.

## Linear-congruential ID hashing

Published inference reconstructs the training hash in `_hash_ids`. Arithmetic is **numpy `int64`**, including wrapping overflow, so Python arbitrary-precision ints will not match.

For each ID and each hash function `j`:

```text
raw = (id * scales[j] + biases[j]) % modulus
hash = 0                              if id == 0
hash = (raw % (num_buckets - 1)) + 1  otherwise
```

| Rule | Effect |
|---|---|
| `id == 0` | Output is 0 for every hash function. Unused history slots and missing IDs stay padding. |
| Nonzero ID | Output is in `[1, num_buckets - 1]`. Bucket 0 is reserved. |
| `num_buckets` | The matching vocab size: `user_vocab_size`, `item_vocab_size`, or `author_vocab_size`. |
| `scales` / `biases` length | One pair per hash function. Must equal `num_*_hashes`. |

`build_hash_functions(config)` then wraps `_hash_ids` with the pad offset (next section) and returns `(hash_user, hash_item, hash_author)`.

Published `config.json` keys consumed here:

| Key | Role |
|---|---|
| `hash_params.user_hash_scales` / `user_biases` / `user_modulus` | User LCG |
| `hash_params.item_hash_scales` / `item_biases` / `item_modulus` | Post LCG |
| `hash_params.author_hash_scales` / `author_biases` / `author_modulus` | Author LCG |
| `user_vocab_size` / `item_vocab_size` / `author_vocab_size` | `num_buckets` and table widths |
| `num_user_hashes` / `num_item_hashes` / `num_author_hashes` | Copied into `HashConfig` |

The numeric `scales`, `biases`, and `modulus` values live only in the extracted artifact `config.json` files (Git LFS). This checkout does not embed those constants in source.

<Warning>
`run_pipeline.py` builds **two** hasher triples: one from `retrieval/config.json`, one from `ranker/config.json`. The same `user_id` is hashed twice. Use `hash_user` with the retrieval table and `rank_hash_user` with the ranker table.
</Warning>

## Pad-offset unified table

Exported `embedding_tables.npz` stores three split arrays: `user_embeddings`, `item_embeddings`, `author_embeddings`. `build_unified_emb_table` concatenates them behind a **hard-coded** pad of 65 zero rows so a single integer index can address every entity type.

```text
index:  0 .............. 64 | 65 .......... 65+uv-1 | 65+uv .... 65+uv+iv-1 | 65+uv+iv .... 65+uv+iv+av-1
region: pad (zeros)         | user_embeddings       | item_embeddings       | author_embeddings
offset: 0                   | pad                   | pad + uv              | pad + uv + iv
```

`pad = 65` is not a `config.json` key. Both `build_hash_functions` and `build_unified_emb_table` hard-code it.

After LCG, nonzero hashes are shifted:

| Hasher | Index written |
|---|---|
| `hash_user` | `0` if LCG is 0, else `h + 65` |
| `hash_item` | `0` if LCG is 0, else `h + 65 + user_vocab_size` |
| `hash_author` | `0` if LCG is 0, else `h + 65 + user_vocab_size + item_vocab_size` |

Because LCG never emits 0 for a nonzero ID, the first row of each split table (`table[pad]`, `table[pad+uv]`, `table[pad+uv+iv]`) is not selected on the published path. Padding IDs stay at unified index `0` (a zero vector).

Table length is `65 + user_vocab_size + item_vocab_size + author_vocab_size` by `emb_size`, `float32`.

## RecsysBatch hash tensors

`create_dummy_batch_from_config` is the shape contract used at runner init:

| Field | Shape | Dtype in dummy / pipeline |
|---|---|---|
| `user_hashes` | `[B, num_user_hashes]` | `int32` |
| `history_post_hashes` | `[B, history_seq_len, num_item_hashes]` | `int32` |
| `history_author_hashes` | `[B, history_seq_len, num_author_hashes]` | `int32` |
| `candidate_post_hashes` | `[B, candidate_seq_len, num_item_hashes]` | `int32` |
| `candidate_author_hashes` | `[B, candidate_seq_len, num_author_hashes]` | `int32` |
| `user_ip_hashes` | optional | unused on the published path |

Matching `RecsysEmbeddings` ranks add `emb_size` on the last axis, for example `history_post_embeddings`: `[B, S, num_item_hashes, D]`.

Validity is **only** the first hash function:

| Reducer | Valid when |
|---|---|
| `block_user_reduce` | `user_hashes[:, 0] != 0` |
| `block_history_reduce` | `history_post_hashes[:, :, 0] != 0` |
| `block_candidate_reduce` | `candidate_post_hashes[:, :, 0] != 0` |
| Retrieval candidate tower | same first-hash test on `candidate_post_hashes` |

A row whose first hash is 0 is padding even if later hash functions are nonzero.

## Lookup before the transformer

### Published pipeline

`run_pipeline.py` gathers with numpy advanced indexing on the unified table:

```python
user_hashes = hash_user(np.array([user_id], dtype=np.uint64))
hist_post_h = hash_item(history_post_ids).reshape(1, hist_len, -1)
hist_author_h = hash_author(history_author_ids).reshape(1, hist_len, -1)

emb_batch = RecsysEmbeddings(
    user_embeddings=ret_emb[user_hashes],
    history_post_embeddings=ret_emb[hist_post_h],
    history_author_embeddings=ret_emb[hist_author_h],
    candidate_post_embeddings=...,
    candidate_author_embeddings=...,
)
```

History IDs are written into zero-initialized `uint64` buffers of length `history_seq_len`. Unused tail slots remain `0`, hash to `0`, and look up the pad row.

Ranking walks retrieved posts in chunks of `candidate_seq_len`. A short last chunk is `np.pad(..., constant 0)` on the hash tensors so the ranker still sees a full candidate axis; those pad hashes stay 0 and are masked out by `block_candidate_reduce`.

### Reduce, then transformer

`PhoenixModel.build_inputs` does not index any embedding table. It:

1. Embeds `history_product_surface` and `candidate_product_surface` from a learned `product_surface_embedding_table` (vocab default 16).
2. Projects multi-hot `history_actions` through `action_projection`.
3. Calls `block_user_reduce` → concatenates the `num_user_hashes` vectors to `[B, 1, num_user_hashes * D]`, multiplies by `proj_mat_1`, optional IP add.
4. Calls `block_history_reduce` → concatenates flattened post hashes, author hashes, action embedding, product-surface embedding (plus optional dwell / age), multiplies by `proj_mat_3`.
5. Calls `block_candidate_reduce` → concatenates flattened post hashes, author hashes, product-surface embedding (plus optional post-age), multiplies by `proj_mat_2`.
6. Concatenates `[user, history, candidates]` on the sequence axis and hands that to the transformer.

Retrieval's user tower reuses `block_user_reduce` and `block_history_reduce`, then mean-pools the transformer outputs. The candidate tower does **not** use `block_candidate_reduce`. It concatenates `candidate_post_embeddings` and `candidate_author_embeddings` on the hash axis (`axis=2`) and runs `CandidateTower` (SiLU MLP when `enable_linear_proj=True`, which `build_model_config` sets for retrieval).

During retrieval user encoding, `run_pipeline.py` still materializes the candidate tower by concatenating 64 ghost-negative slots (`N_neg = 64`) with hash tensors of ones and zero embeddings. Those ones are only for Haiku parameter creation; corpus scoring uses precomputed `sports_corpus.npz` `candidate_representations`, not a live candidate-tower pass over the corpus.

### Demo path (no LCG)

`create_example_batch` samples hash indices in `[1, num_*_embeddings)` and synthesizes random embedding tensors of the matching rank. Hash `0` is reserved for padding; history tails are zeroed. `run_ranker.py` and `run_retrieval.py` use this path. They do not read `hash_params` or `embedding_tables.npz`.

<Note>
Phoenix unit tests (`test_recsys_model.py`, `test_recsys_retrieval_model.py`) also go through `create_example_batch`. They do not assert LCG arithmetic or the pad-65 layout. Those live only in `run_pipeline.py`.
</Note>

## Reconstruct from artifacts

<Steps>
<Step title="Extract the published tables">
After Git LFS checkout, unzip `phoenix/artifacts/oss-phoenix-artifacts.zip` so both `retrieval/` and `ranker/` contain `config.json` and `embedding_tables.npz`.
</Step>
<Step title="Build one hasher and one table per stage">

```python
ret_cfg = json.load(open("artifacts/oss-phoenix-artifacts/retrieval/config.json"))
rank_cfg = json.load(open("artifacts/oss-phoenix-artifacts/ranker/config.json"))

hash_user, hash_item, hash_author = build_hash_functions(ret_cfg)
rank_hash_user, rank_hash_item, rank_hash_author = build_hash_functions(rank_cfg)

ret_emb = build_unified_emb_table(load_embedding_table(".../retrieval/embedding_tables.npz"), ret_cfg)
rank_emb = build_unified_emb_table(load_embedding_table(".../ranker/embedding_tables.npz"), rank_cfg)
```
</Step>
<Step title="Hash IDs, then index">
Pass `uint64` IDs into the hasher for that stage. Index the matching unified table. Reshape post/author hashes to `[1, seq, num_hashes]` before building `RecsysBatch` / `RecsysEmbeddings`.
</Step>
<Step title="Verify padding">
History shorter than `history_seq_len` must hash to all-zero rows. Candidate chunks shorter than `candidate_seq_len` must pad hashes with 0, not with a repeated last ID. After lookup, pad rows are the zero vector at unified index 0.
</Step>
</Steps>

## Constraints

| Constraint | Source behavior |
|---|---|
| Hash `0` is padding | LCG short-circuits on `id == 0`; reducers mask on first-hash `!= 0` |
| Do not reuse one hasher across stages | Retrieval and ranker each have their own `hash_params` and tables |
| `pad` is 65, not configurable | Changing it without rebuilding both hashers and the table misaligns every index |
| `int64` modular multiply | Required to match training overflow |
| First hash decides validity | Later hash functions cannot un-pad a row |
| Model does not look up IDs | Forgetting the gather and passing empty / random `RecsysEmbeddings` trains or scores the wrong vectors |
| Published vocabs are 1M × 3 | README mini config; confirm against the extracted `*_vocab_size` keys before allocating |

## Failure modes

| Symptom | Likely cause |
|---|---|
| All history embeddings are ~0 | History IDs were left as 0, or hashes were not pad-offset so they landed in the pad region |
| Ranked scores look like a different user | Retrieval hasher used against the ranker table, or the reverse |
| `IndexError` on `table[hashes]` | Hasher from one vocab size applied to a table built with another, or `pad` / offset skipped |
| Candidates in a short last chunk all score similarly | Last IDs were repeated instead of padding hashes with 0; first-hash validity then treats pads as real items |
| `proj_mat_*` shape errors at init | `HashConfig.num_*_hashes` does not match the last-but-one embedding axis produced by lookup |
| LFS / missing `config.json` | `hash_params` never load; see [Troubleshooting](/troubleshooting) |

## Next

<CardGroup>
<Card title="Phoenix model configuration" href="/phoenix-model-configuration">
`PhoenixModelConfig`, `PhoenixRetrievalModelConfig`, `RecsysBatch`, `RecsysEmbeddings`, and published `config.json` keys.
</Card>
<Card title="Candidate isolation" href="/candidate-isolation">
How the transformer attends after hash embeddings are reduced: candidates see user + history + self only.
</Card>
<Card title="Customize a user sequence" href="/customize-user-sequence">
`example_sequence.json` fields, history padding to `history_seq_len`, and the IDs that enter these hashers.
</Card>
<Card title="Run the inference pipeline" href="/run-inference-pipeline">
Load both checkpoints, hash the example sequence, retrieve from `sports_corpus.npz`, and rank.
</Card>
</CardGroup>