# Installation > Python 3.11+, uv or pip, Git LFS for the Phoenix artifact archive, and the extract layout required before inference. - Repository: xai-org/x-algorithm - GitHub: https://github.com/xai-org/x-algorithm - Human docs: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c - Complete Markdown: https://grok-wiki.com/public/docs/xai-org-x-algorithm-23c09c39074c/llms-full.txt ## Source Files - `phoenix/pyproject.toml` - `phoenix/README.md` - `.gitattributes` - `phoenix/uv.lock` - `phoenix/run_pipeline.py` - `phoenix/artifacts/oss-phoenix-artifacts.zip` --- --- title: "Installation" description: "Python 3.11+, uv or pip, Git LFS for the Phoenix artifact archive, and the extract layout required before inference." --- The only complete local install surface in this checkout is the Phoenix Python project under `phoenix/`. `phoenix/pyproject.toml` requires Python 3.11+, names the project `grok-1` `0.1.0`, and pins `jax==0.8.1` plus `dm-haiku>=0.0.13`, `numpy>=1.26.4`, and `pyright>=1.1.408`. `phoenix/uv.lock` resolves those packages for `sys_platform == 'darwin'` and `sys_platform == 'linux'` only. End-to-end inference additionally needs the Git LFS object `phoenix/artifacts/oss-phoenix-artifacts.zip` (2,903,518,802 bytes) extracted so `run_pipeline.py` can open `retrieval/config.json`, `ranker/config.json`, the matching `*.npz` checkpoints, `sports_corpus.npz`, and `example_sequence.json`. A clone that skipped Git LFS leaves `phoenix/artifacts/oss-phoenix-artifacts.zip` as a 135-byte pointer (`version https://git-lfs.github.com/spec/v1`). `unzip` and `run_pipeline.py` both fail until `git lfs pull` replaces that pointer with the real archive. ## What this checkout installs | Surface | Installable from this tree? | What you get | |---|---|---| | `phoenix/` | Yes | JAX/Haiku retrieval + ranking, `run_pipeline.py`, unit tests | | `phoenix/run_ranker.py`, `phoenix/run_retrieval.py` | Yes, after `uv sync` | Random-init demos; no checkpoint archive required | | `phoenix/run_pipeline.py` | Yes, after LFS + extract | Frozen mini Phoenix checkpoint on `sports_corpus.npz` | | `home-mixer/`, `thunder/`, `candidate-pipeline/` | No | Rust snapshots with no `Cargo.toml`; they import unpublished `xai_*` crates | | `grox/` | No | Python snapshot with no `pyproject.toml`; `grox.main` imports missing `grox.service` and `grox.config` | `[tool.uv] environments` in `phoenix/pyproject.toml` lists only Darwin and Linux. The locked `jaxlib==0.8.1` wheels are CPU builds for `macosx_11_0_arm64`, `manylinux_2_27_aarch64`, and `manylinux_2_27_x86_64`. There is no CUDA extra, no Windows environment, and no `macosx` x86_64 `jaxlib` wheel in the lockfile. ## Prerequisites | Requirement | Constraint | |---|---| | Python | `>=3.11` (`requires-python` in `phoenix/pyproject.toml` and `phoenix/uv.lock`) | | OS | macOS (Apple Silicon for the locked JAX stack) or Linux (`aarch64` or `x86_64`) | | Package manager | [uv](https://docs.astral.sh/uv/getting-started/installation/) (preferred; uses `uv.lock`) or `pip` | | Git LFS | Tracks `*.zip` and `*.npz` via `.gitattributes` | | Disk | ~2.90 GB for the LFS zip; plan ~6 GB if you keep both the zip and the extract | | Working directory | All Phoenix commands below run from `phoenix/` | Tests do **not** need the artifact archive. They construct dummy `RecsysBatch` / embedding tensors in process. ## Clone and fetch LFS artifacts ```bash git lfs install git clone https://github.com/xai-org/x-algorithm.git cd x-algorithm ``` `.gitattributes` marks `*.zip` and `*.npz` as LFS. The published object is `phoenix/artifacts/oss-phoenix-artifacts.zip`. Individual `*.npz` checkpoints live inside that zip, not as sibling LFS files. ```bash git lfs pull # or, scoped: git lfs pull --include="phoenix/artifacts/oss-phoenix-artifacts.zip" ``` Verify the object, not the pointer: ```bash wc -c phoenix/artifacts/oss-phoenix-artifacts.zip # expect 2903518802 file phoenix/artifacts/oss-phoenix-artifacts.zip # expect a Zip archive, not "ASCII text" ``` A pointer file starts with: ```text version https://git-lfs.github.com/spec/v1 oid sha256:fbc6017d00588754e22e0c7eb2f786a008a74d309c03c8085fa2fad418a83dac size 2903518802 ``` ## Python environment Install from `phoenix/`. The lockfile package name is `grok-1`; `uv sync` creates the project environment there. ```bash cd phoenix uv sync ``` `uv sync` installs the locked runtime set and the default `dev` group (`pytest`). Runtime pins from `phoenix/uv.lock`: | Package | Locked version | Role | |---|---|---| | `jax` | `0.8.1` | Exact pin in `pyproject.toml` | | `jaxlib` | `0.8.1` | Transitive CPU wheels | | `dm-haiku` | `0.0.16` | `>=0.0.13` | | `numpy` | `2.4.1` | `>=1.26.4` | | `pyright` | `1.1.408` | Runtime dependency, not only a type-check extra | | `pytest` | `9.0.2` | `dev` group | Skip tests with `uv sync --no-dev` if you only need inference. ```bash cd phoenix python3.11 -m venv .venv source .venv/bin/activate pip install 'jax==0.8.1' jaxlib 'dm-haiku>=0.0.13' 'numpy>=1.26.4' pip install pytest # only if you will run tests ``` The Phoenix README's unpinned `pip install jax jaxlib dm-haiku numpy` can resolve a newer JAX than `0.8.1`. Match the `pyproject.toml` pin. `pyright` is listed as a project dependency but is not required to import the model modules or run `run_pipeline.py`. Import check (no artifacts required): ```bash cd phoenix uv run python -c "import jax, haiku as hk, numpy as np; print(jax.__version__, np.__version__)" ``` Expect `0.8.1` for JAX when using the lockfile. ## Extract the Phoenix archive `run_pipeline.py` does not read the zip. It opens files under `--artifacts_dir`. ```bash cd phoenix unzip artifacts/oss-phoenix-artifacts.zip -d artifacts/ ``` That command creates `artifacts/oss-phoenix-artifacts/`. Point `--artifacts_dir` at **that** directory, not at `artifacts/`. `run_pipeline.py` defaults `--artifacts_dir` to `./artifacts`. After `unzip … -d artifacts/`, `./artifacts/retrieval/config.json` does not exist. The documented invocation is `--artifacts_dir artifacts/oss-phoenix-artifacts`. ### Expected extract layout `run_pipeline.py` requires this tree as the value of `--artifacts_dir`: :::files oss-phoenix-artifacts/ retrieval/ model_params.npz # retrieval transformer + candidate tower embedding_tables.npz # user / item / author hash tables config.json # architecture + hash_params ranker/ model_params.npz # ranking transformer + action head embedding_tables.npz # user / item / author hash tables config.json # architecture + hash_params sports_corpus.npz # post_ids, candidate_representations, author_ids, optional topics example_sequence.json # user_id + history[] ::: Phoenix documents those embedding tables at about 1.4 GB each and the model param files at about 3 MB each. `run_pipeline.py` reconstructs a pad-offset unified table (`pad=65` plus `user_vocab_size` + `item_vocab_size` + `author_vocab_size`) from `user_embeddings`, `item_embeddings`, and `author_embeddings` inside each `embedding_tables.npz`. Architecture values are **not** hardcoded for the published checkpoint. `build_model_config()` reads `emb_size`, `history_seq_len`, `candidate_seq_len`, `num_heads`, `num_layers`, `num_actions`, vocab sizes, and `hash_params` from each `config.json`. ## Verify the install ```bash cd phoenix uv run pytest test_recsys_model.py test_recsys_retrieval_model.py ``` `test_recsys_model.py` covers `make_recsys_attn_mask` and related helpers. `test_recsys_retrieval_model.py` is a `unittest` module that pytest collects; it builds tiny random models (`emb_size=64`, `num_layers=1`) via `create_example_batch` / `create_example_corpus`. Neither test opens `artifacts/`. ```bash test -f artifacts/oss-phoenix-artifacts/retrieval/config.json \ && test -f artifacts/oss-phoenix-artifacts/ranker/config.json \ && test -f artifacts/oss-phoenix-artifacts/sports_corpus.npz \ && test -f artifacts/oss-phoenix-artifacts/example_sequence.json \ && echo ok ``` ```bash cd phoenix uv run run_pipeline.py --artifacts_dir artifacts/oss-phoenix-artifacts ``` Success is a `PIPELINE RESULTS — User ` table with columns `Rank`, `Score`, `Ret`, `Fav`, `Reply`, `RT`, `Dwell`, `VQV`, `Topics`, and a post URL, plus a final `Weighted score range: [low, high]` line. Defaults retrieve 200 corpus rows (`--top_k_retrieval`) and print 30 (`--top_k_display`). Random-init scripts that do **not** need the archive, once the venv exists: ```bash cd phoenix uv run run_retrieval.py uv run run_ranker.py ``` Those scripts hardcode a tiny demo transformer (`emb_size=128`, `num_layers=2`, `history_seq_len=32`, `candidate_seq_len=8`) and call `initialize()` on synthetic batches. ## `run_pipeline.py` paths and flags Directory that contains `retrieval/`, `ranker/`, `sports_corpus.npz`, and `example_sequence.json`. After the documented unzip, pass `artifacts/oss-phoenix-artifacts`. User history JSON. Default: `/example_sequence.json`. Must include `user_id` and `history[]` with `post_id`, `author_id`, and `actions`. Corpus NPZ. Default: `/sports_corpus.npz`. Required keys: `post_ids`, `candidate_representations`, `author_ids`. Optional: `topics`. How many corpus rows to keep after the retrieval dot product, capped at corpus length. How many ranked rows to print. ```bash cd phoenix uv run run_pipeline.py \ --artifacts_dir artifacts/oss-phoenix-artifacts \ --sequence_file artifacts/oss-phoenix-artifacts/example_sequence.json \ --corpus_file artifacts/oss-phoenix-artifacts/sports_corpus.npz \ --top_k_retrieval 200 \ --top_k_display 30 ``` ```text ======================================================================================================================== PIPELINE RESULTS — User History: 3 items | Corpus: posts Retrieved top 200 → Ranked by engagement model ======================================================================================================================== Rank Score Ret Fav Reply RT Dwell VQV Topics Post URL ------------------------------------------------------------------------------------------------------------------------ 1 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 … https://x.com/a/status/ … Weighted score range: [low, high] ======================================================================================================================== ``` `load_model_params()` reads each `model_params.npz` into a Haiku params dict keyed by `/`-separated module paths. `load_embedding_table()` returns the raw NPZ mapping. ## What does not install Home Mixer (`home-mixer/main.rs`) starts through `XServiceBuilder` and unpublished crates (`xai_home_mixer`, `xai_home_mixer_proto`, `xai_x_rpc`, `xai_x_service_builder`, `xai_dark_traffic`, `xai_stringcenter`). Thunder (`thunder/main.rs`) needs `xai_http_server` plus Kafka/Strato wiring. There is no `Cargo.toml` or `rust-toolchain` in this tree. Grox (`grox/main.py`) calls `Engine`, `Dispatcher`, and `GrpcServer`, but `grox.service` and `grox.config` are not in the snapshot, and there is no Grox package manifest. Treat those directories as source reference, not a second install path. ## Common failures | Symptom | Cause | Fix | |---|---|---| | `unzip` reports not a zip / empty archive | LFS pointer still on disk (135 bytes, ASCII) | `git lfs install && git lfs pull`, then re-check `wc -c` | | `FileNotFoundError` for `retrieval/config.json` or `ranker/config.json` | `--artifacts_dir` is `./artifacts` after `unzip -d artifacts/` | Pass `--artifacts_dir artifacts/oss-phoenix-artifacts` | | `FileNotFoundError` for `sports_corpus.npz` / `example_sequence.json` | Incomplete extract, or overrides point elsewhere | Confirm the four required leaves listed above | | `uv sync` refuses the platform | Windows, or a Darwin/Linux combo outside `[tool.uv] environments` | Use macOS arm64 or Linux; do not expect a locked Windows env | | `jaxlib` wheel missing on Intel macOS | Lockfile ships `macosx_11_0_arm64` only for Darwin | Use Apple Silicon or Linux | | Tests pass, pipeline missing | Tests never load LFS objects | Extract the archive before `run_pipeline.py` | | Want Home Mixer / Thunder / Grox locally | No crate/package manifests; unpublished modules | See runtime boundaries; this checkout cannot build those services | `phoenix/.gitignore` only ignores `__pycache__/`. Extracted `artifacts/oss-phoenix-artifacts/` is untracked local data sitting next to the LFS zip. ## Next Extract the archive, run `run_pipeline.py`, and treat the ranked table as the success signal. How retrieval and ranking load checkpoints, encode `example_sequence.json`, and score `sports_corpus.npz`. `uv run pytest` targets and the attention-mask / retrieval assertions they encode. What this checkout can execute versus Home Mixer, Thunder, and Grox snapshots. LFS pointer failures, wrong `--artifacts_dir`, and other setup errors in more detail.