# Development and testing
> Running integration suites under `tests/suite`, spawning the binary via `test_app_server`, debug env hooks, notify-capture bins, and `just test -p codex-app-server` expectations.
- Repository: openai/codex
- GitHub: https://github.com/openai/codex
- Human docs: https://grok-wiki.com/public/docs/openai-codex-c82680b15ec1
- Complete Markdown: https://grok-wiki.com/public/docs/openai-codex-c82680b15ec1/llms-full.txt
## Source Files
- `tests/all.rs`
- `tests/common/test_app_server.rs`
- `tests/common/lib.rs`
- `tests/suite/mod.rs`
- `src/main.rs`
- `src/bin/notify_capture.rs`
- `BUILD.bazel`
---
---
title: "Development and testing"
description: "Running integration suites under `tests/suite`, spawning the binary via `test_app_server`, debug env hooks, notify-capture bins, and `just test -p codex-app-server` expectations."
---
The `codex-app-server` crate validates protocol behavior through one Cargo integration binary (`tests/all.rs`), a large `tests/suite` tree, shared helpers in `app_test_support`, and unit tests colocated under `src/`. Most end-to-end coverage spawns the real `codex-app-server` binary over stdio JSON-RPC; a smaller set uses the in-process API to avoid subprocess teardown races.
## Test layout
```text
codex-app-server/
├── src/ # library + main; #[cfg(test)] unit modules
├── src/bin/notify_capture.rs # test-only notify sink binary
├── tests/
│ ├── all.rs # single integration test crate → mod suite
│ ├── common/ # app_test_support (TestAppServer, mocks)
│ └── suite/
│ ├── mod.rs # auth, strict_config, fuzzy_file_search, v2, …
│ └── v2/ # v2 JSON-RPC integration modules (~70 files)
└── BUILD.bazel # Bazel: 16-way shard, long timeout, no-sandbox
```
| Layer | Entry | Role |
| --- | --- | --- |
| Integration (stdio subprocess) | `tests/all.rs` → `tests/suite/**` | Full binary, JSON-RPC over stdin/stdout |
| Integration helpers | `tests/common` (`app_test_support`) | Spawn server, mock model API, rollouts, auth fixtures |
| Unit / in-process | `src/**` `#[cfg(test)]`, `src/in_process.rs` | Fast checks without a child process when appropriate |
| Direct CLI | `tests/suite/strict_config.rs` | Spawns `codex-app-server` via `std::process::Command` |
`tests/all.rs` declares only `mod suite;`, so Cargo builds one integration test binary that includes every module registered in `tests/suite/mod.rs`. The v2 surface is grouped under `tests/suite/v2/mod.rs` (threads, turns, config RPC, MCP, plugins, websocket transports, and related areas).
## Running tests locally
From the repository root, `just` recipes run with working directory `codex-rs`. Install [cargo-nextest](https://nexte.st/) if it is not already available:
```bash
cargo install --locked cargo-nextest
```
```bash
just test -p codex-app-server
```
This invokes `cargo nextest run --no-fail-fast` with `RUST_MIN_STACK=8388608` (8 MiB), then runs `just bench-smoke`. Prefer `just test` over raw `cargo test` so behavior matches CI defaults.
```bash
just test -p codex-app-server -- turn_start
just test -p codex-app-server -- suite::v2::initialize::
```
Nextest accepts standard filters after `--`.
From `codex-rs`:
```bash
just fmt
just fix -p codex-app-server # before large changes / PRs
```
Do not re-run tests after `fmt` or `fix` unless you changed behavior again.
`codex-rs/.config/nextest.toml` puts all `package(codex-app-server) & kind(test)` integration tests in the `app_server_integration` group with `max-threads = 1`, because each case spawns a fresh app-server subprocess. Library unit tests in `codex_app_server` still run in parallel.
### Protocol-only changes
When you change `codex-app-server-protocol` types or wire shapes:
```bash
just write-app-server-schema
just write-app-server-schema --experimental # if experimental API fixtures change
just test -p codex-app-server-protocol
```
## `TestAppServer` harness
`TestAppServer` in `tests/common/test_app_server.rs` is the primary integration client. It resolves the `codex-app-server` binary with `codex_utils_cargo_bin::cargo_bin`, pipes stdio, and speaks newline-delimited JSON-RPC.
### Default child environment
On spawn, the harness typically sets:
| Setting | Value | Purpose |
| --- | --- | --- |
| `CODEX_HOME` | Temp or test dir | Isolated config and rollouts |
| `RUST_LOG` | `warn` | Quieter default logs |
| `CODEX_APP_SERVER_MANAGED_CONFIG_PATH` | `{codex_home}/managed_config.toml` | Avoid host `/etc` managed config |
| CLI args | `--disable-plugin-startup-tasks-for-tests` | Skip plugin startup background work |
It removes `CODEX_INTERNAL_ORIGINATOR_OVERRIDE_ENV_VAR` from the child environment so tests do not inherit host overrides.
### Constructors
| Method | When to use |
| --- | --- |
| `TestAppServer::new` | Default: managed-config path under `codex_home`, plugin startup disabled |
| `new_without_managed_config` | Sets `CODEX_APP_SERVER_DISABLE_MANAGED_CONFIG=1` on the child |
| `new_without_managed_config_with_env` | Same, plus extra env tuples |
| `new_with_env` / `new_with_args` | Override or remove env vars; add CLI flags |
| `new_with_program_and_env` | Point at a custom binary path (e.g. websocket tests) |
| `new_with_plugin_startup_tasks` | Enable real plugin startup (omit the disable flag) |
Env overrides use `(key, Some(value))` to set and `(key, None)` to remove a variable from the child only.
### Handshake and RPC helpers
- `initialize()` — `initialize` RPC with `clientInfo.name = "codex-app-server-tests"`, `capabilities.experimentalApi: true`, then `initialized` notification.
- Typed `send_*_request` methods — one per v2 method (e.g. `send_thread_start_request`, `send_turn_start_request`).
- Stream readers — `read_stream_until_response_message`, `read_stream_until_notification_message`, `read_stream_until_request_message` (server-initiated approvals), with a `pending_messages` buffer for out-of-order notifications.
- `interrupt_turn_and_wait_for_aborted` — Sends `turn/interrupt` and waits for `turn/completed`; avoids nextest `LEAK` when tests end mid-turn.
`DEFAULT_CLIENT_NAME` is `"codex-app-server-tests"`. `DISABLE_PLUGIN_STARTUP_TASKS_ARG` is the string `"--disable-plugin-startup-tasks-for-tests"`.
### Teardown and logging
`Drop` closes stdin, polls briefly for graceful exit, then `start_kill()` with a bounded wait so the child is reaped before nextest teardown. Child stderr is forwarded to the test process as `[mcp stderr] …`. Request/response traffic is logged with `eprintln!` when debugging a failing case.
## Debug-only hooks (integration tests)
These exist so tests can steer config and login without writing to system paths. They are compiled for **debug** builds only unless noted.
### Environment variables
Points the server at a test-managed `managed_config.toml` instead of the host default. `TestAppServer::new` sets this to `{codex_home}/managed_config.toml`. `strict_config.rs` and config RPC tests set it explicitly when layering managed config is required.
When set to `1`, `true`, or `yes` (case variants accepted), `main` uses `LoaderOverrides::without_managed_config_for_tests()`. `TestAppServer::new_without_managed_config*` sets this on the child.
Overrides the ChatGPT login issuer URL in `account_processor` (debug only). Account integration tests set this to a wiremock base URL.
### Hidden CLI flag
Debug-only, hidden. Skips `PluginStartupTasks` so integration tests do not wait on plugin background startup. Passed by default from `TestAppServer::new`; omit via `new_with_plugin_startup_tasks` when testing plugin startup behavior.
`src/main.rs` wires managed-config env vars into `LoaderOverrides` before `run_main_with_transport_options`.
## `codex-app-server-test-notify-capture`
Cargo declares a second binary:
```toml
[[bin]]
name = "codex-app-server-test-notify-capture"
path = "src/bin/notify_capture.rs"
```
The program takes **two** arguments: output path, then JSON payload string. It writes to `{path}.tmp`, syncs, and atomically renames into the final path so concurrent readers never see a partial file.
Turn-notification tests (for example `turn_start_notify_payload_includes_initialize_client_name` in `tests/suite/v2/initialize.rs`) register this binary in `config.toml` under `notify` alongside a JSON output file. After `turn/completed`, the test reads the captured JSON and asserts fields such as `client` from `initialize` `clientInfo.name`.
Resolve the binary in tests with:
```rust
codex_utils_cargo_bin::cargo_bin("codex-app-server-test-notify-capture")?
```
## `app_test_support` helpers
The `tests/common` crate (`app_test_support`) is re-exported for all suite modules. Common utilities:
| Category | Examples |
| --- | --- |
| Model API mocks | `create_mock_responses_server_sequence`, `create_mock_responses_server_repeating_assistant`, SSE helpers from `core_test_support::responses` |
| Config | `write_mock_responses_config_toml`, `write_mock_responses_config_toml_with_chatgpt_base_url` |
| Auth | `write_chatgpt_auth`, `ChatGptAuthFixture`, `encode_id_token` |
| Rollouts | `create_fake_rollout`, `rollout_path` |
| Response parsing | `to_response::(JSONRPCResponse)` |
| Analytics | `start_analytics_events_server` |
Many v2 tests follow the same shape: `TempDir` → write `config.toml` pointing at a wiremock `/v1/responses` server → `TestAppServer::new` → `initialize()` → RPC + `timeout(DEFAULT_READ_TIMEOUT, …)` on stream reads. `DEFAULT_READ_TIMEOUT` is often 60 seconds where subprocess or auth RPC latency matters under CI load.
## In-process vs subprocess
Most integration tests use the subprocess harness because they exercise real stdio transport and process lifecycle.
Use **in-process** (`in_process::start` from the library) when subprocess teardown would flake leak detection—for example `mcp_resource_read_returns_error_for_unknown_thread` in `tests/suite/v2/mcp_resource.rs`, which documents that choice explicitly.
Unit tests under `src/` cover processors, serialization, thread state, and tracing (`message_processor_tracing_tests.rs` uses `serial_test::serial` where shared global state requires it).
## CI and Bazel expectations
`BUILD.bazel` configures the crate via `codex_rust_crate`:
| Setting | Value |
| --- | --- |
| Integration test target | `app-server-all-test` (from `tests/all.rs`) |
| Shard count | 16 |
| Timeout | `long` |
| Tags | `no-sandbox` |
| Extra test data | `//codex-rs/bwrap:bwrap` |
Bazel comments note that even one shard can run long and that failed shards retry up to three times; high shard count limits total CI time when a shard fails.
## Flaky tests and platform guards
Some modules use `#[ignore]`, `#[cfg_attr(windows, ignore = …)]`, or `serial_test::serial` for auth flows that mutate global login state. Check the module before assuming a failure is a product regression. Windows-specific process and sandbox tests may be ignored or given longer nextest timeouts via `codex-rs/.config/nextest.toml` overrides (mostly for other packages; app-server integration still benefits from the global `app_server_integration` serialization).
## Adding a new integration test
Add a file under `tests/suite/v2/` (or an existing top-level suite module) and register it in the parent `mod.rs`.
Use `tempfile::TempDir` for `CODEX_HOME`, wiremock for HTTP model providers, and `TestAppServer::new_without_managed_config` when managed-config layering is not part of the scenario.
Call `initialize()`, then typed `send_*` helpers. Use `read_stream_until_*` with timeouts; call `interrupt_turn_and_wait_for_aborted` if the test leaves a turn running.
```bash
just test -p codex-app-server -- your_test_name
```
Returning from a test while a turn is still in flight without `turn/interrupt` and a terminal `turn/completed` can produce intermittent nextest `LEAK` reports. Prefer `interrupt_turn_and_wait_for_aborted` for in-flight turn scenarios.
## Related pages
Framing, request ids, and notification ordering—the same wire rules `TestAppServer` implements.
Regenerate protocol fixtures after changing v2 types exercised by these tests.
Why tests call `initialize` with `experimentalApi: true` by default.
Runtime errors surfaced by integration scenarios (overload, init, strict config).