# Run regression tests

> Clone-only scripts/run-regression against vendored PG 18.3 SQL/expected files, psql client requirements, selective test names, and optional isolation/auth/recovery TAP harnesses.

- Repository: malisper/pgrust
- GitHub: https://github.com/malisper/pgrust
- Human docs: https://grok-wiki.com/public/docs/malisper-pgrust-7e0c8e82bc21
- Complete Markdown: https://grok-wiki.com/public/docs/malisper-pgrust-7e0c8e82bc21/llms-full.txt

## Source Files

- `scripts/run-regression`
- `scripts/run-pg-regress`
- `scripts/run-pg-regress-serial`
- `scripts/run-pg-isolation`
- `scripts/run-auth-tap`
- `scripts/run-recovery-tap`
- `crates/backend/test_regress/src/lib.rs`

---

---
title: "Run regression tests"
description: "Clone-only scripts/run-regression against vendored PG 18.3 SQL/expected files, psql client requirements, selective test names, and optional isolation/auth/recovery TAP harnesses."
---

`scripts/run-regression` is the clone-only main-suite runner: it boots `target/release/postgres` with pgrust `--initdb` against `vendor/postgres-18.3/share`, feeds vendored SQL from `vendor/postgres-18.3/regress/sql` through a PostgreSQL 18 `psql` client, and diffs each result against `vendor/postgres-18.3/regress/expected`. No PostgreSQL source tree (`PG_SRC`) and no C `initdb` install are required for that path. Optional scripts (`run-pg-regress-serial`, `run-pg-isolation`, `run-auth-tap`, `run-ldap-tap`, `run-recovery-tap`) still need an external PG 18.3 source and C install prefix.

## What the clone-only runner needs

| Input | Role |
|-------|------|
| `target/release/postgres` | Server and `--initdb` (override with `PGRUST_BIN`) |
| `vendor/postgres-18.3/share` | Bootstrap catalogs, tz data, samples (`-L` for initdb) |
| `vendor/postgres-18.3/regress/{sql,expected,data,parallel_schedule}` | Suite inputs and oracle |
| PostgreSQL 18 `psql` | Client only (only external dependency) |

<Note>
`scripts/run-pg-regress` is a thin wrapper that `exec`s `scripts/run-pg-regress-serial`. That serial harness is **not** clone-only: it requires `PG_SRC` and `PG_INSTALL` and uses C `initdb` / `psql` / `createdb`. Prefer `scripts/run-regression` for a fresh clone.
</Note>

## Prerequisites

1. Release binary:

```bash
cargo build --release --locked --bin postgres
```

2. Vendored tree present under `vendor/postgres-18.3/` (checked for `share/postgres.bki`, `regress/sql`, `regress/expected`, `regress/parallel_schedule`).

3. A PostgreSQL 18 `psql` client, resolved in order:

| Priority | Source |
|----------|--------|
| 1 | `PGRUST_PSQL` |
| 2 | `command -v psql` |
| 3 | `/tmp/pgrust_pginstall/bin/psql` |

If none is found, the script exits with install hints (`brew install libpq` on macOS, or set `PGRUST_PSQL`).

## Run the main suite

<Steps>
<Step title="Full parallel_schedule order">
```bash
PGRUST_BIN="$PWD/target/release/postgres" \
scripts/run-regression
```
With no arguments, the runner flattens every `test:` line from `vendor/postgres-18.3/regress/parallel_schedule` into a **serial** order (groups are not run in parallel).
</Step>
<Step title="Selective tests">
```bash
scripts/run-regression boolean int4 join
```
Named tests run only those SQL files. If `test_setup` is not in the argument list, it is auto-prepended so fixtures (`onek` / `tenk` / …) exist.
</Step>
<Step title="Stop early while debugging">
```bash
STOP_AFTER=join scripts/run-regression
```
Stops after the named test finishes.
</Step>
</Steps>

### Environment (clone-only runner)

| Variable | Default | Purpose |
|----------|---------|---------|
| `PGRUST_BIN` | `$repo/target/release/postgres` | pgrust server binary |
| `PGRUST_PSQL` | auto-detect | `psql` client path |
| `PGPORT` | `55434` | postmaster port |
| `OUTDIR` | temp under `$TMPDIR` / `/tmp` | logs, outputs, `_status.tsv` |
| `STOP_AFTER` | unset | halt after this test name |
| `RUST_MIN_STACK` | `33554432` | Rust stack floor |
| `PGTZ` | `America/Los_Angeles` | match oracle timezone |
| `PGDATESTYLE` | `Postgres, MDY` | match oracle date style |
| `PGOPTIONS` | `-c intervalstyle=postgres_verbose` | match interval output |

Client locale is forced toward C (`LANG` / `LC_MESSAGES`, with several `LC_*` unset) so diffs stay stable.

### Server process settings applied by the runner

The postmaster is started with (among others):

- `-c listen_addresses=` (Unix socket only)
- `-k` short socket dir under `/tmp/pgr.XXXXXX` (macOS `sun_path` limit)
- `-c io_method=sync` (required; default worker IO is unported)
- `-c max_stack_depth=60000`
- `-c checkpoint_timeout=3600 -c autovacuum=on -c max_logical_replication_workers=0`
- `ulimit -s 65520` when the shell allows it

Init uses:

```bash
"$PGRUST_BIN" --initdb -D "$DATADIR" -L "$SHARE" \
  --no-locale --encoding UTF8 -U postgres
```

## Execution model

```text
run-regression
  ├─ validate vendor + PGRUST_BIN + psql
  ├─ --initdb -L vendor/.../share
  ├─ postmaster (io_method=sync, short socket)
  ├─ CREATE DATABASE regression
  ├─ for each test in schedule / args:
  │    psql < sql/T.sql  > OUTDIR/T.out
  │    best-of expected/T.out | T_1.out | T_2.out
  │    status: PASS | FAIL | PSQL_FAIL | CRASHED
  └─ summary PASS=n/total … ; exit 1 if any non-PASS
```

| Status | Meaning |
|--------|---------|
| `PASS` | exact match vs best expected file (0-line `diff`) |
| `FAIL` | output differs; first ~40 diff lines printed |
| `PSQL_FAIL` | `psql` non-zero exit |
| `CRASHED` | postmaster no longer answers `SELECT 1` after the test |

psql is invoked with `-X -a -q` and variables:

- `HIDE_TABLEAM=on`, `HIDE_TOAST_COMPRESSION=on`
- `abs_srcdir` → vendored `regress/`
- `abs_builddir` → `OUTDIR` (and `OUTDIR/results` is pre-created for COPY / largeobject)
- `libdir='$libdir'`, `dlsuffix='.so'` (path need not exist on disk; see below)

### Vendored layout

:::files
vendor/postgres-18.3/
  share/                 # install-time share/postgresql (postgres.bki, timezone, …)
  regress/
    sql/                 # ~230 SQL scripts
    expected/            # oracle .out (+ optional _1 / _2 variants)
    data/                # COPY fixtures (onek.data, tenk.data, …)
    parallel_schedule    # test: groups (flattened serially by the runner)
  isolation/             # specs + expected + isolation_schedule (not clone-only)
:::

`vendor/postgres-18.3/README.md` states isolation files are for a separate `pg_isolation_regress` runner and are **not** covered by the clone-only script.

## Regress helpers as a builtin library

`test_setup.sql` creates C-language functions via `AS '$libdir/regress', '<symbol>'`. pgrust cannot `dlopen` a real `regress.so`. Instead `crates/backend/test_regress` ports the helpers and registers library name `regress` with `dfmgr_seams::register_builtin_library` from `test_regress::init_seams()`, which `seams-init` calls at process startup.

`scripts/run-regression` still defines `:libdir` / `:dlsuffix` so the constructed path is well-formed; the dynamic loader strips directory and suffix and resolves basename `regress` in-process.

Ported symbols include (non-exhaustive): `binary_coercible`, `regress_setenv`, `test_atomic_ops`, encoding helpers used by `conversion.sql` / `encoding.sql`, `widget_in` / `widget_out`, `get_environ`, and others returned by the crate’s `lookup` table. Unknown symbols resolve as if missing from `regress.so`.

## Optional harnesses (need C PG 18.3)

These are for deeper suites. They share patterns: pgrust as the **server**, C tools for client/bootstrap, and `io_method=sync` injected somewhere.

| Script | Requires | Default port / notes |
|--------|----------|----------------------|
| `scripts/run-pg-regress` | `PG_SRC`, `PG_INSTALL` | wraps serial harness |
| `scripts/run-pg-regress-serial` | `PG_SRC`, `PG_INSTALL` | default `PGPORT=55433`; C initdb + real `$PG_LIBDIR` for `libdir` |
| `scripts/run-pg-isolation` | `PG_SRC`, `PG_INSTALL`, built `pg_isolation_regress` | default `PGPORT=55434`; `--use-existing` against a running pgrust; optional `PG_BUILD` / `PG_ISOLATION_REGRESS` |
| `scripts/run-auth-tap` | `PG_SRC`, `PG_INSTALL`, `prove`, IPC::Run | fake install; default tests `t/003_peer.pl` … `t/007_pre_auth.pl` |
| `scripts/run-ldap-tap` | same pattern | LDAP TAP under `src/test/ldap` |
| `scripts/run-recovery-tap` | same + more C tools | default archive-recovery group; `PG_TEST_TIMEOUT_DEFAULT` default `90` |

### TAP “fake install” contract

Auth / LDAP / recovery TAP scripts:

1. Build `$INST/bin` with symlinks to `$PG_INSTALL/bin/*`.
2. **Copy** C `pg_ctl` and **copy** pgrust as `postgres` (not symlinks) so `find_other_exec` keeps realpath inside `$INST/bin` and launches pgrust, not C postgres.
3. Self-check: `strings` on resolved postgres must contain `PGRUST_TRACE`.
4. Inject via `TEMP_CONFIG`:

```text
io_method = sync
max_stack_depth = 7000kB
```

5. Export capability flags so Perl `skip_all` does not treat unset env as fatal (`with_icu=no`, `with_ssl=openssl_no`, …). Auth also sets `enable_injection_points=no`.
6. Point `PG_REGRESS` at the C tree’s `pg_regress` binary for Cluster.pm `--config-auth`.

Cluster **bootstrap** intentionally still uses C `initdb` (which runs C `postgres --boot` via realpath escape); only the long-running server is pgrust.

### Isolation example

```bash
export PG_SRC=/path/to/postgres-18.3
export PG_INSTALL=/path/to/pg-install
# optional: PG_ISOLATION_REGRESS=/path/to/pg_isolation_regress
scripts/run-pg-isolation
# or: scripts/run-pg-isolation some_spec_name
```

### Recovery TAP example

```bash
export PG_SRC=/path/to/postgres-18.3
export PG_INSTALL=/tmp/pgrust_pginstall
export PGRUST_BIN="$PWD/target/release/postgres"
scripts/run-recovery-tap
# or: scripts/run-recovery-tap t/002_archiving.pl
```

## Expected outputs and success signals

Clone-only runner stderr includes:

```text
regression output: <OUTDIR>
psql client:       <path>
socket dir:        /tmp/pgr.XXXXXX
running pgrust --initdb ...
```

Per-test TSV lines (also in `$OUTDIR/_status.tsv`):

```text
test_name<TAB>STATUS<TAB>diff_lines<TAB>best_expected_path
```

Final summary:

```text
PASS=n/total  FAIL=…  PSQL_FAIL=…  CRASHED=…
```

Exit code `0` only when every status is `PASS`. README documents a verified launch result matching Postgres expected output across 46k+ regression queries; treat that as a project milestone signal, not a guarantee of your local host.

## Failure modes

| Symptom | Likely cause |
|---------|----------------|
| `pgrust postgres not found` | Build release binary or set `PGRUST_BIN` |
| `no psql client found` | Install PG 18 client or set `PGRUST_PSQL` |
| `missing vendored file/dir` | Incomplete checkout of `vendor/postgres-18.3` |
| `pgrust --initdb failed` | Inspect `$OUTDIR/initdb.log` (share path / flags) |
| `postmaster did not start` / died | Inspect `$OUTDIR/postmaster.log`; often missing `io_method=sync` or stack limits if you reimplemented the boot line |
| Many `FAIL` with locale/tz noise | Do not override `PGTZ` / date style / intervalstyle away from runner defaults |
| `could not access file` for regress helpers | Builtin `regress` not registered (build without seams-init path) or bad `:libdir` expansion |
| TAP times out / measures C postgres | Fake-install regressed (symlink-only `pg_ctl`/`postgres`); self-check should fail without `PGRUST_TRACE` |
| TAP false FAIL on skip tests | Unset `with_*` / `enable_injection_points` under `warnings FATAL` |

## Related pages

<CardGroup>
<Card title="Build from source" href="/build-from-source">
Produce `target/release/postgres` and bake share paths.
</Card>
<Card title="Test harness environment reference" href="/test-harness-reference">
Full env defaults for run-regression, serial, isolation, and TAP scripts.
</Card>
<Card title="Compatibility and status" href="/compatibility">
Regression-oracle model and non-production boundaries.
</Card>
<Card title="Seams and builtin libraries" href="/seams-and-builtins">
How `regress` and contrib replace `dlopen`.
</Card>
<Card title="Run the server" href="/run-server">
Required `io_method=sync` and stack settings outside the harness.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Missing psql, stack overflows, share path, and boot failures.
</Card>
</CardGroup>