# Evaluate enrichment output

> Score enrichment runs with dynamic golden-free metrics or golden-based eval: structural validity, hallucination checks, fact recall, consistency across runs, and report artifacts.

- Repository: GoogleCloudPlatform/knowledge-catalog
- GitHub: https://github.com/GoogleCloudPlatform/knowledge-catalog
- Human docs: https://grok-wiki.com/public/docs/googlecloudplatform-knowledge-catalog-9cee6ee3cba5
- Complete Markdown: https://grok-wiki.com/public/docs/googlecloudplatform-knowledge-catalog-9cee6ee3cba5/llms-full.txt

## Source Files

- `agents/enrichment/eval/__main__.py`
- `agents/enrichment/eval/dynamic_eval.py`
- `agents/enrichment/eval/golden_eval.py`
- `agents/enrichment/eval/metrics.py`
- `agents/enrichment/eval/goldens/README.md`
- `agents/enrichment/eval/goldens/TEMPLATE.json`

---

---
title: "Evaluate enrichment output"
description: "Score enrichment runs with dynamic golden-free metrics or golden-based eval: structural validity, hallucination checks, fact recall, consistency across runs, and report artifacts."
---

The `agents/enrichment/eval` package scores enrichment agent output from `agents/enrichment/`. Run `python -m eval` from `agents/enrichment/` to score an existing output directory (`catalog/` plus `trajectory.json`) or to execute golden cases end-to-end with `--run`. Scores are normalized 0–1 internally and displayed 0–100 in the terminal scorecard and Markdown reports.

<Info>
Judge-based metrics use Vertex AI via Application Default Credentials. Set `GOOGLE_CLOUD_PROJECT` (or pass `--project` in `--run` mode) and run `gcloud auth application-default login`. Without auth, deterministic metrics still run; judge metrics show `n/a`.
</Info>

## Evaluation modes

| Mode | Command pattern | Input | Output |
|------|-----------------|-------|--------|
| **Score (dynamic)** | `--output-dir <dir>` | Agent output dir with `catalog/` and `trajectory.json` | Scorecard + `eval_report.md` in the output dir |
| **Score (golden)** | `--output-dir <dir> --golden <file>` | Same, plus a golden JSON answer key | Scorecard + report in `$TMPDIR/kc_golden_eval_reports/` |
| **Run case** | `--run --goldens <file> --project <p>` | Golden with a `run` block | Agent runs N times, then scores each; reports in a timestamped batch folder |

```text
agents/enrichment/
├── eval/
│   ├── __main__.py          # CLI entry: python -m eval
│   ├── dynamic_eval.py      # Golden-free scoring
│   ├── golden_eval.py       # Golden-based scoring
│   ├── metrics.py           # Deterministic + LLM-as-judge metrics
│   ├── aggregate.py         # Multi-run roll-up + consistency
│   ├── runner.py            # --run agent orchestration
│   ├── loaders.py           # catalog/ + trajectory.json readers
│   └── goldens/             # Bundled cases + TEMPLATE.json
└── src/agent_runner.py      # Spawned by --run
```

## Prerequisites

<Steps>
<Step title="Install dependencies">

From `agents/enrichment/`:

```bash
pip install -r eval/requirements.txt
```

For `--run` (which spawns the agent), also install agent dependencies and build `kcmd`:

```bash
pip install -r src/requirements.txt
cd ../mdcode && npm run build
```

</Step>

<Step title="Configure Vertex AI auth">

```bash
export GOOGLE_CLOUD_PROJECT=<your-project>
gcloud auth application-default login
```

`--run` sets `GOOGLE_CLOUD_PROJECT` and `GOOGLE_GENAI_USE_VERTEXAI=True` from `--project`.

</Step>

<Step title="Produce or locate agent output">

Score mode expects an output directory containing:

- `catalog/` — generated Metadata-as-Code (entry YAML, `.overview.md`, optional `.queries.md`)
- `trajectory.json` — tool calls, responses, token usage, latency, `agent_type`

This is the same `--output_dir` passed to `agent_runner.py`.

</Step>
</Steps>

## Score an existing run

### Dynamic (golden-free) eval

Dynamic eval needs no reference answers. It grounds hallucination checks in `trajectory.json` tool responses and scores structural validity, performance, groundedness, and rubric dimensions.

```bash
cd agents/enrichment
python -m eval --output-dir /tmp/enrich_out
python -m eval --output-dir /tmp/enrich_out --model gemini-2.5-pro
```

<Check>
Verification: the terminal prints a scorecard with metrics out of 100, and `eval_report.md` appears next to `trajectory.json` with full untruncated rationales.
</Check>

### Golden-based eval

Pass `--golden` to compare output against a hand-authored answer key. Golden eval runs the full dynamic metric set plus golden-specific metrics (concept recall, fact recall, section coverage, and others depending on mode).

```bash
python -m eval --output-dir /tmp/enrich_out --golden eval/goldens/supply_chain.json
python -m eval --output-dir /tmp/enrich_out --golden eval/goldens/thelook_ecommerce.json --persona analyst
```

Score several goldens at once:

```bash
python -m eval --output-dir /tmp/enrich_out \
  --goldens eval/goldens/supply_chain.json,eval/goldens/phone_services.json
```

Golden reports land in `$TMPDIR/kc_golden_eval_reports/` as `golden_report_<golden>__<run>.md`.

## Run golden cases end-to-end

`--run` generates Metadata-as-Code via the agent (using the golden's `run` block), repeats each case `--runs` times, scores every run, and aggregates results.

```bash
python -m eval --run --goldens eval/goldens/thelook_ecommerce.json \
  --project <your_gcp_project> --model gemini-2.5-pro --runs 3
```

Run multiple bundled cases:

```bash
python -m eval --run --project <p> --goldens \
  eval/goldens/thelook_ecommerce.json,eval/goldens/financial_services.json,\
eval/goldens/phone_services.json,eval/goldens/supply_chain.json
```

Dry-run to inspect the plan without executing:

```bash
python -m eval --run --goldens eval/goldens/supply_chain.json --project <p> --dry-run
```

### Bundled runnable goldens

| Golden | Mode | Setup |
|--------|------|-------|
| `thelook_ecommerce.json` | table | Copies `bigquery-public-data.thelook_ecommerce` into your project |
| `financial_services.json` | doc | Grounds on `eval/corpora/financial_services` |
| `phone_services.json` | doc | Grounds on `eval/corpora/phone_services` |
| `supply_chain.json` | doc | Grounds on `eval/corpora/supply_chain` |

Doc-mode goldens use `{project}.global.kc-eval-<name>` entry groups; `{project}` is replaced at run time.

## Metrics reference

Scores are 0–1 internally. The scorecard and reports display 0–100. `None` means the metric self-skipped (excluded from the average).

### Dynamic metrics (always attempted)

| Metric | Type | Gated | Description |
|--------|------|-------|-------------|
| `structural_validity` | Deterministic | Yes | Entry YAML parses, required fields present, entry type matches mode, overviews are clean Markdown |
| `perf` | Deterministic | No | Token usage, latency, output size — report-only, always passes |
| `hallucination_free` | Judge | Yes | Fraction of extracted factual claims grounded in retrieved source (+ table schema in table mode) |
| `redundancy_index` | Judge | Yes | Novel synthesis vs tautological schema restatement |
| `disambiguation_efficacy` | Judge | Yes | Entry grain and purpose explicit enough to distinguish from similar entries |
| `absence_of_contradictions` | Judge | Yes | No conflicting join keys, enums, metrics, or freshness across entries |

### Golden-specific metrics (when declared)

| Metric | Type | Mode | Golden field | Description |
|--------|------|------|--------------|-------------|
| `concept_recall` | Judge | doc | `expected_topics` | Expected concepts produced as entries (semantic match) |
| `concept_precision` | Judge | doc | `expected_topics`, `acceptable_extra_concepts` | Produced entries map to expected concepts |
| `fact_recall` | Judge | doc / table | `expected_topics[].golden_facts` or `tables[].golden_facts` | Golden facts conveyed in matched entries |
| `enrichment_diversity` | Deterministic | both | `expected_headings` | Expected sections present (queries sidecar satisfies "Sample Queries") |
| `business_terms_presence` | Judge | both | `business_terms` | Expected terms covered (semantic/flavor match) |
| `business_terms_validity` | Judge | both | `business_terms` | Dedicated per-term MaC files — typically low today |
| `entry_grounding` | Deterministic | table | — | Generated entries correspond to real dataset tables |
| `trajectory` | Deterministic | both | `trajectory` | `must_call` / `must_not_call` tool categories from `trajectory.json` |
| `context_preservation` | Judge | both | `prebaked_facts` | Pre-existing facts preserved through enrichment |
| `persona_alignment` | Judge | doc | `personas` + `--persona` | Output emphasizes persona focus areas, retains shared concepts |

Default pass thresholds for concept and fact metrics: 0.7 (`concept_recall`, `concept_precision`, `fact_recall`).

### Cross-run consistency metrics

When `--run --runs N` produces N independent agent outputs (distinct output dirs), two informational stability metrics are added. They never gate the case and are excluded from the average.

| Metric | Description |
|--------|-------------|
| `concept_consistency` | Same set of concepts produced across runs (semantic match via judge) |
| `content_consistency` | Recurring concepts state consistent facts across runs |

With fewer than two independent runs, consistency metrics show `n/a` with guidance to use `--run --runs N`.

<Warning>
`--runs` on a single `--output-dir` in score mode re-scores the same output and does not measure cross-run stability. Use `--run --runs N` for independent runs.
</Warning>

## Report artifacts

### Dynamic eval report

Written to `<output_dir>/eval_report.md` alongside `trajectory.json`. Contains full rationales, insights, per-run breakdowns (when aggregated), and telemetry.

### Golden eval reports

| Artifact | Location | Contents |
|----------|----------|----------|
| Per-run report | `$TMPDIR/kc_golden_eval_reports/golden_run_<time>_<id>/<golden>/run<N>.md` | Full metrics for one agent run |
| Aggregate report | `.../<golden>/aggregate.md` | Mean scores, per-metric `run_scores`, per-run breakdown |
| Manifest | `.../manifest.json` | Run ID, project, model, cases, per-case averages |
| Score-only golden | `$TMPDIR/kc_golden_eval_reports/golden_report_<golden>__<run>.md` | Report when scoring existing output |

### Terminal scorecard

The CLI prints a formatted table with metric name, score (0–100), and truncated rationale. Multi-run cases prefix rationales with `runs k/n [s1, s2, …]`. Use `--json` for machine-readable output.

<ResponseExample>

```text
Dynamic eval — /tmp/enrich_out/mdcode
  mode: doc  (agent_type=doc)

  metric                              score   rationale
  ----------------------------------- ------- ----------------------------------------
  structural_validity                 100.0   All 8 generated entries are valid...
  perf                                100.0   Completed in 142s. Used 48,231 tokens...
  hallucination_free                   92.3   2 of 27 claims unsupported...
  redundancy_index                     78.5   ...
  ----------------------------------- -------
  AVERAGE                              89.2

  tokens: 48,231 (in 41,102 / out 7,129)  ·  tool calls: 14  ·  latency: 142.0s
```

</ResponseExample>

## Golden file schema

Start from `eval/goldens/TEMPLATE.json`. Keep fields that match your enrichment mode.

```jsonc
{
  "expected_topics": [
    {
      "canonical": "Reorder Point",
      "flavor_hints": ["reorder level", "ROP"],
      "golden_facts": ["ROP = average daily demand × lead time, plus safety stock."]
    }
  ],
  "acceptable_extra_concepts": [{"name": "SKU", "aliases": ["stock keeping unit"]}],
  "tables": [{"table": "order_items", "golden_facts": ["grain is one unit per order line"]}],
  "expected_headings": ["Lineage", "Sample Queries"],
  "business_terms": ["session", "event"],
  "trajectory": {"must_call": [], "must_not_call": ["dataset_pull"]},
  "personas": {
    "analyst": {
      "instruction": "Focus on operational metrics",
      "focus_areas": ["inventory turnover"],
      "shared_concepts": ["order lifecycle"]
    }
  },
  "run": {
    "mode": "table",
    "topic": "Metadata enrichment",
    "folders": "eval/corpora/my_corpus",
    "entry_group": "{project}.global.kc-eval-foo",
    "setup": {
      "copy_public_dataset": {
        "source": "bigquery-public-data.thelook_ecommerce",
        "dataset": "thelook_ecommerce"
      }
    }
  }
}
```

Mode is auto-detected from `trajectory.json` `agent_type` (`doc`, `table`, or `context_overlay`). Context overlay runs skip entry-type checks and table-only metrics.

### Building goldens

1. **Author deliberately** — write `golden_facts` and `expected_topics` for scenarios you care about.
2. **Work backward from documented data** — hold out human-written descriptions, run the agent, use held-out text as `golden_facts`.
3. **Harvest from human review** — capture approved or corrected entries as goldens.

<Note>
A golden is an imperfect oracle. Spot-check low scores — a correct output not listed in the golden can register as a false miss.
</Note>

## CLI flags

<ParamField body="--output-dir" type="string">
Agent output directory containing `catalog/` and `trajectory.json`. Required in score mode. Comma-separated to pair with multiple `--goldens`. In `--run` mode, optional report root (defaults to `$TMPDIR/kc_golden_eval_reports`).
</ParamField>

<ParamField body="--golden" type="string">
Single golden JSON file for golden-based scoring.
</ParamField>

<ParamField body="--goldens" type="string">
Comma-separated golden files. Run or score several cases at once.
</ParamField>

<ParamField body="--run" type="boolean">
Execute each golden's `run` block on the agent, then score. Requires `--project`.
</ParamField>

<ParamField body="--project" type="string" required>
GCP project for agent execution and dataset copy. Sets `GOOGLE_CLOUD_PROJECT` for the judge.
</ParamField>

<ParamField body="--runs" type="integer">
Times to run each case. Default 3 in `--run` mode. Only valid with golden case-runs; rejected for dynamic eval on a single output dir.
</ParamField>

<ParamField body="--concurrency" type="integer">
Max concurrent agent processes in `--run`. Default 2, overridable via `KC_EVAL_MAX_CONCURRENCY`.
</ParamField>

<ParamField body="--model" type="string">
Vertex AI model for agent (`--run`) and judge. Default `gemini-2.5-pro`.
</ParamField>

<ParamField body="--persona" type="string">
Persona ID from the golden's `personas` block. Golden mode only.
</ParamField>

<ParamField body="--dry-run" type="boolean">
Print the `--run` plan without executing.
</ParamField>

<ParamField body="--json" type="boolean">
Emit raw JSON results instead of the formatted scorecard.
</ParamField>

## Mode-specific behavior

### Doc mode

Uses `expected_topics` for concept recall/precision and fact recall. Trajectory checks derive tool categories from `trajectory.json` `tool_uses` (mapped to `drive_fetch`, `dataset_pull`, `github_fetch`).

### Table mode

Uses `tables[].golden_facts` for fact recall. Adds `entry_grounding` (no invented tables). Hallucination grounding includes pulled schema, reference sidecars (`.ref.yaml`, `.ref.overview.md`), and generated YAML.

### Context overlay mode

Keeps `agent_type=context_overlay`. Structural validity skips entry-type enforcement. Table-only metrics (`entry_grounding`, per-table fact recall) do not apply. Score with `business_terms`, `expected_headings`, and trajectory-grounded `hallucination_free`.

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| Judge metrics show `n/a` | Missing Vertex auth | Set `GOOGLE_CLOUD_PROJECT`, run `gcloud auth application-default login` |
| `[score] skip … no trajectory.json` | Agent run incomplete | Check agent deps, built `kcmd`, and agent logs in the output dir |
| `error: --run needs --project` | Missing project in run mode | Pass `--project <your-gcp-project>` |
| `error: --runs only applies to golden case-runs` | `--runs` with dynamic eval | Drop `--runs` or use `--run --goldens` |
| Agent exits non-zero in `--run` | Setup or quota failure | Check `[run] FAILED` stderr tail; verify BigQuery access for dataset copy |
| Consistency metrics `n/a` | Single independent run | Use `--run --runs 3` for distinct outputs |
| `business_terms_validity` low | Agent emits no per-term files | Expected gap; check `business_terms_presence` instead |
| `context_preservation` low | Agent regenerates from scratch | Merge-into-existing path not yet implemented |

<AccordionGroup>
<Accordion title="Environment variables">

| Variable | Purpose |
|----------|---------|
| `GOOGLE_CLOUD_PROJECT` | Vertex AI project for judge (and agent in `--run`) |
| `GOOGLE_GENAI_USE_VERTEXAI` | Set to `True` by `--run` |
| `GOOGLE_CLOUD_LOCATION` | Vertex location (default `global`) |
| `KC_EVAL_MAX_CONCURRENCY` | Default `--concurrency` cap (default 2) |
| `KC_AGENT_DIR` | Override path to `agents/enrichment/src` |

</Accordion>

<Accordion title="Hallucination check internals">

The judge extracts atomic domain claims from overviews, then verifies each claim against the full grounding corpus in overlapping chunks (45K chars, 1.5K overlap). Claims are checked in parallel (up to 3 workers). A claim is hallucinated only if no chunk supports it. Table mode adds schema/reference metadata to the grounding corpus.

</Accordion>
</AccordionGroup>

## Next

<CardGroup>
<Card title="Run catalog enrichment agent" href="/run-catalog-enrichment-agent">
Execute table, doc, or context_overlay enrichment modes and produce the output directories this evaluator scores.
</Card>
<Card title="Enrichment workflows" href="/enrichment-workflows">
End-to-end flow from source metadata through enrichment to catalog publication.
</Card>
<Card title="Produce OKF bundles" href="/produce-okf-bundles">
Alternative enrichment path that emits OKF bundles instead of mdcode workspaces.
</Card>
<Card title="Publish enriched metadata" href="/publish-enriched-metadata">
Push scored mdcode workspaces to Knowledge Catalog with kcmd after eval passes your bar.
</Card>
<Card title="Troubleshooting" href="/troubleshooting">
Auth, billing, kcmd build, and model credential failures across the tooling surface.
</Card>
</CardGroup>