# Instrumentation and evals

> defineInstrumentation OTel setup, workflow run tags, defineEval/defineEvalConfig authoring under evals/, eve eval flags, judges, assertions, and reporters (Braintrust, JUnit).

- Repository: vercel/eve
- GitHub: https://github.com/vercel/eve
- Human docs: https://grok-wiki.com/public/docs/vercel-eve-759e1d74a10f
- Complete Markdown: https://grok-wiki.com/public/docs/vercel-eve-759e1d74a10f/llms-full.txt

## Source Files

- `docs/guides/instrumentation.md`
- `docs/evals/overview.mdx`
- `packages/eve/src/public/definitions/instrumentation.ts`
- `packages/eve/src/evals/define-eval.ts`
- `packages/eve/src/evals/cli/eval.ts`
- `packages/eve/src/compiler/channel-instrumentation-types.ts`

---

---
title: "Instrumentation and evals"
description: "defineInstrumentation OTel setup, workflow run tags, defineEval/defineEvalConfig authoring under evals/, eve eval flags, judges, assertions, and reporters (Braintrust, JUnit)."
---

Eve exposes two complementary quality surfaces: **instrumentation** for runtime observability (`agent/instrumentation.ts`, OpenTelemetry spans, and framework-owned workflow run tags) and **evals** for repeatable scored checks (`evals/*.eval.ts`, `evals/evals.config.ts`, and the `eve eval` CLI). Instrumentation answers what happened during production sessions; evals drive the same HTTP session protocol your users hit and grade the result before you ship a change.

## Observability surfaces

Eve observes an agent through three distinct surfaces. They write to different backends and are configured differently:

| Surface | Configured in `instrumentation.ts`? | What it records |
| --- | --- | --- |
| Workflow run tags (`$eve.*`) | No — automatic | Framework-owned attributes on each Vercel Workflow run. Dashboards stitch session, turn, and subagent runs into a tree and surface model and token usage. |
| OpenTelemetry export | Yes — `setup`, `recordInputs`, `recordOutputs`, `functionId` | Where AI SDK spans are exported and what they record. |
| Runtime context events | Yes — `events["step.started"]` | Per-model-call values merged into AI SDK telemetry spans. |

Workflow run tags are queryable in the Vercel Workflow dashboard (and the **Agent Runs** observability tab on Vercel deployments). OpenTelemetry and runtime context both ride on AI SDK spans and export to any OTel-compatible backend you configure.

```mermaid
flowchart TB
  subgraph authored ["Authored (agent/instrumentation.ts)"]
    setup["setup() → registerOTel"]
    events["events['step.started'] → runtimeContext"]
    record["recordInputs / recordOutputs / functionId"]
  end

  subgraph automatic ["Framework-owned"]
    tags["$eve.* workflow run tags"]
    spans["ai.eve.turn parent span + AI SDK children"]
    ctx["eve.session.id, eve.turn.id, eve.channel.kind, …"]
  end

  subgraph sinks ["Sinks"]
    otel["OTel backend (Braintrust, Datadog, Jaeger, …)"]
    workflow["Vercel Workflow / Agent Runs dashboard"]
  end

  setup --> otel
  events --> spans
  record --> spans
  ctx --> spans
  spans --> otel
  tags --> workflow
```

## `defineInstrumentation`

Eve auto-discovers `agent/instrumentation.ts` and runs it at server startup before any agent code. Export the result of `defineInstrumentation` as the default export. **The file's presence implicitly enables telemetry** — there is no separate `isEnabled` toggle.

<CodeGroup>
```ts title="agent/instrumentation.ts — OTel setup"
import { BraintrustExporter } from "@braintrust/otel";
import { defineInstrumentation } from "eve/instrumentation";
import { registerOTel } from "@vercel/otel";

export default defineInstrumentation({
  setup: ({ agentName }) =>
    registerOTel({
      serviceName: agentName,
      traceExporter: new BraintrustExporter({
        parent: `project_name:${agentName}`,
        filterAISpans: true,
      }),
    }),
});
```

```ts title="agent/instrumentation.ts — runtime context"
import { defineInstrumentation, isChannel } from "eve/instrumentation";
import supportChannel from "./channels/support.js";

export default defineInstrumentation({
  events: {
    "step.started"(input) {
      if (!isChannel(input.channel, supportChannel)) return undefined;
      return {
        runtimeContext: {
          "support.channel_id": input.channel.metadata.channelId ?? "",
          "support.user_id": input.channel.metadata.triggeringUserId ?? "",
        },
      };
    },
  },
});
```
</CodeGroup>

### Fields

<ParamField body="setup" type="(context) => void">
Invoked at server startup with `{ agentName }`. Use it to call `registerOTel` or another OTel provider setup. `agentName` is resolved at compile time from the package `name` (falling back to the app directory name).
</ParamField>

<ParamField body="recordInputs" type="boolean" default="true">
Records full message history on each step span. Set `false` for sensitive inputs or to reduce span payload size.
</ParamField>

<ParamField body="recordOutputs" type="boolean" default="true">
Records model outputs on spans. Set `false` to disable output recording.
</ParamField>

<ParamField body="functionId" type="string">
Overrides `ai.telemetry.functionId` on spans. Defaults to the agent name.
</ParamField>

<ParamField body="events['step.started']" type="(input) => { runtimeContext } | undefined">
Runs once per model-call attempt after Eve assembles the final model input. Returned `runtimeContext` values merge onto AI SDK spans; child spans inherit them. Keys beginning with `eve.` are reserved and dropped. Return `undefined` to contribute nothing.
</ParamField>

The `step.started` callback receives `session` (id, auth, parent lineage), `turn` (id and sequence), `step` (zero-based index), `channel` (`kind` and channel-owned `metadata`), and `modelInput` (resolved instructions and messages). Narrow authored channels with `isChannel(input.channel, myChannel)` or by checking `input.channel.kind === "channel:<filename>"`. Framework channels use `http`, `schedule`, or `subagent`.

### Trace hierarchy

When telemetry is enabled, each turn produces a span tree:

```text
ai.eve.turn  {eve.session.id, eve.turn.id, …}
  +-- ai.streamText                           step 1
  |     +-- ai.streamText.doStream            model call
  |     +-- ai.toolCall  {toolName: search}   tool exec
  +-- ai.streamText                           step 2
        +-- ai.streamText.doStream
        +-- ai.toolCall  {toolName: read}
```

Eve creates the `ai.eve.turn` parent span per turn and injects framework runtime context (`eve.version`, `eve.session.id`, `eve.environment`, `eve.turn.id`, `eve.turn.sequence`, `eve.step.index`, `eve.channel.kind`) alongside any values your `step.started` callback returns.

<Note>
Any OTel-compatible backend works. Install the exporter package you need and configure it in `setup`. The Braintrust example above is one option — Honeycomb, Datadog, and Jaeger follow the same pattern.
</Note>

## Workflow run tags

Separately from OpenTelemetry, Eve tags every workflow run with reserved `$eve.*` attributes. Authored code cannot set or override the `$eve.` namespace. Tags are emitted automatically on every session, turn, and subagent run, whether or not `instrumentation.ts` is present. Tag writes are best-effort: a failure is logged once per process and swallowed so a broken tag emit never breaks the agent.

### Structural tags

| Tag | Values / meaning |
| --- | --- |
| `$eve.type` | `"session"`, `"turn"`, or `"subagent"` |
| `$eve.parent` | Session id of the immediate parent |
| `$eve.root` | Session id of the root session (group a tree with `$eve.root=<id>`) |
| `$eve.parent_call` | Parent runtime-action tool call id (subagent rows only) |
| `$eve.parent_turn` | Parent turn id that dispatched the subagent (subagent rows only) |
| `$eve.subagent` | Compiled graph node id (subagent runs only) |
| `$eve.trigger` | Channel kind that started the run |
| `$eve.title` | Truncated title from the first user message |

### Per-turn usage tags

Written on each step of a turn, accumulating cumulative totals (last write wins):

| Tag | Meaning |
| --- | --- |
| `$eve.model` | Model id for the turn |
| `$eve.input_tokens` | Running input token count |
| `$eve.output_tokens` | Running output token count |
| `$eve.cache_read_tokens` | Running cache-read token count |
| `$eve.cache_write_tokens` | Running cache-write token count |
| `$eve.tool_count` | Number of tools available to the turn |

On Vercel, the platform auto-detects `eve` and surfaces an **Agent Runs** view under **Observability**. That view is separate from OTel export — use OTel when you want spans in a third-party backend.

## Eval authoring

Evals live under the app-root `evals/` directory. Eve discovers `*.eval.ts` files recursively and requires exactly one `evals/evals.config.ts` at the root of `evals/`.

:::files
my-agent/
├── agent/
├── evals/
│   ├── evals.config.ts          # required run-wide defaults
│   ├── smoke.eval.ts
│   └── weather/
│       ├── brooklyn-forecast.eval.ts
│       └── no-tools-for-greetings.eval.ts
└── package.json
:::

Eval identity is **path-derived** — do not author `id` or `name`. `evals/weather/brooklyn-forecast.eval.ts` becomes id `weather/brooklyn-forecast`. Directories group related evals; shared helpers go in sibling non-eval files (any name that does not end in `.eval.ts`).

### `defineEval`

Each eval file default-exports one `defineEval({ ... })` call (or an array of them for dataset fan-out). The only required field is `test`:

```ts title="evals/weather/brooklyn-forecast.eval.ts"
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.completed();
    t.calledTool("get_weather");
    t.check(t.reply, includes("Sunny"));
  },
});
```

Optional per-eval fields:

| Field | Purpose |
| --- | --- |
| `description` | Human-readable summary (shown in `--list` output) |
| `judge` | Override judge model for `t.judge.*` on this eval |
| `tags` | Filter with `eve eval --tag` |
| `metadata` | Passed through to reporters |
| `timeoutMs` | Per-eval timeout (overridden by CLI `--timeout`) |
| `reporters` | Reporters scoped to this eval only |

`defineEval` rejects legacy keys (`input`, `run`, `checks`, `scores`, `expected`, `thresholds`, `parseOutput`, `model`, `requires`, `cases`). Drive the agent inside `test` and assert inline.

### `defineEvalConfig`

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "openai/gpt-5.4-mini" },
  reporters: [Braintrust({ projectName: "my-agent" })],
  maxConcurrency: 4,
  timeoutMs: 120_000,
});
```

All fields are optional — an empty `defineEvalConfig({})` satisfies the requirement. When set:

| Field | Default | Precedence |
| --- | --- | --- |
| `judge.model` | none | Per-call `model` → per-eval `judge` → config `judge` |
| `reporters` | none | Suppressed by `--skip-report` |
| `maxConcurrency` | `8` | CLI `--max-concurrency` overrides |
| `timeoutMs` | none | Per-eval `timeoutMs` → CLI `--timeout` |

Array-exported eval files derive ids as `<file-id>/<zero-padded-index>` (e.g. `weather/0000`, `weather/0001`).

## The `t` context

`t` is both the session driver and the assertion surface passed to `test(t)`. There are no separate `input`, `run`, `checks`, or `scores` fields.

**Drive** the agent:

| Method | Purpose |
| --- | --- |
| `t.send(input)` | Send one turn; resolves when the turn settles |
| `t.respond(...)` / `t.respondAll(optionId)` | Resolve HITL input requests and resume |
| `t.sendFile(text, path, mediaType?)` | Send text with a local file attached |
| `t.expectInputRequests(filter?)` | Assert the run parked on HITL input |
| `t.newSession()` | Start an additional independent session |
| `t.reply` | Last assistant message on the primary session |
| `t.events` / `t.sessionId` / `t.state` | Stream events, session id, resumable state |
| `t.log(message)` | Structured log (visible with `--verbose`) |

Evals exercise the same HTTP session protocol as production clients. The runner boots a local dev server or targets a remote deployment via `--url`.

## Assertions

Assertions record results during `test(t)` and are evaluated when the run completes. Every assertion returns a chainable handle; severity rides on the assertion itself — there is no separate thresholds map.

### Three assertion surfaces

| Surface | Examples | Default severity |
| --- | --- | --- |
| Run-level methods | `t.completed()`, `t.calledTool("get_weather")`, `t.usedNoTools()`, `t.toolOrder([...])` | gate |
| `t.check(value, assertion)` | `t.check(t.reply, includes("sunny"))` with builders from `eve/evals/expect` | depends on builder |
| `t.judge.autoevals.*` | `t.judge.autoevals.closedQA("cites a source")` | soft |

**Run-level methods** observe the whole run:

| Method | Asserts |
| --- | --- |
| `t.completed()` | Run did not fail and did not park on unanswered HITL input |
| `t.didNotFail()` | No terminal failure (parked runs pass) |
| `t.waiting()` | Run parked on HITL input |
| `t.messageIncludes(token)` | Joined assistant text contains token |
| `t.outputEquals(value)` / `t.outputMatches(schema)` | Structured output equality or Standard Schema validation |
| `t.calledTool(name, opts?)` | Matching tool call (`input`, `output`, `isError`, `times`) |
| `t.notCalledTool(name)` | No call to `name` |
| `t.toolOrder([...names])` | Tool names appear in order |
| `t.usedNoTools()` | No tool calls |
| `t.maxToolCalls(n)` | At most `n` tool calls |
| `t.calledSubagent(name, opts?)` | Subagent delegation |
| `t.noFailedActions()` | No tool, subagent, or skill action reported failure |
| `t.event(predicate, label)` | Escape hatch over the typed event stream |

**Value builders** from `eve/evals/expect`:

| Builder | Scores | Default |
| --- | --- | --- |
| `includes(substring)` | substring match | gate |
| `equals(value)` | deep structural equality | gate |
| `matches(schema)` | Standard Schema validation | gate |
| `similarity(expected)` | normalized Levenshtein, 0–1 | soft |

### Gate vs soft

| Severity | Behavior |
| --- | --- |
| **Gate** | Failed gate → eval `failed` → `eve eval` exits non-zero |
| **Soft** (no threshold) | Tracked in reports; never fails |
| **Soft** with `.atLeast(n)` | Below threshold → eval `scored`; fatal under `--strict` |

Override per assertion: `.gate(threshold?)`, `.soft(threshold?)`, `.atLeast(threshold)`.

```ts
t.completed();                                              // gate
t.calledTool("get_weather").soft();                         // tracked metric
t.check(t.reply, similarity("Sunny")).atLeast(0.8);         // soft bar
t.judge.autoevals.closedQA("cites a source").gate(0.7);     // hard judge gate
```

<Warning>
`t.calledTool` and `t.usedNoTools` are mutually exclusive in the same run. `t.completed()` subsumes `t.didNotFail()` — use `didNotFail` only when you intentionally allow a parked run.
</Warning>

## LLM-as-judge

When deterministic assertions cannot capture "good enough," grade with `t.judge.autoevals.*`. The judge model is **never** the agent under test — Eve uses it only for scoring.

| Grader | Grades |
| --- | --- |
| `t.judge.autoevals.factuality(expected)` | Factual consistency against an expected answer |
| `t.judge.autoevals.summarizes(expected)` | Summary quality against expected text |
| `t.judge.autoevals.closedQA(criteria)` | Free-form yes/no criterion |
| `t.judge.autoevals.sql(expected)` | Semantic SQL equivalence |

Judge model resolution (innermost wins):

1. Per-call: `t.judge.autoevals.closedQA("…", { model, modelOptions })`
2. Per-eval: `defineEval({ judge: { model, modelOptions }, test })`
3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`

A string model id (e.g. `"openai/gpt-5.4-mini"`) routes through the Vercel AI Gateway and needs gateway credentials in the environment. An AI SDK `LanguageModel` instance runs directly. Calling `t.judge.*` with no judge model resolved records a failed gate. With a model configured but no credentials, judge-backed evals **skip visibly** rather than failing spuriously.

## `eve eval`

<Steps>
<Step title="Author evals">
Create `evals/evals.config.ts` and one or more `evals/**/*.eval.ts` files with `defineEval`.
</Step>
<Step title="Run locally">
```bash
eve eval                       # all discovered evals against a local dev server
eve eval weather               # one eval or every eval under evals/weather/
eve eval --tag fast            # only evals carrying a tag
```
</Step>
<Step title="Target remote or tune CI">
```bash
eve eval --url https://<app>    # deployed instance
eve eval --strict --junit .eve/junit.xml   # CI: soft thresholds fail too
```
</Step>
</Steps>

### CLI flags

<ParamField body="[evalIds...]" type="string[]">
Eval ids or directory prefixes. Omit to run all discovered evals. `weather` matches `evals/weather.eval.ts`, everything under `evals/weather/`, and array entries from `evals/weather.eval.ts`.
</ParamField>

<ParamField body="--url" type="string">
Remote agent URL. Skips local dev server startup.
</ParamField>

<ParamField body="--tag" type="string[]">
Run only evals carrying any requested tag.
</ParamField>

<ParamField body="--strict" type="boolean">
Fail exit code when any soft assertion falls below its threshold.
</ParamField>

<ParamField body="--list" type="boolean">
Print discovered evals without running them. Use `--json` for machine-readable output.
</ParamField>

<ParamField body="--timeout" type="integer">
Per-eval timeout in milliseconds. Non-negative integer.
</ParamField>

<ParamField body="--max-concurrency" type="integer">
Max concurrent eval executions. Positive integer; default `8`.
</ParamField>

<ParamField body="--json" type="boolean">
Output run summary as JSON. Suppresses the console reporter.
</ParamField>

<ParamField body="--junit" type="path">
Write JUnit XML to the given path.
</ParamField>

<ParamField body="--skip-report" type="boolean">
Skip config-level and eval-defined reporters (e.g. Braintrust).
</ParamField>

<ParamField body="--verbose" type="boolean">
Stream per-eval `t.log` lines to stdout.
</ParamField>

### Exit codes and verdicts

| Exit code | Meaning |
| --- | --- |
| `0` | Every eval passed its gates (and soft thresholds under `--strict`) |
| `1` | Any eval failed — gate miss, execution error, or strict threshold miss |
| `2` | Configuration error — no evals discovered, no tag matches, invalid flags, missing `evals.config.ts` |

Per-eval verdicts:

| Verdict | Meaning |
| --- | --- |
| `passed` | No execution error; every gate held; every soft threshold met |
| `failed` | Gate assertion failed or execution errored (timeout, transport, thrown task) |
| `scored` | Every gate held but a soft assertion fell below its threshold |

### Artifacts

Each run writes artifacts under `.eve/evals/<timestamp>/`:

| File | Contents |
| --- | --- |
| `summary.json` | Aggregated run outcome |
| `results.jsonl` | One line per eval result |
| `evals/<id>.json` | Per-eval assertions, verdict, logs |
| `evals/<id>.events.ndjson` | Captured NDJSON event stream |

The console summary stays tight; when an eval fails, the artifact directory has the full event stream and assertion details.

## Reporters

Eve runs and grades everything itself. Reporters ship results to external destinations.

### Braintrust

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEvalConfig({
  reporters: [Braintrust({ projectName: "weather-agent" })],
});
```

`Braintrust({ projectName?, projectId?, experimentName?, baseExperimentName?, baseExperimentId?, update? })` uploads to Braintrust experiments. Gate assertions log as binary scores under a `gate:` prefix. Requires the `braintrust` package and `BRAINTRUST_API_KEY`. Use `--skip-report` to suppress during local iteration.

### JUnit

```bash
eve eval --strict --junit .eve/junit.xml
```

`JUnit({ filePath })` writes one `<testcase>` per eval (named by path-derived id). Failed gates and execution errors appear as failure messages. The `--junit` flag is usually preferable in CI because the output path is owned by the pipeline, not the eval file.

### Custom reporters

Implement `EvalReporter` from `eve/evals/reporters`:

```ts
interface EvalReporter {
  onRunStart(evaluations, target): void | Promise<void>;
  onEvalComplete(result): void | Promise<void>;
  onRunComplete(summary): void | Promise<void>;
}
```

Attach reporters in `evals.config.ts` (observes every eval) or on individual eval definitions (scoped). The same reporter instance is not double-reported if listed in both places.

## Troubleshooting

| Symptom | Likely cause |
| --- | --- |
| No spans in OTel backend | Missing `agent/instrumentation.ts` or `setup` not calling `registerOTel` |
| `eve eval` exits `2` with config error | Missing `evals/evals.config.ts` — create `defineEvalConfig({})` at minimum |
| Judge assertions always fail | No `judge.model` resolved at any level, or missing gateway credentials |
| Eval passes locally, fails in CI | Run `eve eval --strict`; check model-provider credentials in the CI environment |
| Duplicate eval id error | Array-exported file collides with a nested eval path (e.g. `weather.eval.ts` and `weather/0000.eval.ts`) |

For discovery diagnostics on the agent side, run `eve info` and inspect `.eve/discovery/diagnostics.json`.

## Related pages

<CardGroup>
<Card title="Project layout" href="/project-layout">
Where `evals/` and `agent/instrumentation.ts` fit in the authored tree and how path-derived naming works.
</Card>
<Card title="CLI reference" href="/cli-reference">
Full `eve eval` command reference, exit codes, and the edit-info-dev-build-start loop.
</Card>
<Card title="Sessions and streaming" href="/sessions-and-streaming">
The HTTP session protocol evals drive and the NDJSON events assertions read.
</Card>
<Card title="Execution model" href="/execution-model">
Session/turn/step nesting, workflow durability, and the run tree behind `$eve.*` tags.
</Card>
<Card title="Auth and deployment" href="/auth-and-deployment">
Model credentials for evals and the Vercel Agent Runs observability surface.
</Card>
<Card title="TypeScript API" href="/typescript-api">
`defineInstrumentation`, `defineEval`, `defineEvalConfig`, and eval entrypoint import paths.
</Card>
</CardGroup>