# Observability and Langfuse

> Optional OpenTelemetry path to Grafana, VictoriaMetrics, Jaeger, and Loki; Langfuse LLM analytics compose stack; OTEL_HOST and LANGFUSE_* keys; what each stack measures.

- Repository: vxcontrol/pentagi
- GitHub: https://github.com/vxcontrol/pentagi
- Human docs: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5
- Complete Markdown: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5/llms-full.txt

## Source Files

- `docker-compose-observability.yml`
- `docker-compose-langfuse.yml`
- `backend/docs/observability.md`
- `backend/docs/langfuse.md`
- `observability/otel/config.yml`
- `backend/pkg/config/config.go`
- `.env.example`

---

---
title: "Observability and Langfuse"
description: "Optional OpenTelemetry path to Grafana, VictoriaMetrics, Jaeger, and Loki; Langfuse LLM analytics compose stack; OTEL_HOST and LANGFUSE_* keys; what each stack measures."
---

PentAGI exposes two independent, optional observability paths: **OpenTelemetry** (`OTEL_HOST`) for platform traces, logs, and metrics into Grafana/VictoriaMetrics/Jaeger/Loki, and **Langfuse** (`LANGFUSE_*`) for LLM and agent analytics. Both are disabled when their env keys are empty; `cmd/pentagi` treats `ErrNotConfigured` as non-fatal and continues with no-op clients.

## What each stack measures

| Stack | Signal focus | Primary UIs | Compose file |
| --- | --- | --- | --- |
| OpenTelemetry platform stack | App spans, structured logs, process/Go runtime metrics, host/container scrapes | Grafana `:3000`, Jaeger query, Loki Explore | `docker-compose-observability.yml` |
| Langfuse | LLM generations, agent/tool/chain observations, token usage, scores, flow sessions | Langfuse Web `:4000` | `docker-compose-langfuse.yml` |

They are complementary: OTEL answers “is the service healthy and where is latency?”, Langfuse answers “what did the models and agents do, with which tokens and outcomes?”.

## Architecture

```mermaid
flowchart LR
  subgraph app [PentAGI process]
    Main["cmd/pentagi"]
    Obs["pkg/observability Observer"]
    Logrus["logrus hook"]
    Main --> Obs
    Logrus --> Obs
  end

  subgraph otelPath [OpenTelemetry path]
    OTEL["otelcol hostname<br/>gRPC :8148 / HTTP :4318"]
    VM["VictoriaMetrics"]
    Jaeger["Jaeger + ClickHouse clickstore"]
    Loki["Loki"]
    Grafana["Grafana"]
    NodeExp["node-exporter"]
    cAdvisor["cadvisor"]
    OTEL --> VM
    OTEL --> Jaeger
    OTEL --> Loki
    NodeExp --> OTEL
    cAdvisor --> OTEL
    VM --> Grafana
    Jaeger --> Grafana
    Loki --> Grafana
  end

  subgraph lfPath [Langfuse path]
    LFWeb["langfuse-web :4000"]
    LFWorker["langfuse-worker"]
    LFPG["langfuse-postgres"]
    LFCH["langfuse-clickhouse"]
    LFRedis["langfuse-redis"]
    LFMinio["langfuse-minio"]
    LFWeb --> LFPG
    LFWeb --> LFCH
    LFWeb --> LFRedis
    LFWeb --> LFMinio
    LFWorker --> LFPG
    LFWorker --> LFCH
    LFWorker --> LFRedis
    LFWorker --> LFMinio
  end

  Obs -->|"OTEL_HOST gRPC OTLP"| OTEL
  Obs -->|"LANGFUSE_BASE_URL HTTP"| LFWeb
  LFWeb -.->|"optional LANGFUSE_OTEL_EXPORTER_OTLP_ENDPOINT"| OTEL
```

Collector pipelines (`observability/otel/config.yml`):

| Pipeline | Receivers | Exporters |
| --- | --- | --- |
| `traces` | OTLP | OTLP → `jaeger:4317` |
| `logs` | OTLP | OTLP/HTTP → `http://loki:3100/otlp` |
| `metrics` | OTLP + Prometheus scrapes (node-exporter, cAdvisor, clickstore, jaeger, loki, pgexporter, host Docker engine) | Prometheus remote write → `http://victoriametrics:8428/api/v1/write` |

Grafana is pre-provisioned with datasources **VictoriaMetrics**, **Jaeger** (trace→log to Loki), and **Loki** (log→trace to Jaeger).

## Prerequisites

- Core stack networks exist: run `docker-compose.yml` first so external networks `pentagi-network`, `observability-network`, and `langfuse-network` are created.
- `.env` copied from `.env.example` and loaded by Compose.
- Overlay files present next to the core compose file (or fetched from the repo).

<Warning>
If Compose fails with missing `pentagi-network`, `observability-network`, or `langfuse-network`, start the core stack first, then add overlays.
</Warning>

## Enable the OpenTelemetry stack

<Steps>
  <Step title="Set the collector endpoint for PentAGI">
    In `.env`, point the app at the collector **hostname** and **gRPC** port (the telemetry client dials gRPC, not HTTP):

```bash
OTEL_HOST=otelcol:8148
```

    The collector service is named `otel` in Compose; DNS hostname is `otelcol`. Host-published ports default to `127.0.0.1:8148` (gRPC) and `127.0.0.1:4318` (HTTP).
  </Step>
  <Step title="Start core + observability overlay">
```bash
docker compose -f docker-compose.yml -f docker-compose-observability.yml up -d
```
  </Step>
  <Step title="Verify">
    - Open Grafana at `http://localhost:3000` (bind via `GRAFANA_LISTEN_IP` / `GRAFANA_LISTEN_PORT`, default host `127.0.0.1:3000`).
    - Confirm datasources VictoriaMetrics, Jaeger, and Loki resolve.
    - Restart PentAGI after setting `OTEL_HOST` so `NewTelemetryClient` connects; empty `OTEL_HOST` yields `ErrNotConfigured` and skips exporters.
  </Step>
</Steps>

### Observability services

| Service | Role | Notes |
| --- | --- | --- |
| `otel` | OTLP collector (contrib `0.116.1`) | Pipelines for traces, logs, metrics; joins `observability-network`, `langfuse-network`, `pentagi-network` |
| `grafana` | Dashboards / Explore | Config and dashboards under `observability/grafana/` |
| `victoriametrics` | Metrics TSDB | Remote-write target for collector metrics |
| `jaeger` | Trace UI / query | ClickHouse plugin storage via `clickstore` |
| `clickstore` | Trace storage (ClickHouse) | DB `jaeger`, user/password `clickhouse` in compose defaults |
| `loki` | Log store | Config `observability/loki/config.yml` |
| `node-exporter` | Host metrics scrape | Scraped by collector Prometheus receiver |
| `cadvisor` | Container metrics scrape | Docker-only mode |

### App-side OTEL behavior

On startup (`backend/cmd/pentagi/main.go`):

1. `NewTelemetryClient(ctx, cfg)` — requires `OTEL_HOST`; exports logs, traces, and metrics over gRPC OTLP with ~30s batch intervals.
2. `InitObserver` attaches a logrus hook for Debug/Info/Warn/Error so log lines become span events and OTEL log records when a span is active.
3. `StartProcessMetricCollect` and `StartGoRuntimeMetricCollect` publish process and Go runtime metrics when telemetry is live.

HTTP (Gin) and GraphQL middleware emit structured request logs with context for correlation. Profiling is separate: `pkg/observability/profiling` serves pprof on `:7777` under `/profiler/*` (heap, CPU, goroutine, mutex, trace).

## Enable the Langfuse stack

<Steps>
  <Step title="Harden Langfuse infrastructure secrets">
    Set non-default credentials in `.env` for Postgres, ClickHouse, Redis, MinIO/S3, salt, encryption key, and NextAuth before exposing beyond localhost. Generate a 32-byte hex encryption key with `openssl rand -hex 32`.
  </Step>
  <Step title="Wire PentAGI → Langfuse API">
```bash
LANGFUSE_BASE_URL=http://langfuse-web:3000
LANGFUSE_PROJECT_ID=          # default: LANGFUSE_INIT_PROJECT_ID
LANGFUSE_PUBLIC_KEY=          # default: LANGFUSE_INIT_PROJECT_PUBLIC_KEY
LANGFUSE_SECRET_KEY=          # default: LANGFUSE_INIT_PROJECT_SECRET_KEY
```

    Align project ID and keys with the Langfuse init project (`LANGFUSE_INIT_*`).
  </Step>
  <Step title="Start core + Langfuse overlay">
```bash
docker compose -f docker-compose.yml -f docker-compose-langfuse.yml up -d
```
  </Step>
  <Step title="Open Langfuse UI">
    Browse `http://localhost:4000` (default bind `LANGFUSE_LISTEN_IP`/`LANGFUSE_LISTEN_PORT` → `127.0.0.1:4000`).

    Sign in with `LANGFUSE_INIT_USER_EMAIL` / `LANGFUSE_INIT_USER_PASSWORD` (compose demo defaults include `admin@pentagi.com`; change them).
  </Step>
</Steps>

### Langfuse services

| Service | Role |
| --- | --- |
| `langfuse-web` | UI + API (image `langfuse/langfuse:3`), on `langfuse-network` and `pentagi-network` |
| `langfuse-worker` | Background ingestion/processing |
| `postgres` (`langfuse-postgres`) | Primary relational store |
| `clickhouse` (`langfuse-clickhouse`) | Analytics queries (reads prefer ClickHouse in default env) |
| `redis` (`langfuse-redis`) | Queue/cache |
| `minio` (`langfuse-minio`) | S3-compatible media/event storage |

### App-side Langfuse behavior

`NewLangfuseClient` requires `LANGFUSE_BASE_URL`. Client options use public/secret keys and project ID; the observer batches with defaults: send interval **10s**, timeout **10s**, queue size **10**, max attempts **3**, and attaches the binary release version.

When base URL is empty, the process keeps a no-op path. Observations are non-blocking and fail soft so LLM analytics outages do not stop flows.

Observation types supported by `pkg/observability/langfuse`: span, event, generation, score, agent, tool, chain, retriever, evaluator, embedding, guardrail. Inputs/outputs convert LangChainGo message shapes to OpenAI-compatible JSON for Langfuse UI (tool calls, thinking, multimodal content, table-friendly tool results).

Typical flow worker metadata: trace name like `{id} flow worker`, user id, session `flow-{id}`, tags such as `controller`, and flow/user metadata.

## Run both stacks (and Graphiti)

```bash
docker compose \
  -f docker-compose.yml \
  -f docker-compose-langfuse.yml \
  -f docker-compose-observability.yml \
  up -d
```

Optional third overlay: `docker-compose-graphiti.yml` (knowledge graph; separate from observability).

To export Langfuse’s own OTEL into the platform collector:

```bash
LANGFUSE_OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4318
```

Shell aliases from the project README:

```bash
alias pentagi="docker compose -f docker-compose.yml -f docker-compose-langfuse.yml -f docker-compose-graphiti.yml -f docker-compose-observability.yml"
alias pentagi-up="... up -d"
alias pentagi-down="... down"
```

## Configuration reference

### PentAGI process (required to enable each path)

<ParamField body="OTEL_HOST" type="string">
OpenTelemetry collector endpoint for gRPC OTLP. Example: `otelcol:8148`. Empty disables telemetry client (`TelemetryEndpoint` in `Config`).
</ParamField>

<ParamField body="LANGFUSE_BASE_URL" type="string">
Langfuse API base URL. Example: `http://langfuse-web:3000`. Empty disables Langfuse client.
</ParamField>

<ParamField body="LANGFUSE_PROJECT_ID" type="string">
Langfuse project id; should match `LANGFUSE_INIT_PROJECT_ID` for the demo project.
</ParamField>

<ParamField body="LANGFUSE_PUBLIC_KEY" type="string">
Project public key (`pk-lf-...`). Treated as a secret redaction candidate in config sanitization.
</ParamField>

<ParamField body="LANGFUSE_SECRET_KEY" type="string">
Project secret key (`sk-lf-...`). Treated as a secret redaction candidate.
</ParamField>

### Host bind ports (optional)

| Variable | Purpose | Compose default bind |
| --- | --- | --- |
| `GRAFANA_LISTEN_IP` / `GRAFANA_LISTEN_PORT` | Grafana host bind | `127.0.0.1:3000` |
| `OTEL_GRPC_LISTEN_IP` / `OTEL_GRPC_LISTEN_PORT` | Collector gRPC host bind | `127.0.0.1:8148` |
| `OTEL_HTTP_LISTEN_IP` / `OTEL_HTTP_LISTEN_PORT` | Collector HTTP host bind | `127.0.0.1:4318` |
| `LANGFUSE_LISTEN_IP` / `LANGFUSE_LISTEN_PORT` | Langfuse Web host bind | `127.0.0.1:4000` |
| `LANGFUSE_NEXTAUTH_URL` | Public URL for Langfuse auth | `http://localhost:4000` |

### Langfuse infrastructure (high-value secrets)

Change before any non-local exposure:

| Area | Variables |
| --- | --- |
| DB | `LANGFUSE_POSTGRES_USER`, `LANGFUSE_POSTGRES_PASSWORD`, `LANGFUSE_POSTGRES_DB` |
| ClickHouse | `LANGFUSE_CLICKHOUSE_USER`, `LANGFUSE_CLICKHOUSE_PASSWORD`, `LANGFUSE_CLICKHOUSE_URL`, `LANGFUSE_CLICKHOUSE_MIGRATION_URL` |
| Redis | `LANGFUSE_REDIS_HOST`, `LANGFUSE_REDIS_PORT`, `LANGFUSE_REDIS_AUTH` |
| S3/MinIO | `LANGFUSE_S3_BUCKET`, `LANGFUSE_S3_ACCESS_KEY_ID`, `LANGFUSE_S3_SECRET_ACCESS_KEY`, `LANGFUSE_S3_ENDPOINT` |
| Crypto / auth | `LANGFUSE_SALT`, `LANGFUSE_ENCRYPTION_KEY`, `LANGFUSE_NEXTAUTH_SECRET` |
| Bootstrap project | `LANGFUSE_INIT_ORG_*`, `LANGFUSE_INIT_PROJECT_*`, `LANGFUSE_INIT_USER_*` |
| Optional OTEL from Langfuse | `LANGFUSE_OTEL_EXPORTER_OTLP_ENDPOINT`, `LANGFUSE_OTEL_SERVICE_NAME` |

Full key inventory lives under [Environment variables](/environment-variables).

## Repository layout

:::files
observability/
  otel/config.yml              # collector pipelines
  grafana/config/              # grafana.ini + provisioning
  grafana/dashboards/          # home, node, docker, pentagi service
  loki/config.yml
  jaeger/                      # clickhouse plugin binaries + config
  clickhouse/prometheus.xml
docker-compose-observability.yml
docker-compose-langfuse.yml
backend/pkg/observability/     # Observer, OTEL client, Langfuse SDK
backend/docs/observability.md
backend/docs/langfuse.md
:::

## Failure modes and operations

| Symptom | Likely cause | Mitigation |
| --- | --- | --- |
| Overlay compose: network not found | Core stack not started | `docker compose -f docker-compose.yml up -d` first |
| No traces/metrics in Grafana | `OTEL_HOST` empty or wrong host/port | Set `otelcol:8148`; restart PentAGI; confirm collector container `otel` healthy |
| Fatal on startup mentioning telemetry | Collector unreachable with `OTEL_HOST` set | Only empty endpoint is soft-skipped; a set but dead endpoint fails `NewTelemetryClient` hard |
| No Langfuse traces | Missing `LANGFUSE_BASE_URL` or key/project mismatch | Match init project keys; open Langfuse UI and confirm project |
| Langfuse UI login fails | Init user/password not rotated or NextAuth URL wrong | Align `LANGFUSE_INIT_USER_*` and `LANGFUSE_NEXTAUTH_URL` with access URL |
| Logs without parent traces | `logrus` without request context | Always `logrus.WithContext(ctx)` so the hook attaches to the active span |
| Container registry pull fails for cAdvisor | `gcr.io/cadvisor/cadvisor` not reachable | Mirror/proxy that registry or skip observability overlay in restricted networks |

<Tip>
Use Grafana Explore (Loki ↔ Jaeger links) for platform debugging, and Langfuse sessions/traces for multi-agent LLM workflows and token usage. Keep both optional in air-gapped or BYOK-only installs by leaving `OTEL_HOST` and `LANGFUSE_BASE_URL` empty.
</Tip>

## Next

<CardGroup>
  <Card title="Installation" href="/installation">
    Core Compose, `.env`, SSL/volumes, and how observability, Langfuse, and Graphiti overlays attach.
  </Card>
  <Card title="Environment variables" href="/environment-variables">
    Authoritative `Config` and `.env.example` keys including OTEL, Langfuse, and related defaults.
  </Card>
  <Card title="Knowledge graph" href="/knowledge-graph">
    Optional Graphiti/Neo4j overlay (`docker-compose-graphiti.yml`) orthogonal to OTEL/Langfuse.
  </Card>
  <Card title="Configure LLM providers" href="/configure-llm-providers">
    BYOK providers whose generations appear in Langfuse when analytics is enabled.
  </Card>
  <Card title="Development and testing" href="/development-and-testing">
    Local backend build/test paths and contributor compose usage.
  </Card>
</CardGroup>