# Memory layers

> Repo-specific L0 conversation, L1 atom, L2 scene, and L3 persona model: producers, storage shapes, pipeline triggers, and drill-down paths.

- Repository: TencentCloud/TencentDB-Agent-Memory
- GitHub: https://github.com/TencentCloud/TencentDB-Agent-Memory
- Human docs: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-14fefdd76c97
- Complete Markdown: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-14fefdd76c97/llms-full.txt

## Source Files

- `src/core/conversation/l0-recorder.ts`
- `src/core/record/l1-extractor.ts`
- `src/core/scene/scene-extractor.ts`
- `src/core/persona/persona-generator.ts`
- `src/utils/pipeline-manager.ts`
- `src/core/hooks/auto-capture.ts`
- `src/core/hooks/auto-recall.ts`

---

---
title: "Memory layers"
description: "Repo-specific L0 conversation, L1 atom, L2 scene, and L3 persona model: producers, storage shapes, pipeline triggers, and drill-down paths."
---

`MemoryPipelineManager` schedules the long-term pyramid **L0 → L1 → L2 → L3**. Live capture enters through `performAutoCapture` (`agent_end`): L0 is written immediately, then `notifyConversation(sessionKey, [])` advances conversation counters. Extraction is **not** done in the hook; L1/L2/L3 run when the pipeline thresholds fire. Recall enters through `performAutoRecall` (before prompt build): hybrid/keyword/embedding L1 search, full L2 scene navigation, and L3 `persona.md`. Default OpenClaw data root: `~/.openclaw/memory-tdai/`.

## Layer map

| Layer | Role | Producer | Primary on-disk shape | Index / search | Downstream consumer |
| --- | --- | --- | --- | --- | --- |
| **L0 Conversation** | Incremental raw user/assistant turns | `recordConversation` via `performAutoCapture` | `conversations/YYYY-MM-DD.jsonl` (one message per line) | Optional `IMemoryStore.upsertL0` (+ deferred embed on SQLite) | L1 runner (`queryL0GroupedBySessionId` or JSONL fallback) |
| **L1 Atom** | Structured facts: `persona` / `episodic` / `instruction` | `extractL1Memories` → `writeMemory` | `records/YYYY-MM-DD.jsonl` (append-only) | Dual-write `upsertL1` + FTS/vector | L2 scene extraction; `tdai_memory_search`; auto-recall prepend |
| **L2 Scene** | Thematic Markdown scene blocks + index | `SceneExtractor.extract` (tool-enabled LLM, sandboxed to `scene_blocks/`) | `scene_blocks/*.md` + `.metadata/scene_index.json` | Profile sync optional (`type: "l2"`) | L3 persona; auto-recall `<scene-navigation>` |
| **L3 Persona** | User profile Markdown | `PersonaGenerator.generateLocalPersona` when `PersonaTrigger` fires | `persona.md` (body + appended scene nav) | Profile sync optional (`type: "l3"`) | Auto-recall `<user-persona>` |

**Heterogeneous storage rule:** lower layers (L0/L1) prefer DB/JSONL for retrieval and evidence; upper layers (L2/L3) prefer human-readable Markdown for high-density structure. VectorStore (SQLite or TCVDB) is the retrieval engine when healthy; JSONL remains the local append-only backup for L0/L1.

## End-to-end flow

```mermaid
flowchart TD
  AE[agent_end / performAutoCapture] --> L0W[L0 JSONL + optional L0 vectors]
  L0W --> N[notifyConversation]
  N -->|count >= threshold or idle timeout| L1Q[SerialQueue L1]
  L1Q --> L1X[extractL1Memories + writeMemory]
  L1X --> L1S[records/*.jsonl + upsertL1]
  L1X -->|delayAfterL1 + min/max interval| L2Q[SerialQueue L2]
  L2Q --> L2X[SceneExtractor on scene_blocks/]
  L2X --> L3G{PersonaTrigger}
  L3G -->|should generate| L3X[PersonaGenerator → persona.md]
  BP[before_prompt_build / performAutoRecall] --> R1[L1 hybrid search]
  BP --> R2[scene_index → navigation]
  BP --> R3[persona.md body]
  R1 --> PRE[prependContext relevant-memories]
  R2 --> SYS[appendSystemContext]
  R3 --> SYS
```

Wiring: `createPipeline` / `createL1Runner` / `createL2Runner` / `createL3Runner` in the pipeline factory attach runners to `MemoryPipelineManager`. The same factories serve live OpenClaw registration and `seed` CLI.

## L0 — Conversation

**Producer:** `performAutoCapture` holds a per-session file lock via `CheckpointManager.captureAtomically`, then calls `recordConversation`.

**Incremental capture (dual guard):**
1. **Position slice** — `originalUserMessageCount` from `before_prompt_build` keeps only messages appended after that prompt.
2. **Timestamp cursor** — `afterTimestamp` (checkpoint max timestamp; cold start may floor on plugin start time) keeps `timestamp > cursor`.

**Sanitization:** strip injected tags (`sanitizeText`), strip assistant fenced code blocks, drop noise via `shouldCaptureL0`. Polluted user text (framework `prependContext`) is replaced with cached `originalUserText` when timestamps match.

**JSONL line shape (`L0MessageRecord`):**

```json
{
  "sessionKey": "agent:main:main",
  "sessionId": "...",
  "recordedAt": "2026-07-09T12:00:00.000Z",
  "id": "msg_…",
  "role": "user",
  "content": "…",
  "timestamp": 1720000000000
}
```

Daily shard: all sessions merge into the same `conversations/YYYY-MM-DD.jsonl`; `sessionKey` is a field, not the filename.

**Vector path:** when `vectorStore` is present, each message becomes an `L0Record`. SQLite (`supportsDeferredEmbedding`) writes metadata/FTS first and embeds in the background; TCVDB embeds synchronously when dimensions &gt; 0.

**Scheduler note:** capture calls `notifyConversation(sessionKey, [])` with an **empty** buffer. L1 does not consume in-memory messages; it re-reads L0 from VectorStore (primary) or JSONL (fallback) using `last_l1_cursor`.

## L1 — Atom memories

**Producer:** L1 runner groups new L0 messages by `sessionId`, then `extractL1Memories`.

**Pipeline inside one L1 run:**
1. Quality gate `shouldExtractL1` (stricter than L0).
2. Split background vs new (`maxMessagesPerExtraction` default 10, background default 5).
3. Single LLM call: scene segmentation + memory extraction (JSON mode).
4. Optional batch conflict detection against existing L1 (`enableDedup`, vector top-K = `embedding.conflictRecallTopK`, default 5).
5. Persist via `writeMemory` with actions `store` | `update` | `merge` | `skip`.

**Record shape (`MemoryRecord`):**

| Field | Meaning |
| --- | --- |
| `id` | `m_<ts>_<hex>` |
| `content` | Atomic memory text |
| `type` | `persona` \| `episodic` \| `instruction` |
| `priority` | 0–100 (higher more important); `-1` = strict global instruction |
| `scene_name` | Soft scene label from L1 prompt (not yet an L2 file) |
| `source_message_ids` | L0 message `id`s that grounded this atom |
| `metadata` | e.g. episodic `activity_start_time` / `activity_end_time` |
| `timestamps` | Merge history trail |
| `createdAt` / `updatedAt` | ISO times |
| `sessionKey` / `sessionId` | Provenance |

**Storage:** append to `records/YYYY-MM-DD.jsonl`. On update/merge, VectorStore deletes target IDs immediately; JSONL stays append-only and is reconciled later by memory-cleaner.

**Cursor:** L1 advances `last_l1_cursor` to max L0 `recordedAtMs` (write time), not message timestamp — TCVDB-safe.

## L2 — Scene blocks

**Producer:** L2 runner loads L1 rows for the session with `updatedAfter` cursor (`queryMemoryRecords` / JSONL filter), then `SceneExtractor.extract`.

**LLM sandbox:** workspace is `scene_blocks/` only. System files (`.metadata/scene_index.json`, checkpoint, `persona.md`) are not visible to the tool-enabled runner.

**Scene file format:**

```markdown
-----META-START-----
created: …
updated: …
summary: …
heat: 0
-----META-END-----

… scene narrative …
```

**Index:** engineering-side `syncSceneIndex` rebuilds `.metadata/scene_index.json` from `*.md` META fields (`filename`, `summary`, `heat`, `created`, `updated`).

**Capacity policy (`persona.maxScenes`, default 15):**
- ≥ max: must MERGE before CREATE
- max − 1: UPDATE only (CREATE blocked)
- ≥ max − 3: soft prefer UPDATE/MERGE

**Out-of-band L3 signal:** extractor text may contain `[PERSONA_UPDATE_REQUEST]…[/PERSONA_UPDATE_REQUEST]` or `PERSONA_UPDATE_REQUEST: …`, which sets checkpoint `request_persona_update` for the next L3 evaluation.

**Backup:** scene directory snapshots under `.backup/` before extract (`sceneBackupCount`, default 10).

## L3 — Persona

**Producer:** `createL3Runner` → `PersonaTrigger.shouldGenerate` → `PersonaGenerator.generateLocalPersona`.

**Trigger priority (`PersonaTrigger`):**

| Priority | Condition |
| --- | --- |
| P1 | `request_persona_update` set (agent/L2 explicit request) |
| P2 | Cold start: `scenes_processed > 0`, never generated, scene files exist |
| P2.5 | Recovery: generated before but `persona.md` body empty/missing |
| P3 | First scene block: `scenes_processed === 1` and new memories since persona |
| P4 | Threshold: `memories_since_last_persona >= persona.triggerEveryN` (default **50**) |

**Generation:** reads changed scene blocks since `last_persona_time`, mode `first` | `incremental`, tool-writes `persona.md` under dataDir, then strips LLM-added nav, escapes XML tags, and appends fresh scene navigation from the index.

**Global mutex:** L3 queue concurrency = 1 with pending-flag dedup so concurrent L2 completions collapse into one persona run.

## Pipeline triggers and timers

Config group: `pipeline` (defaults from `resolveMemoryTdaiConfig`).

| Knob | Default | Effect |
| --- | --- | --- |
| `everyNConversations` | `5` | L1 when per-session `conversation_count` reaches threshold |
| `enableWarmup` | `true` | Threshold sequence `1 → 2 → 4 → … → everyN`, then steady-state |
| `l1IdleTimeoutSeconds` | `600` | Resettable idle debounce; flush below-threshold buffers |
| `l2DelayAfterL1Seconds` | `10` | After L1 success, pull L2 fire time earlier to `max(now+delay, lastL2+min)` |
| `l2MinIntervalSeconds` | `900` | Floor between L2 runs per session |
| `l2MaxIntervalSeconds` | `3600` | After L2 completes, schedule next at `now+maxInterval` if session still active |
| `sessionActiveWindowHours` | `24` | Cold sessions cancel L2 poll until next L1 |

**L1 paths:** conversation threshold · idle timeout · shutdown flush.  
**L2 paths:** delay-after-L1 (downward-only timer) · maxInterval poll · shutdown flush.  
**L3 path:** after L2 completion, subject to `PersonaTrigger`.

Both L1 and L2 use `SerialQueue` (concurrency 1) per manager. Pipeline session state persists through the checkpoint persister (`mergePipelineStates` → `.metadata/recall_checkpoint.json`).

## Recall injection (read path)

`performAutoRecall` races work against `recall.timeoutMs` (default 5000 ms); on timeout it skips injection rather than blocking the turn.

| Injected region | Source | XML / structure |
| --- | --- | --- |
| `prependContext` (user prefix, dynamic) | L1 search (`recall.strategy` default `hybrid`) | `<relevant-memories>…` |
| `appendSystemContext` (system tail, stable) | L3 persona body | `<user-persona>…` |
| | L2 full navigation from `scene_index` | `<scene-navigation>…` (absolute `Path:` when `dataDir` known) |
| | Static tools guide | `<memory-tools-guide>` — `tdai_memory_search`, `tdai_conversation_search`, `read_file` on scene paths; **combined max 3 tool calls per turn** |

Search strategies: `keyword` (FTS/BM25), `embedding` (cosine), `hybrid` (RRF merge). Defaults: `maxResults=5`, `scoreThreshold=0.3`.

## Drill-down paths

Progressive disclosure is intentional; high layers never replace lower evidence.

| From | To | Mechanism |
| --- | --- | --- |
| Injected L1 line / agent need | More L1 atoms | Tool `tdai_memory_search` |
| L1 atom or timeline need | Raw turns | Tool `tdai_conversation_search` (L0 vectors/FTS) |
| Scene summary in nav | Full scene narrative | `read_file` on absolute path under `scene_blocks/` |
| L1 `source_message_ids` | Grounding messages | Message `id`s recorded at L0 (`msg_…`) and referenced by L1 JSONL |
| L3 persona claim | Scenes that changed | Persona generation diffs scenes by `updated` vs `last_persona_time`; nav lists heat-sorted scenes |
| L2 heat / summary | Priority for agent attention | Navigation heat emoji bands and summary lines |

Offload Mermaid `node_id` drill-down is a **separate** short-term stack (`context-offload`), not this L0–L3 long-term pyramid.

## On-disk layout (long-term tree)

```text
~/.openclaw/memory-tdai/          # OpenClaw default (Hermes: MEMORY_TENCENTDB_ROOT)
├── conversations/YYYY-MM-DD.jsonl   # L0
├── records/YYYY-MM-DD.jsonl         # L1
├── scene_blocks/*.md                # L2
├── persona.md                       # L3 + scene navigation appendix
├── vectors.db                       # SQLite store when storeBackend=sqlite
├── .metadata/
│   ├── recall_checkpoint.json       # cursors, pipeline session state, persona triggers
│   └── scene_index.json
└── .backup/                         # persona + scene snapshots
```

`initDataDirectories` ensures `conversations`, `records`, `scene_blocks`, `.metadata`, `.backup`.

## Verification signals

| Signal | Healthy meaning |
| --- | --- |
| Logs `[memory-tdai] [capture]` with `L0 recorded: N messages` | Capture wrote JSONL |
| Logs `Scheduler notified` / `Conversation threshold reached` | Pipeline counting |
| Logs `[pipeline-factory] [l1] L1 complete: extracted=… stored=…` | Atoms persisted |
| New lines under `records/` and growing VectorStore L1 rows | L1 dual-write |
| Logs `[L2] Extraction complete` and new/updated `scene_blocks/*.md` | Scenes materializing |
| Logs `[L3] Persona generation succeeded` and non-empty `persona.md` body | Profile available for recall |
| Recall log `search=…hits=… persona=…chars scene=loaded` | Injection path armed |
| Tools guide present when any memory context injects | Agent can drill down |

## Config groups that own each layer

| Config path | Layer impact |
| --- | --- |
| `capture.enabled` / `excludeAgents` / `l0l1RetentionDays` | L0 write gate and retention |
| `extraction.enabled` / `enableDedup` / `maxMemoriesPerSession` / `model` | L1 LLM extract |
| `pipeline.*` | L1/L2 scheduling |
| `persona.triggerEveryN` / `maxScenes` / `backupCount` / `sceneBackupCount` / `model` | L2 capacity + L3 cadence |
| `recall.*` | Injection strategy and budgets |
| `embedding.*` / `storeBackend` / `tcvdb.*` / `bm25.*` | Vector/FTS dual-write and search quality |

## Failure modes (layer-specific)

- **L0 empty every turn:** position slice + cursor may filter all messages; cold sessions without checkpoint can floor on plugin start time and skip pre-plugin history intentionally.
- **L1 never runs:** missing LLM runner/config, or pipeline not notified (`capture.enabled` false / no scheduler).
- **L1 store-only, no vectors:** store degraded or embedding provider `none` — JSONL still written; hybrid recall falls back.
- **L2 skipped with `skipped: true`:** no L1 rows after cursor (expected when idle).
- **L2 stuck at scene cap:** merge required when `scene_count >= maxScenes`.
- **L3 never updates:** `memories_since_last_persona` below `triggerEveryN` and no cold-start/request/recovery condition.
- **Recall timeout:** `timeoutMs` exceeded → no injection that turn; tools still available next turns.

## Related pages

<Cards>
  <Card title="Context offload" href="/context-offload">
    Symbolic short-term stack (tool-pair offload, Mermaid canvas) independent of L0–L3.
  </Card>
  <Card title="Data directories" href="/data-directories">
    Full OpenClaw / Hermes / offload tree layout and env overrides.
  </Card>
  <Card title="Configuration reference" href="/configuration-reference">
    Full schema defaults for capture, extraction, persona, pipeline, recall, embedding.
  </Card>
  <Card title="Agent tools" href="/agent-tools">
    `tdai_memory_search` and `tdai_conversation_search` parameters, hybrid RRF, 3-calls-per-turn limit.
  </Card>
  <Card title="Seed historical conversations" href="/seed-conversations">
    Offline L0→L1→L2→L3 population via `openclaw memory-tdai seed`.
  </Card>
  <Card title="Configure storage backends" href="/configure-storage">
    SQLite vs TCVDB binding for L0/L1 vectors and L2/L3 profile sync.
  </Card>
</Cards>