# Storage backends

> sqlite (SQLite + sqlite-vec + FTS5) vs tcvdb store backends, BM25 sparse encoding, hybrid RRF retrieval, embedding service roles, and factory selection constraints.

- Repository: TencentCloud/TencentDB-Agent-Memory
- GitHub: https://github.com/TencentCloud/TencentDB-Agent-Memory
- Human docs: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-5a33bbf5540a
- Complete Markdown: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-5a33bbf5540a/llms-full.txt

## Source Files

- `src/core/store/factory.ts`
- `src/core/store/types.ts`
- `src/core/store/sqlite.ts`
- `src/core/store/tcvdb.ts`
- `src/core/store/embedding.ts`
- `src/core/store/bm25-local.ts`
- `src/core/store/search-utils.ts`

---

---
title: "Storage backends"
description: "sqlite (SQLite + sqlite-vec + FTS5) vs tcvdb store backends, BM25 sparse encoding, hybrid RRF retrieval, embedding service roles, and factory selection constraints."
---

`createStoreBundle()` in `src/core/store/factory.ts` selects the memory persistence stack from `storeBackend` (`sqlite` | `tcvdb`, default `sqlite`) and returns a `StoreBundle`: `store` (`IMemoryStore`), `embedding` (`IEmbeddingService`), optional `bm25Encoder`, and a `storeSnapshot` for manifest binding. Upper layers (hooks, tools, pipeline, `TdaiCore`) depend only on `IMemoryStore` and capability flags—not on concrete backends.

## Architecture

```mermaid
flowchart TB
  subgraph callers [Callers]
    AR[auto-recall / tools]
    AC[auto-capture]
    PF[pipeline-factory]
  end

  subgraph factory [createStoreBundle]
    CFG["config.storeBackend"]
  end

  subgraph sqlite_path [sqlite default]
    VS[VectorStore]
    DB["dataDir/vectors.db"]
    VEC[sqlite-vec vec0]
    FTS[FTS5 BM25]
    EMB[OpenAIEmbeddingService or none]
  end

  subgraph tcvdb_path [tcvdb]
    TMS[TcvdbMemoryStore]
    CLI[TcvdbClient HTTP]
    SSE[server-side embedding]
    BM25[BM25LocalEncoder sparse]
    HYB["hybridSearch dense+sparse+RRF k=60"]
  end

  PF --> factory
  CFG -->|sqlite| VS
  CFG -->|tcvdb| TMS
  VS --> DB
  VS --> VEC
  VS --> FTS
  factory --> EMB
  TMS --> CLI
  TMS --> SSE
  TMS --> BM25
  CLI --> HYB
  AR --> VS
  AR --> TMS
  AC --> VS
  AC --> TMS
```

| Concern | `sqlite` | `tcvdb` |
| --- | --- | --- |
| Implementation | `VectorStore` (`sqlite.ts`) | `TcvdbMemoryStore` (`tcvdb.ts`) |
| On-disk / remote | `{dataDir}/vectors.db` | Tencent VectorDB HTTP API |
| Dense vectors | Client embed → `l1_vec` / `l0_vec` (sqlite-vec) | Collection `embedding` + server-side `embeddingItems` |
| Keyword / sparse | FTS5 + jieba tokenization | Client BM25 sparse vectors (`sparse_vector`) |
| Hybrid RRF | Client-side: FTS list + vector list → `rrfMerge` / local RRF | Native `/document/hybridSearch` with `rerank: { method: "rrf", k: 60 }` |
| Embedding service | Remote OpenAI-compatible when configured; else none / keyword-only | Always `NoopEmbeddingService` (dims `0`) |
| `supportsDeferredEmbedding` | `true` (metadata first, background `updateL0Embedding`) | not set (server embeds on upsert/search) |
| Profile sync (L2/L3) | local files primarily | remote `profiles` collection + optional sync APIs |

## Factory selection

`createStoreBundle(config, { dataDir, logger })` always builds the BM25 encoder from `config.bm25` first, then switches on `config.storeBackend`.

### `storeBackend: "tcvdb"`

Hard requirements (throws if missing):

| Field | Constraint |
| --- | --- |
| `tcvdb.url` | Required |
| `tcvdb.apiKey` | Required |
| `tcvdb.database` | Required (unique database name) |

Other fields passed into `TcvdbMemoryStore`: `username` (default `"root"`), `embeddingModel` (default `"bge-large-zh"`), `timeout` (default `10000`), optional `caPemPath`.

Returned bundle:

- `store`: `TcvdbMemoryStore`
- `embedding`: `NoopEmbeddingService`
- `bm25Encoder`: from `createBM25Encoder` when `bm25.enabled`
- `storeSnapshot`: `{ type: "tcvdb", tcvdbUrl, tcvdbDatabase, tcvdbAlias? }`

### `storeBackend: "sqlite"` (default)

Any value other than `"tcvdb"` resolves to `"sqlite"` in `parseConfig`.

Embedding construction (client-side only):

- Created when `embedding.enabled && embedding.provider !== "local" && embedding.apiKey`
- Otherwise no embedding service (vector search off; FTS may still work)
- `dimensions` from config; `0` when `provider === "none"` defers sqlite-vec tables

Returned bundle:

- `store`: `VectorStore(path.join(dataDir, "vectors.db"), dimensions, logger)`
- `embedding`: remote service or effectively absent
- `bm25Encoder`: still created if enabled (used primarily by tcvdb paths; sqlite keyword search uses FTS5, not sparse vectors)
- `storeSnapshot`: `{ type: "sqlite", sqlitePath: relative "vectors.db" }`

```json
{
  "storeBackend": "sqlite",
  "embedding": {
    "provider": "none"
  }
}
```

```json
{
  "storeBackend": "tcvdb",
  "tcvdb": {
    "url": "http://10.0.1.1:8100",
    "apiKey": "<key>",
    "database": "agent_memory_prod",
    "embeddingModel": "bge-large-zh",
    "username": "root",
    "timeout": 10000
  },
  "bm25": {
    "enabled": true,
    "language": "zh"
  }
}
```

<Warning>
Switching `storeBackend` or TCVDB database binding after data exists is a migration concern. Use the offline migrate path rather than pointing a live process at a different backend without rewriting data.
</Warning>

## `IMemoryStore` contract

All backends implement the same surface in `src/core/store/types.ts`:

- Lifecycle: `init`, `isDegraded`, `getCapabilities`, `close`
- L1 CRUD / search: `upsertL1`, `deleteL1*`, `searchL1Vector`, `searchL1Fts`, optional `searchL1Hybrid`
- L0 CRUD / search: `upsertL0`, optional `updateL0Embedding`, `searchL0Vector`, `searchL0Fts`
- Optional profile ops: `pullProfiles`, `syncProfiles`, `deleteProfiles` (TCVDB)
- `reindexAll` (meaningful for sqlite client embeddings; TCVDB no-ops with server-side embedding)
- Fault tolerance: methods return empty/`false` on failure unless documented otherwise

### Capability flags

```ts
interface StoreCapabilities {
  vectorSearch: boolean;
  ftsSearch: boolean;
  nativeHybridSearch: boolean;
  sparseVectors: boolean;
}
```

| Flag | sqlite (`VectorStore`) | tcvdb (`TcvdbMemoryStore`) |
| --- | --- | --- |
| `vectorSearch` | `vecTablesReady` (false when dimensions=`0` / no vec0) | always `true` |
| `ftsSearch` | FTS5 created successfully | `!!bm25Encoder` |
| `nativeHybridSearch` | always `false` | `!!bm25Encoder` |
| `sparseVectors` | always `false` | `!!bm25Encoder` |

Callers use these to pick strategy and degrade:

- Auto-recall hybrid: if `nativeHybridSearch`, single `searchL1Hybrid` call; else parallel keyword + embedding + client RRF
- Tools (`tdai_memory_search`, conversation search): FTS + vector in parallel, RRF when both produce hits
- Keyword path needs `isFtsAvailable()` (sqlite FTS5 or tcvdb BM25)

## sqlite backend

### Layout

Single SQLite file: `{pluginDataDir}/vectors.db` (Node `node:sqlite` + `sqlite-vec`).

| Layer | Metadata table | Vector | FTS |
| --- | --- | --- | --- |
| L1 | `l1_records` | `l1_vec` (vec0, cosine) | FTS5 virtual table |
| L0 | `l0_conversations` | `l0_vec` | FTS5 virtual table |

Writes use BEGIN/COMMIT for metadata + vector atomicity. vec0 has no `ON CONFLICT`; upsert is delete + insert. WAL mode is enabled.

### Vector path

- Requires configured dimensions and a remote embedding provider with `apiKey`
- Cosine similarity score: `1.0 - cosine_distance`
- When `embedding.provider` is `"none"`, dimensions resolve to `0`, vec0 tables stay deferred, `vectorSearch` is false → keyword/FTS-only

### FTS5 keyword path

- Index/query tokenization prefers `@node-rs/jieba` `cutForSearch`; falls back to Unicode regex
- Write-side: space-joined tokens in FTS content; query-side: OR-joined quoted phrases via `buildFtsQuery`
- Chinese stop-word filter reduces noise
- FTS ranks map to 0–1 via `bm25RankToScore`
- If FTS5 is unavailable at init, `ftsAvailable` stays false and keyword search is skipped (no O(N) full-scan fallback)

### Deferred embedding

`supportsDeferredEmbedding = true`:

1. Capture writes L0 metadata (+ FTS) with `upsertL0(record, undefined)`
2. Background path calls `updateL0Embedding(recordId, embedding)` when the client embed finishes

### Degradation

If sqlite-vec fails to load or init fails hard, the store enters degraded mode; pipeline-factory may drop the store and fall back to keyword-only behavior for higher layers.

## tcvdb backend

### Collections

Names are prefixed with the configured database to avoid cross-database collisions:

| Logical layer | Collection suffix | Notes |
| --- | --- | --- |
| L1 | `{database}_l1_memories` | Server embedding on field `text` → `vector`; sparse `sparse_vector` |
| L0 | `{database}_l0_conversations` | Server embedding on `message_text` |
| L2/L3 profiles | `{database}_profiles` | Embedding disabled; FLAT dummy vector; versioned rows |

Vector index: prefer `DISK_FLAT` (COSINE, dimension 1024); on API error 15113 / DISK_FLAT unsupported, fall back to HNSW (`M=16`, `efConstruction=200`). Sparse index: inverted, metric `IP`.

### Server-side embedding

- Collection config enables embedding with model `tcvdb.embeddingModel` (default `bge-large-zh`)
- Upsert sends text fields only; dense vectors are generated server-side
- Search uses `embeddingItems` / ANN on the text field—not client `Float32Array`s
- Factory therefore installs `NoopEmbeddingService` (`getDimensions() === 0`, provider `"noop"`)
- Capture skips local embed when dimensions are 0
- `reindexAll` is a no-op (server embeds; rebuild would require drop/recreate)

### Time and filters

- Times stored as `uint64` epoch ms; ISO conversion is internal
- Scalar filters: `session_key`, `session_id`, `type`, `agent_id`, time fields, etc.
- Expired-delete has an 80% safety block (refuses mass delete above threshold)

### Init

- Creates database (idempotent); after create, waits ~5s before collections
- Init failures set `degraded = true`; methods return empty/false

## BM25 sparse encoding

Local encoder (`bm25-local.ts`) uses `@tencentdb-agent-memory/tcvdb-text` (`BM25Encoder.default(language)`).

| Config | Type | Default | Role |
| --- | --- | --- | --- |
| `bm25.enabled` | boolean | `true` | When false, encoder is omitted |
| `bm25.language` | `"zh"` \| `"en"` | `"zh"` | Pre-trained tokenizer/params |

API:

- `encodeTexts(texts)` — document upsert (TF-based sparse vectors)
- `encodeQueries(texts)` — search-time sparse queries (IDF-based)

On TCVDB:

- Upsert attaches `sparse_vector` when encode succeeds
- Hybrid search builds `match` on `sparse_vector` from `encodeQueries`
- Without BM25: dense-only `/document/search` with `embeddingItems`
- With BM25: full hybrid path (capabilities mark FTS/hybrid/sparse true)

BM25 is independent of the client embedding provider: TCVDB dense is server-side; sparse is always client-encoded when enabled.

## Hybrid RRF retrieval

RRF constant: **k = 60** (shared paper default).

### Score fusion

Per ranked list, item score contribution is `1 / (k + rank + 1)`. Scores sum across lists; result sorts by descending RRF score.

Shared helper: `rrfMerge` in `search-utils.ts`. Auto-recall hybrid, memory-search, and conversation-search implement the same formula (some with local copies).

### Strategy matrix (`recall.strategy`)

| Strategy | Behavior |
| --- | --- |
| `keyword` | FTS5 (sqlite) or sparse/hybrid path (tcvdb when BM25 available) |
| `embedding` | Client embed + vector search (sqlite); tcvdb needs query text for server embed |
| `hybrid` (default) | Native hybrid if `nativeHybridSearch`; else parallel keyword + embedding + client RRF |

Fallbacks:

- If embedding service is missing for `embedding`/`hybrid` (sqlite with `provider: none`), auto-recall falls back to keyword
- Tools report effective strategy: `"hybrid"` | `"embedding"` | `"fts"` | `"none"`
- Score threshold default `0.3` (`recall.scoreThreshold`); small FTS result sets may bypass threshold when all ranks are low (IDF edge case)

### TCVDB native hybrid request shape

When BM25 is present:

1. ANN: `fieldName: "text"`, `data: [queryText]` (server embed)
2. Match: `fieldName: "sparse_vector"`, sparse from `encodeQueries`
3. `rerank: { method: "rrf", k: 60 }`
4. Single `hybridSearch` HTTP call

Auto-recall short-circuits to this path when `getCapabilities().nativeHybridSearch` is true—avoids a redundant local `embed()` and second HTTP round-trip.

## Embedding service roles

| Role | Class | When |
| --- | --- | --- |
| Remote OpenAI-compatible | `OpenAIEmbeddingService` | sqlite + provider ≠ `local`/`none` + `apiKey` |
| Local GGUF (internal) | `LocalEmbeddingService` | Not exposed as user config; `provider: "local"` is forced off in `parseConfig` |
| Server-side noop | `NoopEmbeddingService` | Always with tcvdb |
| Disabled | no service / dims 0 | `provider: "none"` (default) or invalid remote config |

Remote config fields that matter for sqlite vector quality:

| Field | Notes |
| --- | --- |
| `baseUrl`, `apiKey`, `model`, `dimensions` | Required for remote; missing fields set `configError` and disable embedding |
| `sendDimensions` | Default `true` (Matryoshka); set `false` for BGE-M3-style backends that reject `dimensions` |
| `timeoutMs` / `recallTimeoutMs` / `captureTimeoutMs` | Per-path timeouts |
| `maxInputChars` | Default `5000` truncate |

`EmbeddingProviderInfo` (`provider` + `model`) drives sqlite reindex detection when the provider/model/dimensions change.

Provider neutrality: any OpenAI-compatible HTTP embedding endpoint works for sqlite; TCVDB uses the instance’s built-in model name (`tcvdb.embeddingModel`) instead of a separate client provider. No particular cloud host is hard-coded beyond the TCVDB HTTP client URL you configure.

## Config field reference

<ParamField body="storeBackend" type="string" required>
Enum: `sqlite` | `tcvdb`. Default `sqlite`.
</ParamField>

<ParamField body="tcvdb.url" type="string" required>
VectorDB instance URL (required when backend is `tcvdb`).
</ParamField>

<ParamField body="tcvdb.apiKey" type="string" required>
API key for VectorDB.
</ParamField>

<ParamField body="tcvdb.database" type="string" required>
Database name; prefixes collection names.
</ParamField>

<ParamField body="tcvdb.username" type="string">
Default `root`.
</ParamField>

<ParamField body="tcvdb.embeddingModel" type="string">
Server-side model; default `bge-large-zh`.
</ParamField>

<ParamField body="tcvdb.timeout" type="number">
HTTP timeout ms; default `10000`.
</ParamField>

<ParamField body="tcvdb.caPemPath" type="string">
Optional CA PEM path for HTTPS instances.
</ParamField>

<ParamField body="bm25.enabled" type="boolean">
Default `true`. Disables sparse vectors / native hybrid when false.
</ParamField>

<ParamField body="bm25.language" type="string">
`zh` or `en`; default `zh`.
</ParamField>

<ParamField body="embedding.provider" type="string">
Default `none` (keyword-only on sqlite). Remote names are OpenAI-compatible endpoints.
</ParamField>

<ParamField body="recall.strategy" type="string">
`hybrid` | `embedding` | `keyword`; default `hybrid`.
</ParamField>

## Operational notes

| Topic | Behavior |
| --- | --- |
| Manifest | First init writes store binding; config drift is logged against `storeSnapshot` |
| Degraded store | `isDegraded()` true → pipeline may null out store; searches return empty |
| Migration | Offline `migrate-sqlite-to-tcvdb` moves L0/L1/profile data and can rewrite config |
| Inspection | `read-local-memory` for sqlite artifacts; `export-tencent-vdb` for TCVDB collections |
| Gateway / Hermes | Same factory path via standalone host adapter and `tdai-gateway.json` / ctl config |

### Failure modes

| Symptom | Likely cause |
| --- | --- |
| Factory throw on start | Missing `tcvdb.url` / `apiKey` / `database` |
| No vector hits on sqlite | `embedding.provider` is `none` or missing remote fields; dims 0 |
| Embedding HTTP 400 | Matryoshka/`dimensions` rejected → set `sendDimensions: false` |
| Hybrid collapses to dense-only (tcvdb) | `bm25.enabled: false` or encode failure |
| Keyword empty | FTS5 not available (sqlite) or BM25 off (tcvdb) |
| Store degraded | sqlite-vec load failure or TCVDB init HTTP errors |

## Next

<CardGroup>
  <Card title="Use Tencent VectorDB" href="/use-tcvdb">
    Switch `storeBackend` to `tcvdb`, required connection fields, BM25 language, server embedding model, and CA PEM.
  </Card>
  <Card title="Configure embedding" href="/configure-embedding">
    OpenAI-compatible providers, dimensions, `sendDimensions`, timeouts, and keyword-only degradation.
  </Card>
  <Card title="Migrate SQLite to TCVDB" href="/migrate-to-tcvdb">
    Offline migrate-sqlite-to-tcvdb, layer selection, config rewrite, and verification.
  </Card>
  <Card title="Plugin configuration reference" href="/plugin-config-reference">
    Full schema for storeBackend, tcvdb, bm25, embedding, and recall.
  </Card>
  <Card title="Inspect local memory" href="/inspect-local-memory">
    `vectors.db`, L0–L3 layout, and export tools for both backends.
  </Card>
  <Card title="Agent tools" href="/agent-tools">
    Hybrid / embedding / keyword strategies on `tdai_memory_search` and conversation search.
  </Card>
</CardGroup>