# Migrate SQLite to TCVDB > Offline migrate-sqlite-to-tcvdb workflow: dry-run, layer selection (l0/l1/profile), TCVDB connection flags, config rewrite, manifest updates, and rollback-safe verification. - Repository: TencentCloud/TencentDB-Agent-Memory - GitHub: https://github.com/TencentCloud/TencentDB-Agent-Memory - Human docs: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-5a33bbf5540a - Complete Markdown: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-5a33bbf5540a/llms-full.txt ## Source Files - `scripts/migrate-sqlite-to-tcvdb/README.md` - `scripts/migrate-sqlite-to-tcvdb/cli-entry.ts` - `scripts/migrate-sqlite-to-tcvdb/sqlite-to-tcvdb.ts` - `scripts/migrate-sqlite-to-tcvdb/config-write.ts` - `scripts/migrate-sqlite-to-tcvdb/manifest-write.ts` - `SKILL-MIGRATION.md` - `bin/migrate-sqlite-to-tcvdb.mjs` --- --- title: "Migrate SQLite to TCVDB" description: "Offline migrate-sqlite-to-tcvdb workflow: dry-run, layer selection (l0/l1/profile), TCVDB connection flags, config rewrite, manifest updates, and rollback-safe verification." --- `migrate-sqlite-to-tcvdb` is an offline CLI that copies L0/L1 rows from local `vectors.db` and filesystem profiles (L2 scene blocks + L3 `persona.md`) into Tencent VectorDB (TCVDB), then rewrites OpenClaw plugin config and the data-dir manifest so runtime uses `storeBackend: "tcvdb"`. It does not delete the SQLite source. ## What migrates | Layer flag | Source | Target operation | |---|---|---| | `l0` | SQLite L0 message rows (`VectorStore`) | `upsertL0` / `upsertL0Batch` | | `l1` | SQLite L1 memory rows | `upsertL1` / `upsertL1Batch` | | `l2` or `l3` | Local files via `listLocalProfiles`: `scene_blocks/*.md` (type `l2`) and `persona.md` (type `l3`) | `syncProfiles` | Default `--layers` is `l0,l1,l2,l3`. Selecting either `l2` or `l3` runs the full local profile sync (both scene blocks and persona when present). TCVDB uses **server-side** dense embedding (`embeddingModel`); local SQLite vectors are not copied as client embedding arrays. BM25 sparse encoding can be enabled on the target with `--bm25-language zh|en` (default `zh`). ```text plugin-data-dir/ vectors.db → L0 + L1 source (default path) scene_blocks/*.md → L2 profiles persona.md → L3 profile .metadata/manifest.json → rewritten to store.type = tcvdb openclaw.json plugins.entries..config storeBackend, tcvdb{...}, bm25{...} ``` ## Prerequisites - Node.js ≥ 22.16 - Plugin installed (package bin available), or a local checkout with scripts built - Readable plugin data dir and `openclaw.json` - Reachable TCVDB endpoint, credentials, database name, and embedding model name - Prefer a **stopped** gateway during cutover so nothing writes to SQLite mid-migrate Build the TypeScript migrator before running the bin (required for source checkouts; published packages ship prebuilt `scripts/migrate-sqlite-to-tcvdb/dist/`): ```bash npm run build:migrate-sqlite-to-vdb ``` ## Invoke the CLI Package bin and npm script both resolve to `bin/migrate-sqlite-to-tcvdb.mjs` → compiled `cli-entry.js`. ```bash # Via package bin (after install) migrate-sqlite-to-tcvdb --help # Via npm script in a checkout npm run migrate-sqlite-to-tcvdb -- --help ``` Stdout prints a JSON preflight/summary. Progress and errors go to stderr with tag `[memory-tdai][migrate]` / `[memory-tdai][migrate-cli]`. ## Workflow Inspect source counts and target connection params without writing TCVDB, config, or manifest. ```bash export TCVDB_API_KEY='...' migrate-sqlite-to-tcvdb \ --plugin-data-dir ~/.openclaw/memory-tdai \ --openclaw-config-path ~/.openclaw/openclaw.json \ --tcvdb-url http://127.0.0.1:80 \ --tcvdb-username root \ --tcvdb-api-key-env TCVDB_API_KEY \ --tcvdb-database agent_memory_prod \ --tcvdb-embedding-model bge-large-zh \ --dry-run ``` Expected summary fields (among others): - `source.l0Count` / `l1Count` / `profileCount` - `source.manifestExists` / `manifestStoreType` - `target.*` connection and BM25 flags - `options.*` apply/verify toggles - `dryRun: true` and **no** `migration` block If the data dir or `vectors.db` is missing, the tool returns an empty source summary and does not fail (fresh install / nothing to migrate). Copy `openclaw.json` before a live run. The migrator rewrites `plugins.entries..config` when `--apply-config` is on (default). Manifest backups are automatic (see below); treat OpenClaw config backup as operator-owned. Omit `--dry-run`. Use `--yes` only if your automation expects that flag; **writes are gated by the absence of `--dry-run`**, not by an interactive prompt in the current implementation. ```bash migrate-sqlite-to-tcvdb \ --plugin-data-dir ~/.openclaw/memory-tdai \ --openclaw-config-path ~/.openclaw/openclaw.json \ --tcvdb-url http://127.0.0.1:80 \ --tcvdb-username root \ --tcvdb-api-key-env TCVDB_API_KEY \ --tcvdb-database agent_memory_prod \ --tcvdb-embedding-model bge-large-zh \ --yes ``` Order of work when source has data: 1. Open SQLite source; init `TcvdbMemoryStore` (BM25 encoder attached when enabled) 2. Abort if target already has L0/L1/profiles and `--fail-if-target-nonempty` is true (default) 3. Page migrate L1 then L0 (`DEFAULT_MIGRATION_PAGE_SIZE` = 50) 4. Sync profiles when `l2` or `l3` is selected 5. Optionally verify counts (default: wait ~10s for remote settle, then exact match) 6. Write OpenClaw config patch (`storeBackend: "tcvdb"`, `tcvdb`, `bm25`) 7. Rewrite `/.metadata/manifest.json` to `store.type: "tcvdb"` (backup `manifest.json.migrate.bak` when updating an existing file) Restart OpenClaw gateway (or Hermes-attached gateway) so the plugin reloads with `storeBackend: "tcvdb"`. Smoke-check recall/search and confirm no degraded store init in logs. ## CLI flags ### Required connection and paths Plugin data directory (conversations/records/scene_blocks/vectors.db layout). Path to `openclaw.json` (JSON/JSON5). Used for apply-config writes and preflight readability checks. TCVDB service URL. TCVDB username (examples commonly use `root`). Target TCVDB database name. Server-side embedding model name written into the target store and config (e.g. `bge-large-zh`). Plaintext API key. Mutually exclusive with `--tcvdb-api-key-env`. Environment variable name that holds the API key. Mutually exclusive with `--tcvdb-api-key`. ### Optional controls | Flag | Default | Behavior | |---|---|---| | `--sqlite-path` | `/vectors.db` | Alternate SQLite file | | `--plugin-id` | `memory-tencentdb` | Config entry key under `plugins.entries` | | `--layers` | `l0,l1,l2,l3` | Comma-separated subset of `l0`,`l1`,`l2`,`l3` | | `--tcvdb-alias` | `""` | Alias stored in config/manifest | | `--tcvdb-timeout-ms` | `10000` | Request timeout (must be > 0) | | `--tcvdb-ca-pem` | — | CA PEM path for HTTPS TCVDB during **migration client** init | | `--bm25-language` | `zh` | `zh` or `en` | | `--summary-json-path` | — | Write full summary JSON to this path | | `--job-id` | — | Accepted tracking id (parsed into options) | | `--dry-run` | `false` | Preflight only; no TCVDB/config/manifest writes | | `--yes` | `false` | Documented skip-confirm; parsed into summary | | `--no-apply-config` | apply on | Skip openclaw.json rewrite | | `--no-config-backup` | backup flag on | Parsed into summary; operator should still backup config manually | | `--no-rewrite-manifest` | rewrite on | Skip `.metadata/manifest.json` update | | `--no-fail-if-target-nonempty` | fail on | Allow non-empty target (append-style) | | `--no-verify-counts` | verify on | Skip exact L0/L1/profile count match | | `--no-bm25-enabled` | BM25 on | Disable BM25 sparse vectors on target | Boolean flags use Node `parseArgs` with `allowNegative: true` (`--no-*` form). ## Config rewrite When `applyConfig` is true, the tool merges into: `plugins.entries[].config` ```json { "storeBackend": "tcvdb", "tcvdb": { "url": "...", "username": "...", "apiKey": "...", "database": "...", "alias": "...", "embeddingModel": "...", "timeout": 10000 }, "bm25": { "enabled": true, "language": "zh" } } ``` Notes: - Existing sibling plugin config keys are preserved; `tcvdb` / `bm25` objects are deep-merged. - Output is rewritten as pretty-printed JSON (JSON5 input is accepted). - `caPemPath` is used for the migrator client when `--tcvdb-ca-pem` is set; it is **not** included in the config patch. Set `tcvdb.caPemPath` in OpenClaw config separately if the runtime needs a custom CA. ## Manifest rewrite Path: `/.metadata/manifest.json` - Missing manifest → create `version: 1` with `store.type: "tcvdb"` and TCVDB url/database/alias. - Existing manifest → copy to `manifest.json.migrate.bak` in the same `.metadata/` directory, then update `store` while keeping other fields (e.g. `seed`). ## Verification and safety | Guard | Default | Failure mode | |---|---|---| | Empty / missing source | soft skip | Log and finish without data copy | | Degraded SQLite open | hard fail | Throws on preflight or reopen | | Degraded TCVDB init | hard fail | Throws before migrate | | Non-empty target | hard fail | `Target store is not empty (L1=…, L0=…, profiles=…)` | | Count verify | hard fail | `L1/L0/Profile count verification failed: source=…, target=…` after ~10s settle | | Batch upsert zero rows | hard fail | Failed batch migrate for L0/L1 | **Rollback-safe properties** - Source `vectors.db` and on-disk profiles are not removed by the migrator. - Manifest previous content is recoverable from `manifest.json.migrate.bak` when an existing manifest was updated. - To roll back runtime: restore prior `openclaw.json` (`storeBackend: "sqlite"`), restore or rewrite manifest `store` to sqlite if needed, restart gateway. TCVDB data left behind is harmless if unused. ## Common recipes ```bash title="L1 only" migrate-sqlite-to-tcvdb \ --plugin-data-dir ~/.openclaw/memory-tdai \ --openclaw-config-path ~/.openclaw/openclaw.json \ --tcvdb-url http://127.0.0.1:80 \ --tcvdb-username root \ --tcvdb-api-key-env TCVDB_API_KEY \ --tcvdb-database agent_memory_prod \ --tcvdb-embedding-model bge-large-zh \ --layers l1 \ --yes ``` ```bash title="L0+L1 without profiles" migrate-sqlite-to-tcvdb \ --plugin-data-dir ~/.openclaw/memory-tdai \ --openclaw-config-path ~/.openclaw/openclaw.json \ --tcvdb-url http://127.0.0.1:80 \ --tcvdb-username root \ --tcvdb-api-key-env TCVDB_API_KEY \ --tcvdb-database agent_memory_prod \ --tcvdb-embedding-model bge-large-zh \ --layers l0,l1 \ --yes ``` ```bash title="English BM25, no auto config/manifest" migrate-sqlite-to-tcvdb \ --plugin-data-dir ~/.openclaw/memory-tdai \ --openclaw-config-path ~/.openclaw/openclaw.json \ --tcvdb-url http://127.0.0.1:80 \ --tcvdb-username root \ --tcvdb-api-key-env TCVDB_API_KEY \ --tcvdb-database agent_memory_prod \ --tcvdb-embedding-model bge-large-en-v1.5 \ --bm25-language en \ --no-apply-config \ --no-rewrite-manifest \ --yes ``` ```bash title="Append to non-empty target (CI-style)" migrate-sqlite-to-tcvdb \ --plugin-data-dir ~/.openclaw/memory-tdai \ --openclaw-config-path ~/.openclaw/openclaw.json \ --tcvdb-url http://127.0.0.1:80 \ --tcvdb-username root \ --tcvdb-api-key-env TCVDB_API_KEY \ --tcvdb-database agent_memory_prod \ --tcvdb-embedding-model bge-large-zh \ --no-fail-if-target-nonempty \ --no-verify-counts \ --summary-json-path ./migration-report.json \ --yes ``` ## Summary JSON shape On success, CLI stdout is a `MigrationPreflightSummary`. After a live run, `migration` includes: | Field | Meaning | |---|---| | `l0Migrated` / `l1Migrated` / `profileMigrated` | Rows/profiles written this run | | `targetL0Count` / `targetL1Count` / `targetProfileCount` | Post-run target counts | | `configWritten` | Whether openclaw config was patched | | `manifestWritten` | Whether manifest was created/updated | | `manifestBackupPath` | Path to `manifest.json.migrate.bak` when applicable | Use `--summary-json-path` to persist the same object for automation. ## Troubleshooting | Symptom | Likely cause | Action | |---|---|---| | `Missing required option --…` | Incomplete argv | Pass all required flags | | `Provide either --tcvdb-api-key or --tcvdb-api-key-env, not both` | Dual key sources | Use one | | `Environment variable … is empty or not set` | Bad env name / empty value | Export the key | | `Unsupported layer(s): …` | Typo in `--layers` | Use only `l0,l1,l2,l3` | | `Target store is not empty` | Prior data in DB | Use empty DB or `--no-fail-if-target-nonempty` | | Count verification failed | Incomplete write / timing / partial layers | Re-check target; for partial layer runs disable verify or migrate all layers | | `Target store entered degraded mode` | Bad URL/credentials/network/CA | Fix TCVDB connectivity; try `--tcvdb-ca-pem` for HTTPS | | `Failed to open sqlite store` | Corrupt or locked `vectors.db` | Validate path; stop writers | | Config not switched after migrate | `--no-apply-config` or wrong `--plugin-id` | Confirm `plugins.entries.memory-tencentdb.config.storeBackend` | | Runtime still on SQLite | Gateway not restarted | Restart after config rewrite | This page covers **storage backend** migration (SQLite → TCVDB). Package rename migration (`@tdai/memory-tdai` → `@tencentdb-agent-memory/memory-tencentdb`) is a separate workflow and is not performed by this CLI. ## Related pages Runtime `storeBackend: tcvdb` fields, BM25 language, embedding model, and CA PEM for live traffic. SQLite vs TCVDB factory selection, hybrid retrieval, and embedding roles. All package bins including `migrate-sqlite-to-tcvdb`, export, and inspect tools. Full schema for `storeBackend`, `tcvdb`, and `bm25` after config rewrite. `read-local-memory` / `export-tencent-vdb` for post-migrate validation.