# Troubleshooting

> Source-backed failure modes: plugin disabled, no recall, embedding 400/matryoshka, retention cleanup, offload patch missing, Gateway circuit breaker, auth 401, and log/probe checklist.

- Repository: TencentCloud/TencentDB-Agent-Memory
- GitHub: https://github.com/TencentCloud/TencentDB-Agent-Memory
- Human docs: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-5a33bbf5540a
- Complete Markdown: https://grok-wiki.com/public/docs/tencentcloud-tencentdb-agent-memory-5a33bbf5540a/llms-full.txt

## Source Files

- `SKILL.md`
- `README.md`
- `hermes-plugin/memory/memory_tencentdb/README.md`
- `hermes-plugin/memory/memory_tencentdb/tests/test_memory_tencentdb_recovery.py`
- `scripts/bugfix-20260423/BUGFIX-20260423-SOP.md`
- `src/core/store/embedding.ts`
- `src/gateway/server.ts`
- `SKILL-DIAGNOSTIC-EXPORT.md`

---

---
title: "Troubleshooting"
description: "Source-backed failure modes: plugin disabled, no recall, embedding 400/matryoshka, retention cleanup, offload patch missing, Gateway circuit breaker, auth 401, and log/probe checklist."
---

Runtime failures in `@tencentdb-agent-memory/memory-tencentdb` cluster around host enablement (OpenClaw plugin or Hermes Gateway sidecar), embedding degradation, retention cleanup, context-offload patches, and Hermes client reliability (circuit breaker + Bearer auth). Log tags still use the historical prefix `[memory-tdai]`; the on-disk data directory remains `memory-tdai` under the OpenClaw state dir (or `~/.memory-tencentdb/memory-tdai` for standalone Hermes Gateway).

## Quick triage

| Symptom | First check | Likely fix |
| --- | --- | --- |
| No `[memory-tdai]` logs after restart | Plugin enable flag + Gateway restart | Set `memory-tencentdb.enabled: true`, `openclaw gateway restart` |
| Capture works, no injection | `recall.enabled`, `scoreThreshold`, timeout | Lower threshold; raise `recall.timeoutMs` (default 5000) |
| Vector search empty / keyword-only | `embedding.provider` + apiKey/baseUrl/model/dimensions | Complete remote embedding quadruple, or accept keyword-only |
| HTTP 400 on embed (matryoshka) | `embedding.sendDimensions` | Set `sendDimensions: false` for BGE-M3-style backends |
| History disappears overnight | `capture.l0l1RetentionDays`, `allowAggressiveCleanup` | Use `0` (never clean) or `>= 3`; avoid 1–2 without aggressive flag |
| Offload tools/results not recovered | `offload.enabled`, `plugins.slots.contextEngine`, after-tool-call patch | Register slot + re-run patch after OpenClaw upgrades |
| Hermes: “Gateway not available” / empty tools | Port 8420, auto-discovery, stderr log | Start Gateway, set `MEMORY_TENCENTDB_GATEWAY_CMD`, inspect logs |
| Hermes: “circuit breaker tripped” | 5 consecutive Gateway failures | Fix Gateway health; wait 60s or let watchdog recover |
| HTTP 401 on `/recall` (etc.) | Bearer key mismatch | Align `TDAI_GATEWAY_API_KEY` with client `MEMORY_TENCENTDB_GATEWAY_API_KEY` |

## Plugin disabled or not loading (OpenClaw)

### Signals

- No `[memory-tdai]` lines in Gateway logs after restart
- Missing data dir: `$OPENCLAW_STATE_DIR/memory-tdai/` (default `~/.openclaw/memory-tdai/`)
- Expected subdirs never appear: `conversations/`, `records/`, `scene_blocks/`, `vectors.db`

### Causes and fixes

1. **Plugin not enabled** — zero-config enable is:

```json
{
  "memory-tencentdb": {
    "enabled": true
  }
}
```

in `~/.openclaw/openclaw.json`. Config changes require `openclaw gateway restart`.

2. **Wrong upgrade path** — use `openclaw plugins update @tencentdb-agent-memory/memory-tencentdb` (native OpenClaw commands). Semantic version range installs can leave the plugin effectively disabled.

3. **Host version gate** — OpenClaw `>= 2026.3.13`, Node `>= 22.16.0`. Confirm with `openclaw --version` and `node -v`.

4. **OpenClaw 2026.4.23 hook schema** — Zod `.strict()` can reject `hooks.allowConversationAccess`, blocking non-bundled session hooks. Apply the version-scoped fix:

```bash
openclaw gateway stop
bash scripts/bugfix-20260423/bugfix-20260423.sh
# verify allowConversationAccess under plugins.entries.memory-tencentdb.hooks
openclaw gateway run
```

Later hosts (`>= 2026.4.24`) auto-handle the policy path in plugin registration.

### Verify

```bash
openclaw gateway restart
# logs should show [memory-tdai] Registering plugin ... and Config parsed: ...
ls -la "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/memory-tdai/"
```

## No recall / empty memory injection

Auto-recall runs in `performAutoRecall` with tag `[memory-tdai] [recall]`. It searches L1 (`keyword` / `embedding` / `hybrid`), injects L3 persona and L2 scene navigation, and races against `recall.timeoutMs` (default **5000** ms). On timeout it **skips injection** without blocking the user turn.

### Config knobs

| Key | Default | Failure mode if wrong |
| --- | --- | --- |
| `recall.enabled` | `true` | Auto-recall hooks not registered |
| `recall.scoreThreshold` | `0.3` | Hits filtered out as “below threshold” |
| `recall.strategy` | `hybrid` | `embedding`/`hybrid` fall back to keyword when embedding unavailable |
| `recall.timeoutMs` | `5000` | Slow embedding → timeout skip |
| `recall.maxResults` | `5` | Caps injected items |
| `capture.enabled` / `extraction.enabled` | `true` | No L0/L1 data to recall |

### Checklist

<Steps>
  <Step title="Confirm L0/L1 data exists">
    Inspect `memory-tdai/conversations/` and `records/`, or use `read-local-memory` / OpenClaw `memory-tdai` CLI. No shards means nothing to inject.
  </Step>
  <Step title="Check recall flags">
    Ensure `recall.enabled` is true and `scoreThreshold` is not overly high for sparse early data.
  </Step>
  <Step title="Watch strategy and embedding">
    Logs show `strategy=…`, `embeddingAvailable=…`, FTS/embedding hit counts. Missing embedding forces keyword-only; FTS unavailable yields empty keyword path.
  </Step>
  <Step title="Exercise tools">
    Call `tdai_memory_search` / `tdai_conversation_search` (OpenClaw) or Hermes `memory_tencentdb_*` tools. Combined tool budget is **3 calls per turn**.
  </Step>
</Steps>

<Note>
Default `embedding.provider` is `"none"`: vector search is off and hybrid degrades to keyword. Incomplete remote embedding config disables embedding with a stored `configError` and continues without vectors — the plugin does not throw.
</Note>

## Embedding HTTP 400 / Matryoshka `dimensions`

Remote OpenAI-compatible embedding posts to `{baseUrl}/embeddings`. By default `sendDimensions` is **true**, so the request body includes `dimensions` (Matryoshka-style truncation for models like `text-embedding-3-*`).

Some self-hosted / OSS models (e.g. **BGE-M3**) reject unknown `dimensions` with HTTP **400** (`does not support matryoshka representation`). Client errors in the 4xx range (except 429) are **not retried**.

### Fix

```json
{
  "embedding": {
    "enabled": true,
    "provider": "openai",
    "baseUrl": "http://your-host:port/v1",
    "apiKey": "<KEY>",
    "model": "bge-m3",
    "dimensions": 1024,
    "sendDimensions": false
  }
}
```

### Other embedding failure modes

| Condition | Behavior |
| --- | --- |
| `provider: "none"` (default) | Embedding disabled; keyword path only |
| Missing any of `apiKey` / `baseUrl` / `model` / `dimensions` | Embedding disabled; error message logged via `configError` |
| `provider: "local"` in user config | Treated as disabled at config parse (not exposed) |
| `provider: "qclaw"` without `proxyUrl` (+ full remote fields) | Embedding disabled |
| ZeroEntropy (`provider: "zeroentropy"`) | Uses `/models/embed`; Matryoshka dims must match accepted set when `sendDimensions` is true |
| Local model not ready | `EmbeddingNotReadyError`; callers fall back to keyword-only |

Timeouts default to **10000** ms per call with up to **3** retries on 5xx/429/network errors. Separate `recallTimeoutMs` / `captureTimeoutMs` can shorten or lengthen recall vs capture paths.

## Retention cleanup too aggressive

Daily cleaner (`LocalMemoryCleaner`, tag `[memory-tdai][cleaner]`) deletes aged L0 (`conversations/`) and L1 (`records/`) shards when cleanup is enabled.

| Setting | Rule |
| --- | --- |
| `capture.l0l1RetentionDays: 0` | Cleanup **disabled** (default) |
| `>= 3` | Retention honored |
| `1` or `2` | Honored only if `capture.allowAggressiveCleanup: true`; otherwise cleaned retention is ignored |
| `cleanTime` | Daily run time (default `03:00` local) |

Safety floors: skip deletion if total L0 records `<= 50` or L1 records `<= 20`. Cleanup uses **local calendar days**, not rolling 24h windows.

Offload artifacts use separate `offload.offloadRetentionDays` (reclaim scheduler only when `>= 3`).

## Context offload patch missing or slot not owned

Offload requires three independent conditions:

1. **`offload.enabled: true`** under the plugin config  
2. **`plugins.slots.contextEngine: "memory-tencentdb"`** so OpenClaw routes the context-engine slot here  
3. **Runtime patch** so `after-tool-call` hooks receive the full messages list:

```bash
bash scripts/openclaw-after-tool-call-messages.patch.sh
# or one-shot enable:
bash scripts/setup-offload.sh --enable --user-id <id> --backend-url <url> [--backend-api-key <key>]
```

The patch is **idempotent** (skips already-patched files; backups as `*.pre-offload-patch.bak`). Re-run after every OpenClaw upgrade.

### Log signals when offload is dead

| Log fragment | Meaning |
| --- | --- |
| `slots.contextEngine=… (expected "memory-tencentdb")` | Slot not assigned — **all** offload functions disabled |
| `registerContextEngine returned { ok: false, existingOwner: … }` | Another plugin owns the slot |
| `backendUrl not configured` | Backend mode cannot run L1/L1.5/L2/L4 |
| `No model resolved` / `LLM client not available` | Local offload LLM path disabled |

## Hermes Gateway not available

The Hermes provider is an HTTP client + supervisor for the Node Gateway (default `127.0.0.1:8420`).

### Startup paths

1. **Auto-discovery** of `src/gateway/server.ts` (in-tree → `~/.memory-tencentdb/tdai-memory-openclaw-plugin/` → legacy paths)  
2. **`MEMORY_TENCENTDB_GATEWAY_CMD`** explicit command (wins over discovery)  
3. **Pre-started** Gateway already healthy on `/health`

### Failures

- Directory name must be exactly `memory_tencentdb` (underscore) under Hermes plugins  
- Config aliases `memory-tencentdb` / `tdai` are valid **config** values, not directory names  
- Tools return empty schemas until Gateway is reachable **or** gateway env/port is set optimistically  
- Crash diagnostics: `~/.hermes/logs/memory_tencentdb/gateway.stderr.log` (override `MEMORY_TENCENTDB_LOG_DIR`)  
- LLM for L1/L2/L3 is Gateway-side: `MEMORY_TENCENTDB_LLM_API_KEY` (and optional base URL / model)

```bash
curl -s http://127.0.0.1:8420/health
# or: memory-tencentdb-ctl health / status / logs
```

## Circuit breaker and recovery (Hermes client)

Constants in the Hermes provider:

| Constant | Value | Role |
| --- | --- | --- |
| `_BREAKER_THRESHOLD` | **5** consecutive failures | Opens breaker |
| `_BREAKER_COOLDOWN_SECS` | **60** s | Pause API calls |
| `_RECOVER_COOLDOWN_SECS` | **15** s | Throttle request-path `ensure_running` |
| Watchdog interval | **10** s (production) | Resurrects dead Gateway even with no traffic |
| Capture back-pressure | max **4** in-flight `sync_turn` | 5th waits up to **5** s |

When open, tool calls may return:

```json
{"error": "memory-tencentdb Gateway temporarily unavailable (circuit breaker open)."}
```

Logs: `memory-tencentdb circuit breaker tripped after N failures. Pausing for 60s.`

Recovery paths:

- Request-path `_try_recover_gateway` after failures  
- Lazy probe `_ensure_alive_for_request` before short-circuit guards (when breaker closed)  
- Watchdog thread (resets breaker on successful revive)

Inspect Gateway health and LLM timeouts; hung L1 extractions also produce capture backlog warnings.

## Auth 401 on Gateway routes

Auth is **opt-in**. When `TDAI_GATEWAY_API_KEY` (or `server.apiKey` in `tdai-gateway.yaml` / JSON) is set:

| Route | Auth |
| --- | --- |
| `GET /health` | Always open |
| All other routes (`POST /recall`, `/capture`, `/search/*`, `/session/end`, `/seed`) | Require `Authorization: Bearer <key>` |

### Error messages

- `Unauthorized: missing Bearer token` — header absent or not `Bearer …`  
- `Unauthorized: invalid token` — constant-time compare failed  

### Client alignment (Hermes)

```bash
# Gateway process
export TDAI_GATEWAY_API_KEY="shared-secret"

# Hermes client (preferred name); falls back to TDAI_GATEWAY_API_KEY
export MEMORY_TENCENTDB_GATEWAY_API_KEY="shared-secret"
```

The Gateway **never** reads `MEMORY_TENCENTDB_GATEWAY_API_KEY`. Whitespace in env values is stripped client-side; mismatched secrets still 401.

When auth is unset, startup logs warn; binding non-loopback without a key triggers a louder warning. CORS `*` also warns at startup.

```bash
curl -s -H "Authorization: Bearer $TDAI_GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"test","session_key":"s1"}' \
  http://127.0.0.1:8420/recall
```

## Log and probe checklist

### OpenClaw

| Probe | Command / path |
| --- | --- |
| Host versions | `openclaw --version`, `node -v` |
| Plugin load | Gateway logs: `[memory-tdai]` |
| Config parse | `[memory-tdai] Config parsed: capture=… recall=…` |
| Recall path | `[memory-tdai] [recall]` — strategy, hits, timeout |
| Pipeline | `[pipeline]` / extraction scheduling |
| Cleaner | `[memory-tdai][cleaner]` |
| Offload | `[context-offload]` |
| Embedding | `[memory-tdai][embedding]` |
| Data dir | `${OPENCLAW_STATE_DIR:-~/.openclaw}/memory-tdai/` |
| Gateway logs | `~/.openclaw/logs/gateway.log`, `gateway.err.log` |
| Rolling logs | `/tmp/openclaw/openclaw-YYYY-MM-DD.log` |

### Hermes / standalone Gateway

| Probe | Path / command |
| --- | --- |
| Health | `GET http://127.0.0.1:8420/health` |
| Supervisor logs | `~/.hermes/logs/memory_tencentdb/gateway.stdout.log` / `.stderr.log` |
| Agent log | `~/.hermes/logs/agent.log` (auto-discovery lines) |
| Data dir | `TDAI_DATA_DIR` or `~/.memory-tencentdb/memory-tdai` |
| Ops CLI | `memory-tencentdb-ctl status|health|logs` |

### Diagnostic export package

```bash
bash scripts/export-diagnostic.sh
# optional: bash scripts/export-diagnostic.sh /tmp
```

Produces `~/Downloads/openclaw-diagnostic-<timestamp>.tar.gz` with redacted config, logs, and full `memory-tdai/` data. **Memory data contains raw conversations** — review before sharing. Export stays local; nothing is uploaded automatically.

Log search guide after export:

| Question | Search |
| --- | --- |
| Plugin loaded? | `[memory-tdai]` |
| Recall working? | `[recall]` |
| L1/L2/L3 scheduled? | `[pipeline]` |
| Embedding config | redacted config `plugins.entries` / embedding block |
| Checkpoint | `memory-tdai/.metadata/recall_checkpoint.json` |

## Smoke tests after a fix

<Tabs>
  <Tab title="OpenClaw">
```bash
openclaw gateway restart
# chat 2–3 turns with memorable facts, then start a new turn
# call tdai_memory_search / tdai_conversation_search
ls "${OPENCLAW_STATE_DIR:-$HOME/.openclaw}/memory-tdai/"{conversations,records,scene_blocks}
```
  </Tab>
  <Tab title="Hermes">
```bash
curl -s http://127.0.0.1:8420/health
# confirm memory.provider: memory_tencentdb in ~/.hermes/config.yaml
# watch agent.log for Gateway auto-discovery or breaker warnings
```
  </Tab>
</Tabs>

## Related pages

<CardGroup>
  <Card title="Configure OpenClaw" href="/configure-openclaw">
    Enable flag, capture/pipeline/recall groups, and post-restart verification.
  </Card>
  <Card title="Configure embedding" href="/configure-embedding">
    Remote providers, required fields, sendDimensions, and keyword-only degradation.
  </Card>
  <Card title="Enable context offload" href="/enable-context-offload">
    Slot registration, after-tool-call patch, and compression ratios.
  </Card>
  <Card title="Secure the Gateway" href="/secure-gateway">
    TDAI_GATEWAY_API_KEY, CORS, and Hermes client key alignment.
  </Card>
  <Card title="Gateway lifecycle" href="/gateway-ops">
    memory-tencentdb-ctl start/stop/status/health/logs.
  </Card>
  <Card title="Inspect local memory" href="/inspect-local-memory">
    read-local-memory, diagnostic export, and on-disk layout.
  </Card>
  <Card title="Gateway HTTP API" href="/gateway-http-api">
    Routes, auth exceptions, and error envelope.
  </Card>
  <Card title="Environment variables" href="/environment-variables">
    TDAI_* and MEMORY_TENCENTDB_* resolution order.
  </Card>
</CardGroup>