# Memory and knowledge

> pgvector memory tools, embeddings config, chain summarizer budgets, user knowledge documents API, flow files and resources under /work, and optional Graphiti graph search.

- Repository: vxcontrol/pentagi
- GitHub: https://github.com/vxcontrol/pentagi
- Human docs: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5
- Complete Markdown: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5/llms-full.txt

## Source Files

- `backend/pkg/tools/memory.go`
- `backend/pkg/providers/embeddings/embedder.go`
- `backend/pkg/csum/chain_summary.go`
- `backend/pkg/server/services/knowledge.go`
- `backend/pkg/graphiti/client.go`
- `backend/pkg/resources/resources.go`
- `backend/pkg/flowfiles/files.go`

---

---
title: "Memory and knowledge"
description: "pgvector memory tools, embeddings config, chain summarizer budgets, user knowledge documents API, flow files and resources under /work, and optional Graphiti graph search."
---

PentAGI persists engagement knowledge in PostgreSQL with **pgvector** (collection `langchain`), keeps agent message chains within byte budgets via the **chain summarizer** (`csum`), exposes curated **answer / guide / code** documents through REST and GraphQL, mounts flow **uploads** and **resources** under the sandbox path `/work`, and optionally records episodic memory in **Graphiti** for `graphiti_search`. Embeddings and Graphiti are provider-neutral BYOK/BYOC surfaces: configure any supported embedding backend and your own Graphiti/Neo4j stack without locking to a single cloud vendor.

## Architecture

```mermaid
flowchart TB
  subgraph agents [Agent runtime]
    Tools["Tool handlers<br/>search_in_memory / store_* / graphiti_search"]
    Memorist["memorist agent"]
    Executor["customExecutor.storeToolResult"]
    Csum["csum.SummarizeChain"]
  end

  subgraph api [API /api/v1]
    REST["REST KnowledgeService"]
    GQL["GraphQL knowledge*"]
    Files["Flow files + resources APIs"]
  end

  subgraph storage [Persistence]
    PG[("PostgreSQL + pgvector<br/>langchain_pg_embedding")]
    DataDir["DATA_DIR<br/>flow-N / flow-N-data<br/>resources/*.blob"]
    Neo[("Optional Neo4j via Graphiti")]
  end

  subgraph embed [Embeddings]
    Emb["embeddings.New<br/>EMBEDDING_PROVIDER"]
  end

  Tools --> PG
  Memorist --> Tools
  Executor --> PG
  REST --> PG
  GQL --> PG
  Emb --> PG
  Tools --> Neo
  Files --> DataDir
  DataDir -->|"CopyToContainer /work"| Sandbox["Docker sandbox WorkingDir=/work"]
  Csum -->|"LLM summarize handler"| Chain["Message chain in context"]
```

| Surface | Backing store | Agent tools | User/API surface |
|---|---|---|---|
| Ephemeral flow memory (`doc_type=memory`) | pgvector | `search_in_memory`; auto-store from selected tools | Vecstore logs only |
| Curated knowledge (`answer`, `guide`, `code`) | pgvector | `search_*` / `store_*` | REST `/api/v1/knowledge`, GraphQL `knowledge*` |
| Chain context compression | In-memory chain rewrite | Implicit during agent turns | `SUMMARIZER_*` / `ASSISTANT_SUMMARIZER_*` env |
| Flow files | Host `flow-{id}-data/` → container `/work` | `file`, `terminal` | Flow file APIs; prompt `<task_files>` listing |
| Global resources | Host `resources/{md5}.blob` | Copied into flow `/work/resources` | Resource upload/list APIs |
| Temporal knowledge graph | Graphiti + Neo4j | `graphiti_search` | Compose overlay + `GRAPHITI_*` |

## Embeddings

Vector store and knowledge CRUD require a configured embedder. `embeddings.New` selects a constructor from `EMBEDDING_PROVIDER` and wraps it with strip-newlines and batch options.

| Provider value | Default model (when `EMBEDDING_MODEL` empty) | Notes |
|---|---|---|
| `openai` (default) | `text-embedding-ada-002` | Falls back to `OPENAI_KEY` / `OPENAI_SERVER_URL` if embedding-specific key/URL unset |
| `ollama` | (model required via env) | Uses `EMBEDDING_URL`; key not used |
| `mistral` | `mistral-embed` | Model override not supported in client |
| `jina` | `jina-embeddings-v2-small-en` | |
| `huggingface` | `BAAI/bge-small-en-v1.5` | |
| `googleai` | `embedding-001` | `EMBEDDING_URL` not used |
| `voyageai` | `voyage-4` | `EMBEDDING_URL` not used |
| `none` | — | Embedder unavailable; create/search return errors / HTTP 503 |

<ParamField body="EMBEDDING_PROVIDER" type="string" default="openai">
Backend selector: `openai`, `ollama`, `mistral`, `jina`, `huggingface`, `googleai`, `voyageai`, or `none`.
</ParamField>

<ParamField body="EMBEDDING_URL" type="string">
Optional base URL override for providers that support custom endpoints.
</ParamField>

<ParamField body="EMBEDDING_KEY" type="string">
API key for the embedding backend (OpenAI may use `OPENAI_KEY` instead).
</ParamField>

<ParamField body="EMBEDDING_MODEL" type="string">
Model name; provider-specific defaults apply when empty.
</ParamField>

<ParamField body="EMBEDDING_BATCH_SIZE" type="int" default="512">
Batch size passed to the embedder wrapper.
</ParamField>

<ParamField body="EMBEDDING_STRIP_NEW_LINES" type="bool" default="true">
Strip newlines before embedding.
</ParamField>

<ParamField body="EMBEDDING_MAX_TEXT_BYTES" type="int" default="8192">
Max bytes sent to the embedding model on knowledge create/update; full document text is still stored in the DB.
</ParamField>

pgvector store construction uses collection name `langchain` and either the shared `PgxPool` or `DATABASE_URL`.

<Warning>
Without a working embedder, list/get/delete of knowledge documents can still work from SQL, but create, update (re-embed), and semantic search return “embedding provider is not configured/available” (REST maps this to **503**). Agent vector tools that need the store report as unavailable or fail at handle time.
</Warning>

## pgvector memory tools

### Automatic memory write

After successful tool execution, `customExecutor.storeToolResult` may index results into pgvector when the tool is in `allowedStoringInMemoryTools`:

`terminal`, `file`, `search`, `google`, `duckduckgo`, `tavily`, `traversaal`, `perplexity`, `searxng`, `sploitus`, `maintenance`, `coder`, `pentester`, `advice`

Write path:

1. Format args (JSON) + result as markdown.
2. Split with recursive character splitter (chunk size **2000**, overlap **100**, code blocks and heading hierarchy on).
3. Attach metadata: `user_id`, `flow_id`, optional `task_id` / `subtask_id`, `tool_name`, `tool_description`, `doc_type=memory`, `part_size`, `total_size`.
4. `store.AddDocuments`.

### `search_in_memory`

| Constant | Value |
|---|---|
| Score threshold | `0.2` |
| Result limit (post-merge) | `3` |
| Filter `doc_type` | `memory` |
| Always filter | `flow_id` (current flow) |
| Optional hard filters | `task_id`, `subtask_id` |

**Arguments (`SearchInMemoryAction`):**

| Field | Required | Constraints |
|---|---|---|
| `questions` | yes | 1–5 English semantic queries |
| `task_id` | no | Integer hard filter |
| `subtask_id` | no | Integer hard filter |
| `message` | yes | Engagement-log commentary |

Behavior:

- Runs similarity search per question; continues if one query fails.
- If task/subtask filters yield zero hits, **falls back** to flow-level filters (drops task/subtask).
- Merges, deduplicates, sorts by score, caps at 3 documents.
- Empty result returns: `nothing found in memory store by this question` (not an error).
- Response markdown includes match score, task/subtask IDs, tool name/description, and content.

### Curated knowledge tools (guide / answer / code)

Same similarity constants (**threshold 0.2**, **limit 3**, multi-query merge). Document types and hard filters:

| Tool | Direction | `doc_type` | Required type filter |
|---|---|---|---|
| `search_guide` / `store_guide` | R/W | `guide` | `type`: `install`, `configure`, `use`, `pentest`, `development`, `other` |
| `search_answer` / `store_answer` | R/W | `answer` | `type`: `guide`, `vulnerability`, `code`, `tool`, `other` |
| `search_code` / `store_code` | R/W | `code` | `lang` (markdown language name) |

Store tools expect English content for the shared English index, require co-indexed `question` text, and instruct agents to anonymize IPs, domains, credentials, and paths. Empty guide search returns a message telling the agent to store a guide after solving the case.

### Memorist

`memorist` is an **agent** tool that delegates complex long-term retrieval; with assistant agents disabled, prompts expose direct `search_in_memory` / `search_guide` / `search_answer` / `search_code` instead.

<Info>
Technical-channel fields (`questions`, `question`, store payloads) are specified as **English-only** so multi-engagement retrieval stays aligned with the English-indexed store. Engagement-log `message` fields use the engagement language.
</Info>

## Chain summarizer budgets

`csum.SummarizeChain` rewrites `[]llms.MessageContent` via `ChainAST` before context overflow. Strategies run in order:

1. **Section collapse** — all sections except the last `KeepQASections` become a single completion/summarization body pair.
2. **Last-section preserve** (if `PreserveLast`) — keep recent sections under `LastSecBytes`, with max body-pair size `MaxBPBytes` and a **25%** reserve for future messages.
3. **QA rotation** (if `UseQA`) — cap older QA sections by count (`MaxQASections`) and total bytes (`MaxQABytes`); optional human-message summarization in QA pairs.

Summarized content is marked with prefix `**summarized content:**\n`.

### Flow summarizer (`SUMMARIZER_*`)

| Env | Config field | Default |
|---|---|---|
| `SUMMARIZER_PRESERVE_LAST` | `PreserveLast` | `true` |
| `SUMMARIZER_USE_QA` | `UseQA` | `true` |
| `SUMMARIZER_SUM_MSG_HUMAN_IN_QA` | `SummHumanInQA` | `false` |
| `SUMMARIZER_LAST_SEC_BYTES` | `LastSecBytes` | `51200` (50 KiB) |
| `SUMMARIZER_MAX_BP_BYTES` | `MaxBPBytes` | `16384` (16 KiB) |
| `SUMMARIZER_MAX_QA_SECTIONS` | `MaxQASections` | `10` |
| `SUMMARIZER_MAX_QA_BYTES` | `MaxQABytes` | `65536` (64 KiB) |
| `SUMMARIZER_KEEP_QA_SECTIONS` | `KeepQASections` | `1` |

### Assistant summarizer (`ASSISTANT_SUMMARIZER_*`)

| Env | Default |
|---|---|
| `ASSISTANT_SUMMARIZER_PRESERVE_LAST` | `true` |
| `ASSISTANT_SUMMARIZER_LAST_SEC_BYTES` | `76800` |
| `ASSISTANT_SUMMARIZER_MAX_BP_BYTES` | `16384` |
| `ASSISTANT_SUMMARIZER_MAX_QA_SECTIONS` | `7` |
| `ASSISTANT_SUMMARIZER_MAX_QA_BYTES` | `76800` |
| `ASSISTANT_SUMMARIZER_KEEP_QA_SECTIONS` | `3` |

Built-in algorithm constants match the flow defaults when config values are non-positive (e.g. zero bytes fall back to 50 KiB last section / 16 KiB body pair / 10 QA sections / 64 KiB QA budget / keep 1 section).

## User knowledge documents API

REST and GraphQL share `database/knowledge.KnowledgeStore` over `langchain_pg_embedding` (collection `langchain`). List endpoints **exclude** `doc_type=memory` so agent auto-memory chunks do not appear as user knowledge docs. Manual documents set cmetadata `manual=true`.

### Permissions

| Permission | Effect |
|---|---|
| `knowledge.admin` | Read/write any document (no `user_id` filter) |
| `knowledge.view` | List/get own documents |
| `knowledge.create` | Create |
| `knowledge.edit` | Update own (admin: any) |
| `knowledge.delete` | Delete own (admin: any) |
| `knowledge.search` | Semantic search (own docs unless admin) |

### Document types

| `doc_type` | Optional subtype fields |
|---|---|
| `answer` | `answer_type`: `guide`, `vulnerability`, `code`, `tool`, `other` |
| `guide` | `guide_type`: `install`, `configure`, `use`, `pentest`, `development`, `other` |
| `code` | `code_lang` |

### REST (`/api/v1/knowledge`)

| Method | Path | Purpose |
|---|---|---|
| `GET` | `/knowledge/` | Paginated list (`rdb.TableQuery` + `with_content`) |
| `GET` | `/knowledge/{id}` | Get by UUID |
| `POST` | `/knowledge/` | Create + embed |
| `POST` | `/knowledge/search` | Semantic search |
| `PUT` | `/knowledge/{id}` | Update + re-embed (`content` required) |
| `DELETE` | `/knowledge/{id}` | Delete |

**List filters:** `id`, `doc_type`, `question`, `description`, `guide_type`, `answer_type`, `code_lang`, `manual`, `user_id`, `flow_id`, `task_id`, `subtask_id`, `data` (concat of doc_type + question).

**Create body limits:** `content` 1–65536 chars; `question` 1–2048; `description` ≤1000; `code_lang` ≤100.

**Search body:** `query` required (1–2048); optional `limit` 1–100 (store default **10** when unset at store layer); optional filters for doc types, guide/answer types, code langs, `flow_id`, `manual`. Search threshold **0.2**.

:::endpoint POST /api/v1/knowledge/ Create knowledge document
Requires `knowledge.create`. Computes embedding; stores full content even when embedding input is truncated to `EMBEDDING_MAX_TEXT_BYTES`.

**Body:** `doc_type`, `content`, `question`, optional `description`, `guide_type`, `answer_type`, `code_lang`.

**Responses:** `201` document entry; `400` validation; `403` permission; `503` embedder unavailable.
:::

:::endpoint POST /api/v1/knowledge/search Semantic search
Requires `knowledge.search`. Admin searches all docs; users search own.

**Body:** `query`, optional `limit`, `doc_types`, `guide_types`, `answer_types`, `code_langs`, `flow_id`, `manual`.

**Responses:** `200` scored documents; `400` invalid; `403` permission; `503` store/embedder unavailable.
:::

### GraphQL

| Operation | Permission |
|---|---|
| `knowledgeDocuments(filter, withContent)` | `knowledge.view` |
| `knowledgeDocument(id)` | `knowledge.view` |
| `searchKnowledge(query, filter, limit)` | `knowledge.search` |
| `createKnowledgeDocument(input)` | `knowledge.create` |
| `updateKnowledgeDocument(id, input)` | `knowledge.edit` |
| `deleteKnowledgeDocument(id)` | `knowledge.delete` |
| Subscriptions `knowledgeDocumentCreated/Updated/Deleted` | Scoped by user; admin variants without filter |

`CreateKnowledgeDocumentInput`: `docType`, `content`, `question`, optional `description`, `guideType`, `answerType`, `codeLang`.

## Flow files and resources under `/work`

### Container mount

Docker sandbox containers set `WorkingDir` to `/work` and bind-mount host flow workdir:

- Host: `{DATA_DIR}/flow-{id}` (or a named volume when host workdir is empty)
- Container: `/work`

### Host-side flow file cache

API listing and transfer use a separate cache tree:

```text
{DATA_DIR}/
├── flow-{id}/                 # Docker bind for /work
├── flow-{id}-data/
│   ├── uploads/               # User uploads → /work/uploads
│   ├── container/             # Cached container pulls
│   └── resources/             # Materialized resource blobs → /work/resources
└── resources/
    └── {md5}.blob             # Global content-addressed store
```

| Constant | Value |
|---|---|
| Max single upload file | 300 MB |
| Max files per request | 1000 |
| Max total upload size | 2 GB |
| Max path length (resources) | 4096 |
| Max file name length | 255 |

Paths are sanitized: relative only, no `..`, no control chars or reserved filename characters. Resolve rules for flow file paths require a prefix of `uploads`, `container`, or `resources`.

### Prompt injection

`FileListingForPrompt` builds a compact XML block for agent system prompts when uploads or resources exist:

```xml
<task_files>
<uploads base="/work/uploads">
wordlist.txt
</uploads>
<resources base="/work/resources">
creds/passwords.txt
</resources>
</task_files>
```

Resource blobs are copied into `flow-{id}-data/resources/` then tar-copied into the container at `/work` so agents see them as normal files under `/work/resources/...`.

## Optional Graphiti graph search

Graphiti is **off by default**. When enabled, the client health-checks the Graphiti URL at construction; failures prevent enablement.

| Env | Default | Role |
|---|---|---|
| `GRAPHITI_ENABLED` | `false` | Master switch |
| `GRAPHITI_URL` | empty | Graphiti HTTP base (e.g. `http://graphiti:8000`) |
| `GRAPHITI_TIMEOUT` | `30` | Seconds |
| `GRAPHITI_MODEL_NAME` | (compose/stack) | Model used by Graphiti service side |

Compose overlay: `docker-compose-graphiti.yml` (see knowledge-graph page for Neo4j wiring).

During agent performance, successful agent responses and tool executions can be **written** to Graphiti (group id `flow-{id}`) via templates. Reads go through `graphiti_search`.

### `graphiti_search`

If Graphiti is disabled, the tool returns a soft message (not a hard error): *Graphiti knowledge graph is not enabled...*

| `search_type` | Default max results | Notable params |
|---|---|---|
| `temporal_window` | 15 | `time_start`, `time_end` (ISO 8601) |
| `entity_relationships` | 20 | `center_node_uuid`, `max_depth` (default 2, max 3) |
| `diverse_results` | 10 | `diversity_level`: `low`/`medium`/`high` |
| `episode_context` | 10 | Full agent reasoning / tool outputs |
| `successful_tools` | 15 | `min_mentions` (default 2) |
| `recent_context` | 10 | `recency_window`: `1h`/`6h`/`24h`/`7d` (default `24h`) |
| `entity_by_label` | 25 | `node_labels`, optional `edge_types` |

Always required: English `query`, `search_type`, engagement `message`.

## Operations checklist

<Steps>
  <Step title="Configure embeddings">
    Set `EMBEDDING_PROVIDER` and credentials (`EMBEDDING_KEY` / provider-specific fallbacks). Leave unset or use `none` only if you do not need vector memory or knowledge search.
  </Step>
  <Step title="Verify PostgreSQL + pgvector">
    Core compose must provide PostgreSQL with pgvector. The app uses collection `langchain` in `langchain_pg_embedding`.
  </Step>
  <Step title="Tune summarizer budgets">
    Adjust `SUMMARIZER_*` for autonomous flows and `ASSISTANT_SUMMARIZER_*` for assistant mode if models hit context limits or over-summarize.
  </Step>
  <Step title="Optional Graphiti">
    Deploy with the Graphiti compose overlay, set `GRAPHITI_ENABLED=true` and `GRAPHITI_URL`, then confirm health at startup and that `graphiti_search` no longer returns the disabled message.
  </Step>
  <Step title="Seed knowledge">
    Create documents via UI, REST, or GraphQL (`knowledge.create`). Agents also learn guides/answers/code via store tools during flows.
  </Step>
</Steps>

## Failure modes

| Symptom | Likely cause |
|---|---|
| REST/GraphQL create or search → 503 / “embedding provider is not …” | Embedder not configured, `none`, or init failure |
| `search_in_memory` unavailable / errors on store | pgvector store nil (embedder or DB connection) |
| Empty memory hits within a flow | Threshold 0.2, English query quality, or nothing stored yet for that `flow_id` |
| Knowledge list missing agent tool dumps | Expected: `doc_type=memory` excluded from knowledge list |
| `graphiti_search` “not enabled” string | `GRAPHITI_ENABLED=false`, URL unset, or health check failed at client create |
| Graphiti client construction error at boot | Backend unreachable when `GRAPHITI_ENABLED=true` |
| Resource path rejected | Absolute path, `..`, invalid characters, or length limits |
| Embed token errors on long docs | Truncation uses `EMBEDDING_MAX_TEXT_BYTES` for the embedding call only |

## Related pages

<CardGroup cols={2}>
  <Card title="Knowledge graph" href="/knowledge-graph">
    Graphiti + Neo4j compose overlay, GRAPHITI_* settings, and graph tool failure modes.
  </Card>
  <Card title="Environment variables" href="/environment-variables">
    Full Config struct: embeddings, summarizer, Graphiti, and database pool keys.
  </Card>
  <Card title="Tools reference" href="/tools-reference">
    Argument shapes for memory, guide/answer/code, and graphiti_search tools.
  </Card>
  <Card title="REST API" href="/rest-api">
    Gin routes under /api/v1 including knowledge, files, and resources.
  </Card>
  <Card title="GraphQL API" href="/graphql-api">
    knowledgeDocuments, searchKnowledge, mutations, and subscriptions.
  </Card>
  <Card title="Tools and sandbox execution" href="/tools-and-sandbox">
    Docker isolation, file tools, and how agents access /work.
  </Card>
  <Card title="Agents and supervision" href="/agents-and-supervision">
    Memorist specialist and tool-call limits around retrieval agents.
  </Card>
  <Card title="Development and testing" href="/development-and-testing">
    etester embeddings and local compose for contributors.
  </Card>
</CardGroup>