# Search engines

> Enable and configure DuckDuckGo, Google CSE, Tavily, Traversaal, Perplexity, Sploitus, and Searxng; scraper URLs; proxy and timeout constraints that gate network search tools.

- Repository: vxcontrol/pentagi
- GitHub: https://github.com/vxcontrol/pentagi
- Human docs: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5
- Complete Markdown: https://grok-wiki.com/public/docs/vxcontrol-pentagi-eca0cf4de5f5/llms-full.txt

## Source Files

- `backend/pkg/tools/duckduckgo.go`
- `backend/pkg/tools/google.go`
- `backend/pkg/tools/tavily.go`
- `backend/pkg/tools/perplexity.go`
- `backend/pkg/tools/searxng.go`
- `backend/pkg/tools/sploitus.go`
- `.env.example`
- `backend/docs/config.md`

---

---
title: "Search engines"
description: "Enable and configure DuckDuckGo, Google CSE, Tavily, Traversaal, Perplexity, Sploitus, and Searxng; scraper URLs; proxy and timeout constraints that gate network search tools."
---

PentAGI exposes network search as agent tools under type `SearchNetworkToolType` in `backend/pkg/tools/registry.go`. Each engine is registered only when its `IsAvailable()` gate passes; agents never see a tool that is disabled or missing credentials. HTTP traffic for all engines goes through `system.GetHTTPClient`, which applies `PROXY_URL`, `HTTP_CLIENT_TIMEOUT`, and optional external TLS settings from `backend/pkg/config`.

## How search tools enter a flow

Primary agents usually call the high-level `search` agent tool (`SearchToolName`), which delegates to the **searcher** specialist. The searcher executor (`GetSearcherExecutor` in `backend/pkg/tools/tools.go`) attaches every available network engine plus optional memory/store tools.

When assistant mode runs with `UseAgents=false`, the same engines are registered directly on the assistant executor instead of behind `search`.

```text
Agent / Assistant
        │
        ├─ search (agent tool) ──► Searcher executor
        │                              ├─ google, duckduckgo, tavily, ...
        │                              ├─ browser (if scraper URL set)
        │                              └─ search_answer / store_answer (vector DB)
        │
        └─ (UseAgents=false) ──► engines registered on the executor itself
```

Unconfigured engines are omitted from the LLM tool list. Calling a registered engine that later fails typically returns a string error result (swallowed failure) so the agent can continue; unavailable tools return a hard error from `Handle`.

## Engine matrix

| Tool name | Registry name | Availability gate | Default on? | Auth / endpoint |
|---|---|---|---|---|
| DuckDuckGo | `duckduckgo` | `DUCKDUCKGO_ENABLED` | **Yes** (`true`) | HTML endpoint `https://html.duckduckgo.com/html/` (no API key) |
| Google CSE | `google` | `GOOGLE_API_KEY` **and** `GOOGLE_CX_KEY` non-empty | Off until keys set | Google Custom Search JSON API |
| Tavily | `tavily` | `TAVILY_API_KEY` non-empty | Off until key set | `https://api.tavily.com/search` |
| Traversaal | `traversaal` | `TRAVERSAAL_API_KEY` non-empty | Off until key set | `https://api-ares.traversaal.ai/live/predict` |
| Perplexity | `perplexity` | `PERPLEXITY_API_KEY` non-empty | Off until key set | `https://api.perplexity.ai/chat/completions` |
| Searxng | `searxng` | `SEARXNG_URL` non-empty | Off until URL set | Self-hosted base URL + `/search?format=json` |
| Sploitus | `sploitus` | `SPLOITUS_ENABLED` | **No** (`false`) | `https://sploitus.com/search` (no API key; Cloudflare-sensitive) |
| Browser | `browser` | `SCRAPER_PRIVATE_URL` or `SCRAPER_PUBLIC_URL` non-empty | Depends on compose/scraper | Internal scraper service |

<Note>
`backend/docs/config.md` documents `SPLOITUS_ENABLED` default as `true` in one table, but `config.go` sets `envDefault:"false"`. Runtime follows `config.go`.
</Note>

## Common tool arguments

Most engines use `SearchAction` (`backend/pkg/tools/args.go`):

| Field | Type | Required | Notes |
|---|---|---|---|
| `query` | string | yes | Always English for index quality; short, exact queries work best |
| `max_results` | integer | yes | Schema: min 1, max 10, default guidance 5; engines clamp independently |
| `message` | string | yes | Engagement-log commentary (may use engagement language) |

Sploitus uses `SploitusAction` instead:

| Field | Type | Required | Notes |
|---|---|---|---|
| `query` | string | yes | English-only index (CVE, product, class) |
| `exploit_type` | string | no | `exploits` (default) or `tools` |
| `sort` | string | no | `default`, `date`, or `score` |
| `max_results` | integer | yes | Clamped to 1–25; default 10 if out of range |
| `message` | string | yes | Engagement-log commentary |

Browser uses `Browser` with `url`, `action` (`markdown` \| `html` \| `links`), and `message`.

## Configure engines

Copy keys into `.env` from `.env.example`, then restart the stack so `pentagi` reloads env.

### DuckDuckGo

Free HTML search. Enabled by default.

<ParamField body="DUCKDUCKGO_ENABLED" type="bool" default="true">
When false, `duckduckgo` is not registered.
</ParamField>
<ParamField body="DUCKDUCKGO_REGION" type="string">
Region code such as `us-en` (code default when empty), `uk-en`, `de-de`, `fr-fr`, `jp-jp`, `cn-zh`, `ru-ru`.
</ParamField>
<ParamField body="DUCKDUCKGO_SAFESEARCH" type="string">
`strict` → `kp=1`, `moderate` → `kp=0`, `off` → `kp=-1`; empty leaves `kp` unset.
</ParamField>
<ParamField body="DUCKDUCKGO_TIME_RANGE" type="string">
`d` (day), `w` (week), `m` (month), `y` (year).
</ParamField>

Implementation notes:

- POST form to DuckDuckGo HTML search; parses result list.
- Max results capped at 10; invalid `max_results` resets to 10.
- Client timeout **30s**; up to **3** retries with 1s backoff.
- Empty result set returns the string `No results found`.

### Google Custom Search

Requires a [Programmable Search Engine](https://programmablesearchengine.google.com/) ID plus API key.

<ParamField body="GOOGLE_API_KEY" type="string" required>
API key for Custom Search.
</ParamField>
<ParamField body="GOOGLE_CX_KEY" type="string" required>
Custom Search Engine ID (`cx`). Both key and `cx` must be set for `IsAvailable()`.
</ParamField>
<ParamField body="GOOGLE_LR_KEY" type="string" default="lang_en">
Language restriction passed as `lr` (for example `lang_en`).
</ParamField>

Results format as markdown (`# title`, `## URL`, `## Snippet`). Max results clamped to 1–10.

### Tavily

AI search with answer + ranked links + optional raw page content.

<ParamField body="TAVILY_API_KEY" type="string" required>
Bearer key for Tavily API.
</ParamField>

Fixed request shape in code: `topic=general`, `search_depth=advanced`, `include_answer=true`, `include_raw_content=true`. Raw content over ~3000 chars can be summarized via the flow summarizer when attached.

HTTP status mapping returns agent-readable strings (wrong API key → `API key is wrong`, rate limits, maintenance, etc.).

### Traversaal

Live predict API returning a single answer plus URL list.

<ParamField body="TRAVERSAAL_API_KEY" type="string" required>
Sent as header `x-api-key`.
</ParamField>

Output: `# Answer` block and numbered `# Links`. Non-200 responses become `unexpected status code: N`.

### Perplexity

Chat-completions style web research (`sonar` by default).

<ParamField body="PERPLEXITY_API_KEY" type="string" required>
Bearer token.
</ParamField>
<ParamField body="PERPLEXITY_MODEL" type="string" default="sonar">
Model name sent in the completion request.
</ParamField>
<ParamField body="PERPLEXITY_CONTEXT_SIZE" type="string" default="low">
`search_context_size`: `low`, `medium`, or `high`.
</ParamField>

Hardcoded request knobs: temperature `0.5`, top_p `0.9`, max_tokens `4000`, timeout **60s**, non-streaming. Large answers may be summarized or truncated at ~3000 chars when no summarizer succeeds.

### Searxng

Self-hosted meta-search. No cloud key; availability is entirely URL-based.

<ParamField body="SEARXNG_URL" type="string" required>
Base URL of the instance. Code appends `/search` if the path does not already end with `/search`.
</ParamField>
<ParamField body="SEARXNG_CATEGORIES" type="string" default="general">
Passed as `categories` query param.
</ParamField>
<ParamField body="SEARXNG_LANGUAGE" type="string">
Language filter (for example `en`).
</ParamField>
<ParamField body="SEARXNG_SAFESEARCH" type="string" default="0">
`0` none, `1` moderate, `2` strict.
</ParamField>
<ParamField body="SEARXNG_TIME_RANGE" type="string">
Optional `time_range` (for example `day`, `month`, `year`).
</ParamField>
<ParamField body="SEARXNG_TIMEOUT" type="int">
Seconds; if unset or ≤0, default **30s**.
</ParamField>

Query always requests `format=json`. Empty hit lists return a markdown “No Results Found” block.

### Sploitus

Exploit/tool aggregator (ExploitDB, Packet Storm, GitHub advisories, and others). Disabled by default because the upstream sits behind Cloudflare; the config comment notes egress IP reputation matters.

<ParamField body="SPLOITUS_ENABLED" type="bool" default="false">
Set `true` to register `sploitus`.
</ParamField>

Behavior:

- POST JSON to `https://sploitus.com/search` with browser-like headers.
- Request timeout **30s**.
- HTTP **499** or **422** treated as rate limit.
- Output size hard limits: ~50 KB per source field, ~80 KB total markdown (truncation message when exceeded).

### Example `.env` fragments

```bash
# Zero-config path: DuckDuckGo only (enabled by default)
DUCKDUCKGO_ENABLED=true
DUCKDUCKGO_REGION=us-en

# Google CSE
GOOGLE_API_KEY=AIza...
GOOGLE_CX_KEY=your_cse_id
GOOGLE_LR_KEY=lang_en

# Cloud AI search APIs
TAVILY_API_KEY=tvly-...
TRAVERSAAL_API_KEY=...
PERPLEXITY_API_KEY=pplx-...
PERPLEXITY_MODEL=sonar
PERPLEXITY_CONTEXT_SIZE=low

# Self-hosted meta-search
SEARXNG_URL=http://searxng:8080
SEARXNG_CATEGORIES=general
SEARXNG_SAFESEARCH=0
SEARXNG_TIMEOUT=30

# Exploit search (opt-in)
SPLOITUS_ENABLED=true
```

## Scraper URLs and the browser tool

The `browser` tool is not a search engine API; it is the network tool for opening URLs and returning markdown, HTML, or link lists via the `vxcontrol/scraper` service in `docker-compose.yml`.

<ParamField body="SCRAPER_PRIVATE_URL" type="string">
Internal backend → scraper URL (compose example: `https://someuser:somepass@scraper/`).
</ParamField>
<ParamField body="SCRAPER_PUBLIC_URL" type="string">
URL usable for client-facing screenshot/content access when needed.
</ParamField>
<ParamField body="LOCAL_SCRAPER_USERNAME" type="string" default="someuser">
Scraper basic-auth user for the compose service.
</ParamField>
<ParamField body="LOCAL_SCRAPER_PASSWORD" type="string" default="somepass">
Scraper basic-auth password.
</ParamField>
<ParamField body="LOCAL_SCRAPER_MAX_CONCURRENT_SESSIONS" type="int" default="10">
Mapped into the scraper container as `MAX_CONCURRENT_SESSIONS`.
</ParamField>

`browser.IsAvailable()` is true if **either** private or public scraper URL is set. Empty both → tool never registered.

Compose service (defaults):

- Image: `vxcontrol/scraper:latest`
- Host port: `SCRAPER_LISTEN_IP` / `SCRAPER_LISTEN_PORT` (default `127.0.0.1:9443` → container 443)
- Podman rootless note in `.env.example`: use `http://user:pass@scraper:3000/` for private URL

Binary/non-HTML URLs (`.pdf`, archives, media, etc.) short-circuit with a descriptive hint rather than a generic “content too small” failure.

## Proxy, TLS, and timeouts

All engines build clients with `system.GetHTTPClient(cfg)`:

| Setting | Env | Default | Effect |
|---|---|---|---|
| Proxy | `PROXY_URL` | empty | When set, all search HTTP calls use this HTTP(S) proxy |
| Global timeout | `HTTP_CLIENT_TIMEOUT` | `600` seconds | Client timeout in seconds; `0` means no timeout (not recommended) |
| External CA | `EXTERNAL_SSL_CA_PATH` | empty | PEM appended to system trust pool |
| Skip verify | `EXTERNAL_SSL_INSECURE` | `false` | `InsecureSkipVerify` on the TLS config |

Per-tool overrides after client creation:

| Tool | Effective timeout |
|---|---|
| DuckDuckGo | 30s |
| Perplexity | 60s |
| Sploitus | 30s |
| Searxng | `SEARXNG_TIMEOUT` seconds, else 30s |
| Google / Tavily / Traversaal | inherit `HTTP_CLIENT_TIMEOUT` |

Proxy and TLS apply to search APIs and other external HTTP from the backend; Docker sandbox terminals use separate networking (see Docker sandbox docs).

## Logging and observability

Successful and failed searches (when agent context is present) call `SearchLogProvider.PutLog` with:

- Parent/current agent types
- `database.SearchengineType*` enum (`google`, `duckduckgo`, `tavily`, `traversaal`, `perplexity`, `searxng`, `sploitus`, …)
- Query and result text
- Optional task/subtask IDs

Search failures that are swallowed also emit Langfuse warning events named `search engine error swallowed` (or `sploitus search error swallowed`) with engine metadata.

## Failure modes

| Symptom | Likely cause | Fix |
|---|---|---|
| Agent never calls `google` / `tavily` / … | Tool not in definitions | Set required env keys / enable flags and restart |
| Only DuckDuckGo works | Expected with defaults | Other engines need keys or `SEARXNG_URL` / `SPLOITUS_ENABLED` |
| `browser` missing | No scraper URL | Set `SCRAPER_PRIVATE_URL` (and credentials matching the scraper service) |
| Timeouts to cloud APIs | Proxy/firewall or low timeout | Set `PROXY_URL` if required; raise `HTTP_CLIENT_TIMEOUT` carefully |
| Sploitus HTTP 499/422 | Rate limit / Cloudflare | Wait/retry; improve egress IP reputation; keep disabled if unreliable |
| Google 4xx | Bad key or CSE ID | Verify `GOOGLE_API_KEY` + `GOOGLE_CX_KEY` and CSE permissions |
| Tavily/Perplexity “API key is wrong” | Invalid key | Rotate key in `.env` |
| Searxng unexpected status / decode errors | Instance not JSON-enabled or wrong URL | Ensure instance allows `format=json` and path resolves to `/search` |
| Empty DuckDuckGo results | Upstream HTML change or blocking | Retries already applied; try another engine or proxy |

## Verify configuration

<Steps>
  <Step title="Set env and restart">
    Edit `.env`, then `docker compose up -d` (or restart the `pentagi` service) so config reloads.
  </Step>
  <Step title="Confirm minimum path">
    Leave `DUCKDUCKGO_ENABLED` unset or `true`. Create a flow that needs web research; tool logs should show `duckduckgo` when the searcher runs.
  </Step>
  <Step title="Add at least one richer engine">
    Set `TAVILY_API_KEY` or `PERPLEXITY_API_KEY` for answer-style results, or `SEARXNG_URL` for self-hosted aggregation.
  </Step>
  <Step title="Optional exploit research">
    Set `SPLOITUS_ENABLED=true` only if egress can reach Sploitus reliably.
  </Step>
  <Step title="Optional page fetch">
    Ensure compose `scraper` is up and `SCRAPER_PRIVATE_URL` matches service auth; agents should then receive `browser`.
  </Step>
</Steps>

Search engines are BYOK/BYOC: you supply API keys or self-host Searxng; PentAGI does not require a specific commercial search vendor to run (DuckDuckGo ships enabled by default).

## Related pages

<CardGroup>
  <Card title="Environment variables" href="/environment-variables">
    Full Config struct and `.env.example` reference, including search and proxy keys.
  </Card>
  <Card title="Tools and sandbox execution" href="/tools-and-sandbox">
    Tool categories, timeouts, and how network tools relate to Docker-isolated execution.
  </Card>
  <Card title="Tools reference" href="/tools-reference">
    Named registry entries, argument shapes, and agent/barrier tools.
  </Card>
  <Card title="Agents and supervision" href="/agents-and-supervision">
    Searcher specialist, limited-agent tool-call budgets, and supervision limits.
  </Card>
  <Card title="Installation" href="/installation">
    Compose stack including the scraper service and `.env` bootstrap.
  </Card>
  <Card title="Memory and knowledge" href="/memory-and-knowledge">
    Vector memory and guide/answer stores used alongside network search.
  </Card>
</CardGroup>