# Python API reference

> Exported SDK: `Template.create/get/list`, `BaseAutoType` lifecycle (`parse`, `feed_text`, `search`, `chat`, `dump`, `load`, `build_index`, `show`), `create_client` / `create_llm` / `create_embedder` / `get_client`, and logging helpers.

- Repository: yifanfeng97/Hyper-Extract
- GitHub: https://github.com/yifanfeng97/Hyper-Extract
- Human docs: https://grok-wiki.com/public/docs/yifanfeng97-hyper-extract-7891c7254cdf
- Complete Markdown: https://grok-wiki.com/public/docs/yifanfeng97-hyper-extract-7891c7254cdf/llms-full.txt

## Source Files

- `hyperextract/__init__.py`
- `hyperextract/utils/template_engine/template.py`
- `hyperextract/types/base.py`
- `hyperextract/utils/client.py`
- `hyperextract/utils/logging.py`
- `hyperextract/utils/template_engine/gallery.py`

---

---
title: "Python API reference"
description: "Exported SDK: `Template.create/get/list`, `BaseAutoType` lifecycle (`parse`, `feed_text`, `search`, `chat`, `dump`, `load`, `build_index`, `show`), `create_client` / `create_llm` / `create_embedder` / `get_client`, and logging helpers."
---

The `hyperextract` package exports a single SDK surface: eight `AutoType` primitives, the `Template` factory, LangChain client helpers, and structlog-based logging. Every extraction path returns a `BaseAutoType` subclass that owns LLM extraction, optional FAISS indexing, serialization to a Knowledge Abstract directory, and OntoSight visualization on concrete types.

```python
from hyperextract import (
    Template,
    BaseAutoType,
    AutoModel, AutoList, AutoSet,
    AutoGraph, AutoHypergraph,
    AutoTemporalGraph, AutoSpatialGraph, AutoSpatioTemporalGraph,
    create_client, create_llm, create_embedder, get_client,
    configure_logging, get_logger, set_log_level,
)
```

| Symbol | Role |
| --- | --- |
| `Template` | Resolve YAML presets or method templates into configured `AutoType` instances |
| `BaseAutoType` + eight `Auto*` classes | Knowledge Abstract runtime: extract, index, query, persist |
| `create_client` / `create_llm` / `create_embedder` / `get_client` | Build LangChain LLM and embedder clients from shorthand or `~/.he/config.toml` |
| `configure_logging` / `get_logger` / `set_log_level` | structlog setup; respects `HYPER_EXTRACT_LOG_LEVEL` and `HYPER_EXTRACT_LOG_FILE` |

## Template API

`Template` is a static façade over the gallery, method registry, and `TemplateFactory`. All three methods are stateless.

### `Template.create`

Returns a configured `BaseAutoType` instance.

<ParamField body="source" type="string" required>
Preset path (`general/biography_graph`), method path (`method/light_rag`), or absolute path to a `.yaml` file.
</ParamField>

<ParamField body="language" type="string">
Required for knowledge templates (`general/*`, `finance/*`, custom YAML). Ignored for `method/*` templates (always English).
</ParamField>

<ParamField body="llm_client" type="BaseChatModel">
LangChain chat model. When omitted, loaded via `get_client()` from `~/.he/config.toml`.
</ParamField>

<ParamField body="embedder" type="Embeddings">
LangChain embeddings model. When omitted, loaded via `get_client()`.
</ParamField>

<ParamField body="**kwargs" type="dict">
Overrides template `options` or method constructor args (e.g. `observation_time="2024-06-15"` for temporal methods).
</ParamField>

<ResponseField name="return" type="BaseAutoType">
Configured `AutoType` with `metadata` keys `template`, `lang`, and `type` populated by the factory.
</ResponseField>

<CodeGroup>

```python Knowledge template
from hyperextract import Template

ka = Template.create(
    "general/biography_graph",
    language="en",
)
with open("examples/en/tesla.md") as f:
    result = ka.parse(f.read())
result.show()
```

```python Method template
from hyperextract import Template, create_client

llm, emb = create_client("bailian", api_key="sk-...")
rag = Template.create(
    "method/light_rag",
    llm_client=llm,
    embedder=emb,
)
rag.feed_text(text)
```

```python Custom YAML
ka = Template.create(
    "/path/to/my_template.yaml",
    language="zh",
    llm_client=llm,
    embedder=emb,
)
```

</CodeGroup>

<Warning>
`Template.create` raises `ValueError` when `language` is omitted for knowledge templates. Method paths (`method/<name>`) do not require `language`.
</Warning>

### `Template.get`

```python
config = Template.get("general/biography_graph")  # TemplateCfg | None
method_cfg = Template.get("method/light_rag")       # method TemplateCfg
```

Returns a `TemplateCfg` Pydantic model (`language`, `name`, `type`, `tags`, `description`, `output`, `guideline`, `identifiers`, `options`, `display`) without instantiating an `AutoType`. Method paths delegate to `get_method_cfg`.

### `Template.list`

```python
all_templates = Template.list()
graphs = Template.list(filter_by_type="graph", filter_by_language="en")
biography = Template.list(filter_by_query="biography")
```

| Parameter | Default | Effect |
| --- | --- | --- |
| `filter_by_query` | `None` | Match against `name` or `description` |
| `filter_by_type` | `None` | Filter by autotype (`graph`, `list`, `model`, …) |
| `filter_by_tag` | `None` | Filter by YAML `tags` entry |
| `filter_by_language` | `None` | Filter by supported language code |
| `include_methods` | `True` | Merge registered extraction methods into the result |

Returns `Dict[str, TemplateCfg]` keyed by preset path (e.g. `general/biography_graph`) or method name.

## BaseAutoType lifecycle

`BaseAutoType[T]` is the abstract Knowledge Abstract runtime. Template-created instances and directly constructed `AutoGraph` / `AutoModel` / … objects share the same method surface.

### Construction

```python
AutoGraph(
    node_schema=Entity,
    edge_schema=Relation,
    llm_client=llm,
    embedder=emb,
    node_key_extractor=lambda n: n.name,
    edge_key_extractor=lambda e: (e.source, e.target, e.type),
    nodes_in_edge_extractor=lambda e: (e.source, e.target),
    chunk_size=2048,      # default
    chunk_overlap=256,    # default
    max_workers=10,       # parallel chunk extraction
    verbose=False,
)
```

Extraction uses `llm_client.with_structured_output(data_schema)` and `RecursiveCharacterTextSplitter` for long inputs.

### Extraction methods

| Method | Mutates instance | Returns | Use when |
| --- | --- | --- | --- |
| `parse(text)` | No | New `BaseAutoType` | Preview extraction or branch a KA without touching the original |
| `feed_text(text)` | Yes | `self` (chainable) | Incrementally grow the current KA (`feed_text(t1).feed_text(t2)`) |

Both call `_extract_data`: single-chunk LLM invoke or batched parallel extraction, then `merge_batch_data`. `feed_text` routes through `_update_data_state` (incremental merge); `parse` routes through `_set_data_state` on a fresh instance.

### Indexing, search, and chat

| Method | Notes |
| --- | --- |
| `build_index()` | Abstract; builds FAISS vector store from current data. `AutoGraph.build_index(index_nodes=True, index_edges=True)` indexes nodes and edges separately. |
| `search(query, top_k=3)` | Abstract; signature varies by type. `AutoGraph.search` returns `(nodes, edges)` and requires a prior `build_index()`. |
| `chat(query, top_k=3)` | Concrete on `BaseAutoType`. Runs `search`, formats context, invokes `llm_client`, returns `langchain_core.messages.AIMessage`. Retrieved items are attached at `response.additional_kwargs["retrieved_items"]`. |

<Note>
Call `build_index()` before `search` or `chat` when semantic retrieval is required. Index state is cleared when data is replaced (`parse` on a loaded instance) or incrementally updated (`feed_text`).
</Note>

### Persistence

`dump(folder_path)` writes a Knowledge Abstract directory:

:::files
output/
├── data.json       # Pydantic `data.model_dump()`
├── metadata.json   # timestamps, template, lang, type
└── index/          # FAISS files (non-fatal if save fails)
:::

`load(folder_path)` restores data (required), metadata (optional), and index (optional). Index load failures print a warning; call `build_index()` to rebuild.

Granular helpers: `dump_data`, `load_data`, `dump_metadata`, `load_metadata`, `dump_index`, `load_index`.

### State helpers

| Method / property | Behavior |
| --- | --- |
| `data` | Read-only Pydantic view of stored knowledge |
| `data_schema` | Schema class (`Type[T]`) |
| `empty()` | Whether any data is stored |
| `clear()` | Reset data and index |
| `clear_index()` | Reset index only |
| `__add__(other)` | Merge two instances of the same class and schema into a new instance |

### `show()`

`show()` is **not** defined on `BaseAutoType`. Concrete types implement OntoSight visualization:

- `AutoModel.show(label_extractor=..., top_k=3)`
- `AutoList.show(...)`, `AutoSet.show(...)`
- `AutoGraph.show(node_label_extractor=..., edge_label_extractor=..., top_k_nodes_for_search=3, ...)`

When a vector index exists, `show()` wires interactive search and chat callbacks into the OntoSight viewer.

```mermaid
stateDiagram-v2
    [*] --> Empty: Template.create / direct construct
    Empty --> Populated: parse() or feed_text()
    Populated --> Indexed: build_index()
    Indexed --> Queried: search() / chat()
    Populated --> Persisted: dump()
    Persisted --> Loaded: load()
    Loaded --> Indexed: build_index()
    Queried --> Evolved: feed_text()
    Evolved --> Indexed: build_index()
```

## Client factory

All client helpers return LangChain-compatible objects and accept the `provider:model@url` shorthand.

| Function | Returns | Config source |
| --- | --- | --- |
| `create_llm(spec, api_key="", **kwargs)` | `ChatOpenAI` | String shorthand or dict |
| `create_embedder(spec, api_key="", **kwargs)` | `OpenAIEmbeddings` or `CompatibleEmbeddings` | Uses `CompatibleEmbeddings` when `base_url` is not the official OpenAI endpoint |
| `create_client(llm=, embedder=, provider=, api_key=, **kwargs)` | `(llm, embedder)` tuple | Three patterns below |
| `get_client(config_path=None)` | `(llm, embedder)` tuple | Reads `~/.he/config.toml` via `ConfigManager` |

**Pattern A — single provider (cloud defaults):**

```python
llm, emb = create_client("bailian", api_key="sk-...")
# preset: qwen3.6-plus + text-embedding-v4
```

**Pattern B — separate vLLM services:**

```python
llm, emb = create_client(
    llm="vllm:Qwen3.5-9B@http://localhost:8000/v1",
    embedder="vllm:bge-m3@http://localhost:8001/v1",
    api_key="dummy",
)
```

**Pattern C — mixed deployment:**

```python
llm, emb = create_client(
    llm="bailian:qwen-plus",
    embedder="vllm:bge-m3@localhost:8001/v1",
    api_key="sk-...",
)
```

Shorthand parsing (`_parse_client_spec`):

| Input | Resolves to |
| --- | --- |
| `"bailian"` | Provider preset defaults |
| `"bailian:qwen-plus"` | Provider + model, preset URL |
| `"vllm:Qwen3.5-9B@http://localhost:8000/v1"` | Provider + model + base URL |

Built-in presets: `openai`, `bailian`, `vllm` (no default URL or models).

<Tip>
`CompatibleEmbeddings` sends string inputs (not pre-tokenized integer lists) to OpenAI-compatible endpoints, with tiktoken chunking and conservative `max_batch_size=10` for providers like Bailian/DashScope.
</Tip>

## Logging helpers

```python
from hyperextract import configure_logging, get_logger, set_log_level

configure_logging(level="WARNING", json_output=False, output_file=None)
logger = get_logger(__name__)
set_log_level("DEBUG")
```

| Function | Behavior |
| --- | --- |
| `configure_logging` | Configures structlog processors; console handler on stderr; optional file handler |
| `get_logger(name)` | Returns a `structlog.stdlib.BoundLogger` |
| `set_log_level(level)` | Updates root logger level at runtime |

Environment variables override defaults:

| Variable | Effect |
| --- | --- |
| `HYPER_EXTRACT_LOG_LEVEL` | Overrides `configure_logging(level=...)` |
| `HYPER_EXTRACT_LOG_FILE` | Adds a file handler when `output_file` is not set |

`BaseAutoType` and client code emit structured debug events (`stage=extract_start`, `stage=feed_text_start`, etc.) through `get_logger`.

## End-to-end workflow

<Steps>

<Step title="Configure clients">

```python
from hyperextract import create_client, configure_logging

configure_logging(level="INFO")
llm, emb = create_client("openai")  # or get_client() after he config init
```

</Step>

<Step title="Create and extract">

```python
from hyperextract import Template

ka = Template.create("general/biography_graph", language="en", llm_client=llm, embedder=emb)

with open("examples/en/tesla.md") as f:
    ka.feed_text(f.read())
```

</Step>

<Step title="Index, query, persist">

```python
ka.build_index()

results = ka.search("Tesla alternating current", top_k=3)
answer = ka.chat("What did Tesla invent?")
print(answer.content)

ka.dump("./output/tesla_ka")
```

</Step>

<Step title="Reload and visualize">

```python
loaded = Template.create("general/biography_graph", language="en", llm_client=llm, embedder=emb)
loaded.load("./output/tesla_ka")
loaded.show()
```

</Step>

</Steps>

For extraction algorithms without YAML presets, instantiate method classes directly (`from hyperextract.methods.rag import Light_RAG`) or use `Template.create("method/<name>")`. See [Use extraction methods](/use-extraction-methods).

## Error cases

| Condition | Exception / signal |
| --- | --- |
| Knowledge template without `language` | `ValueError: language is required for knowledge templates` |
| Unknown preset or method path | `ValueError: Template not found: <source>` |
| `create_client()` with no args | `ValueError` with usage examples |
| `load()` on missing directory | `FileNotFoundError` |
| `search()` without index (graph types) | `ValueError: Node index not built. Call build_index() first.` |
| Merging incompatible instances via `+` | `TypeError` (class or schema mismatch) |
| Index dump/load failure | Warning printed; data still persisted |

## Related pages

<CardGroup>

<Card title="Quickstart" href="/quickstart">
First successful extraction with `Template.create` and `feed_text` using the Tesla biography example.
</Card>

<Card title="Knowledge Abstracts" href="/knowledge-abstracts">
On-disk `data.json` / `metadata.json` / `index/` layout and lifecycle semantics.
</Card>

<Card title="Configure providers" href="/configure-providers">
Set up `~/.he/config.toml` or programmatic `create_client()` for mixed deployments.
</Card>

<Card title="Auto-Types" href="/auto-types">
Choose and construct the eight `Auto*` primitives directly.
</Card>

<Card title="Tesla biography recipe" href="/tesla-biography-recipe">
End-to-end CLI and Python workflow with expected artifacts.
</Card>

</CardGroup>