# Contributing

> Development setup with `uv`, running `pytest` and coverage, CI matrix (Python 3.11–3.12, Ubuntu/macOS), lint workflow, optional integration tests, and how to add templates or register new extraction methods.

- Repository: yifanfeng97/Hyper-Extract
- GitHub: https://github.com/yifanfeng97/Hyper-Extract
- Human docs: https://grok-wiki.com/public/docs/yifanfeng97-hyper-extract-7891c7254cdf
- Complete Markdown: https://grok-wiki.com/public/docs/yifanfeng97-hyper-extract-7891c7254cdf/llms-full.txt

## Source Files

- `.github/workflows/test.yml`
- `.github/workflows/lint.yml`
- `.github/workflows/integration.yml`
- `pyproject.toml`
- `tests/conftest.py`
- `hyperextract/methods/registry.py`
- `README.md`

---

---
title: "Contributing"
description: "Development setup with `uv`, running `pytest` and coverage, CI matrix (Python 3.11–3.12, Ubuntu/macOS), lint workflow, optional integration tests, and how to add templates or register new extraction methods."
---

Hyper-Extract targets Python 3.11+, uses `uv` for dependency management, and gates changes through three GitHub Actions workflows: unit tests with coverage (`test.yml`), Ruff lint/format (`lint.yml`), and optional integration tests against a live LLM (`integration.yml`). Preset YAML templates auto-load from `hyperextract/templates/presets/`; extraction methods register through `hyperextract/methods/registry.py`.

## Prerequisites

| Requirement | Value |
|-------------|-------|
| Python | `>=3.11` (repo pins `3.11` in `.python-version`) |
| Package manager | [uv](https://docs.astral.sh/uv/) |
| License | Apache-2.0 |

<Note>
Unit tests run without API keys. `tests/conftest.py` injects `MockChatModel` and `MockEmbeddings` when `OPENAI_API_KEY` is unset or empty. CI explicitly sets `OPENAI_API_KEY: ""` for deterministic runs.
</Note>

## Development setup

<Steps>
<Step title="Clone and enter the repository">

```bash
git clone https://github.com/yifanfeng97/Hyper-Extract.git
cd Hyper-Extract
```

</Step>

<Step title="Install the package in editable mode">

<Tabs>
<Tab title="All provider extras (recommended)">

```bash
uv pip install -e ".[all]"
```

Installs core dependencies plus `anthropic` and `google` optional extras.

</Tab>
<Tab title="Core only">

```bash
uv pip install -e .
```

</Tab>
<Tab title="With dev tools">

```bash
uv pip install -e ".[all]"
uv pip install --group dev pytest pytest-cov ruff
```

The `dev` dependency group in `pyproject.toml` also includes MkDocs tooling for documentation builds.

</Tab>
</Tabs>

</Step>

<Step title="Verify the install">

```bash
he --help
python -c "from hyperextract import Template; print(len(Template.list()))"
```

</Step>
</Steps>

## Running tests

### Unit tests and coverage

The default test suite exercises `hyperextract/` and `tests/` without network calls when no API key is present.

```bash
pytest --cov=hyperextract --cov-report=term -v
```

CI additionally emits XML coverage for Codecov:

```bash
pytest --cov=hyperextract --cov-report=xml --cov-report=term -v
```

| Fixture | Scope | Behavior |
|---------|-------|----------|
| `is_real_env` | session | `True` when `OPENAI_API_KEY` is set and non-placeholder |
| `llm_client` | session | `ChatOpenAI` (real) or `MockChatModel` (mock) |
| `embedder` | session | `OpenAIEmbeddings` (real) or `MockEmbeddings` (mock) |

`tests/conftest.py` loads `.env` from the repo root and the current working directory before fixtures resolve.

### Integration tests

Integration tests live under `tests/integration/` and are marked with `@pytest.mark.integration`. They call the real OpenAI API and skip when `OPENAI_API_KEY` is absent.

```bash
export OPENAI_API_KEY=sk-...
pytest -m integration -v --tb=short
```

<Warning>
Integration tests incur API cost. They are not part of the PR gate. CI runs them nightly (02:00 UTC) and on manual `workflow_dispatch` against the `integration-tests` GitHub environment, which supplies `OPENAI_API_KEY` from repository secrets.
</Warning>

### Test layout

:::files
tests/
├── conftest.py              # Mock/real auto-detection, shared fixtures
├── mocks.py                 # MockChatModel, MockEmbeddings
├── types/                   # AutoType unit tests
├── template_engine/         # Gallery, parser, factory tests
├── cli/                     # CLI behavior tests
├── utils/                   # Client/config tests
├── integration/             # Live-API extraction tests
└── test_data/               # Sample documents (en/, zh/)
:::

## Linting

CI runs Ruff on every push and PR to `main` or `develop` when `hyperextract/**/*.py` or `pyproject.toml` changes.

```bash
uv tool install ruff
ruff check hyperextract
ruff format --check hyperextract
```

Apply fixes locally:

```bash
ruff check --fix hyperextract
ruff format hyperextract
```

`pyproject.toml` configures `[tool.ruff.lint]` with `ignore = ["E731"]`.

## CI workflows

| Workflow | Trigger | Runner matrix | Purpose |
|----------|---------|---------------|---------|
| `test.yml` | Push/PR to `main`, `develop` | Ubuntu + macOS × Python 3.11/3.12 | Unit tests + coverage |
| `lint.yml` | Push/PR to `main`, `develop` | `ubuntu-latest` | Ruff check + format |
| `integration.yml` | Nightly cron, `workflow_dispatch` | `ubuntu-latest` | Live-API integration tests |

### Test matrix detail

The test workflow uses `fail-fast: false` and runs these combinations:

| OS | Python 3.11 | Python 3.12 |
|----|:-----------:|:-----------:|
| `ubuntu-latest` | ✓ | ✓ |
| `macos-latest` | — | ✓ |

macOS + Python 3.11 is excluded to reduce CI time. Coverage uploads to Codecov run only on `ubuntu-latest` + Python 3.12.

Path filters limit CI to changes under `hyperextract/**/*.py`, `tests/**/*.py`, and `pyproject.toml`.

```text
push/PR ──► test.yml ──► matrix(os × python) ──► pytest --cov
         ──► lint.yml ──► ruff check + format --check
         ──► integration.yml (scheduled/manual) ──► pytest -m integration
```

## Adding preset templates

Preset templates are YAML files under `hyperextract/templates/presets/{domain}/`. The `Gallery` singleton scans `*.yaml` at import time and indexes each file as `{domain}/{name}`, where `name` comes from the YAML `name` field (not the filename).

<Steps>
<Step title="Pick a domain and base template">

| Domain | Examples | Base templates |
|--------|----------|----------------|
| `general/` | `biography_graph`, `concept_graph` | `base_model`, `base_list`, `base_graph`, … |
| `finance/` | `earnings_summary`, `event_timeline` | — |
| `medicine/`, `tcm/`, `legal/`, `industry/` | Domain-specific presets | — |

Start from the `base_*` template matching your target AutoType (`model`, `list`, `set`, `graph`, `hypergraph`, `temporal_graph`, `spatial_graph`, `spatio_temporal_graph`).

</Step>

<Step title="Author the YAML">

Follow `hyperextract/templates/DESIGN_GUIDE.md` for field naming, multilingual blocks, and validation. Minimum required fields:

<ParamField body="language" type="string | string[]" required>
Supported locales, e.g. `[zh, en]`.
</ParamField>

<ParamField body="name" type="string" required>
Template ID segment; combined with domain folder to form the gallery key (e.g. `finance/earnings_summary`).
</ParamField>

<ParamField body="type" type="string" required>
One of the eight AutoTypes.
</ParamField>

<ParamField body="output" type="object" required>
Field/entity/relation schemas for extraction.
</ParamField>

<ParamField body="guideline" type="object" required>
Extraction rules and prompts.
</ParamField>

</Step>

<Step title="Validate locally">

```bash
pytest tests/template_engine/ -v
python -c "from hyperextract import Template; assert Template.get('your_domain/your_name')"
```

Gallery tests in `tests/template_engine/test_gallery.py` verify discovery, filtering by type/language/tag, and config preservation.

</Step>

<Step title="Test extraction (optional, requires API key)">

```bash
he parse tests/test_data/en/general/biography_scientist.md \
  -t your_domain/your_name -o /tmp/ka-test -l en
```

Or via the Python API:

```python
from hyperextract import Template

ka = Template.create("your_domain/your_name", "en")
result = ka.parse(open("tests/test_data/en/general/biography_scientist.md").read())
```

</Step>
</Steps>

Custom templates outside the presets tree can also be loaded by file path: `Template.create("/path/to/template.yaml", "en")`.

## Registering extraction methods

Methods are algorithm-driven extractors that subclass an AutoType (typically `AutoGraph` or `AutoHypergraph`) and register in `hyperextract/methods/registry.py`. After registration, they appear as `method/{name}` in `Template.create()`, `Template.get()`, and `he list method`.

<Steps>
<Step title="Implement the method class">

Place the class under `hyperextract/methods/rag/` or `hyperextract/methods/typical/`. The constructor must accept `llm_client` and `embedder`, with optional kwargs forwarded by `TemplateFactory.create_method()`.

```python
from hyperextract.types import AutoGraph

class My_RAG(AutoGraph[NodeSchema, EdgeSchema]):
    def __init__(
        self,
        llm_client: BaseChatModel,
        embedder: Embeddings,
        chunk_size: int = 2048,
        verbose: bool = False,
    ):
        super().__init__(
            node_schema=NodeSchema,
            edge_schema=EdgeSchema,
            llm_client=llm_client,
            embedder=embedder,
            # ... key extractors, prompts, mergers
        )
```

See `hyperextract/methods/rag/light_rag.py` for a complete reference implementation.

</Step>

<Step title="Register in the method registry">

Add a `register_method()` call inside `_init_registry()` in `hyperextract/methods/registry.py`:

```python
register_method(
    name="my_rag",
    method_class=My_RAG,
    autotype="graph",  # graph | hypergraph | model | list | set | ...
    description="Short description shown in he list method",
)
```

<ParamField body="name" type="string" required>
Registry key and CLI `-m` value (e.g. `light_rag`).
</ParamField>

<ParamField body="method_class" type="Type" required>
Class constructor; must accept `llm_client`, `embedder`, and `**kwargs`.
</ParamField>

<ParamField body="autotype" type="string" required>
Output AutoType category stored in method metadata.
</ParamField>

</Step>

<Step title="Export and add a demo script">

Export the class from the appropriate `__init__.py` under `hyperextract/methods/`. Add a runnable demo under `examples/en/methods/` following the existing `*_demo.py` pattern (dotenv, LangChain clients, `feed_text`, `chat`, `show`).

</Step>

<Step title="Verify registration">

```bash
he list method
python -c "from hyperextract import Template; print(Template.get('method/my_rag'))"
pytest tests/ -v -k "method or factory"  # existing factory/registry coverage
```

Invocation paths after registration:

<CodeGroup>
```bash title="CLI"
he parse examples/en/tesla.md -m my_rag -o ./output/
```

```python title="Python API"
from hyperextract import Template

ka = Template.create("method/my_rag")
ka.feed_text(open("examples/en/tesla.md").read())
```

```python title="Direct class"
from hyperextract.methods.rag import My_RAG

rag = My_RAG(llm_client=llm, embedder=embedder)
```
</CodeGroup>

</Step>
</Steps>

<Tip>
Method templates use English prompts only. `TemplateFactory.create_method()` hardcodes `metadata["lang"] = "en"` regardless of any `--lang` flag.
</Tip>

## Pull request checklist

Before opening a PR against `main` or `develop`:

1. Run `pytest --cov=hyperextract --cov-report=term -v` locally (no API key required).
2. Run `ruff check hyperextract` and `ruff format --check hyperextract`.
3. For template changes: run `pytest tests/template_engine/ -v` and confirm `Template.get()` resolves the new key.
4. For method changes: confirm `he list method` includes the new entry and add an `examples/en/methods/` demo when practical.
5. Keep changes scoped; the package build excludes `docs/`, `tests/`, and `.github/` from the wheel per `[tool.hatch.build]`.

Report bugs and feature requests via [GitHub Issues](https://github.com/yifanfeng97/hyper-extract/issues).

## Related pages

<CardGroup>
<Card title="Create custom templates" href="/create-custom-templates">
Author domain YAML templates: type selection, fields, multilingual blocks, and merge strategies.
</Card>
<Card title="Extraction methods reference" href="/extraction-methods-reference">
Registered methods, autotype outputs, registry API, and constructor kwargs.
</Card>
<Card title="Template schema reference" href="/template-schema-reference">
YAML field definitions, valid AutoTypes, identifiers, and merge strategies.
</Card>
<Card title="Template design skills" href="/template-design-skills">
Agent-assisted authoring with `hyperextract-skills` validators and optimizers.
</Card>
</CardGroup>