DeepWiki-Open Technical Onboarding

Complete Markdown

# DeepWiki-Open Technical Onboarding

> An AI-powered repository-to-wiki generator. This wiki serves as a fast-track guide to understanding the engine, its multi-provider adapter system, and its interactive frontend.

## Context Links

- [Agent index](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/llms.txt)
- [Human interactive wiki](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747)
- [GitHub repository](https://github.com/AsyncFuncAI/deepwiki-open)

## Repository Metadata

- Repository: AsyncFuncAI/deepwiki-open

- Generated: 2026-05-19T19:17:21.534Z
- Updated: 2026-05-21T21:35:59.039Z
- Runtime: Antigravity CLI
- Format: First 30 Minutes
- Pages: 6

## Page Index

- 01. [Start Here: The Onboarding Map](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/01-start-here-the-onboarding-map.md) - High-level system overview, fast read order, and core entry points for both the Python backend and Next.js frontend.
- 02. [Configuring Your AI Matrix](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/02-configuring-your-ai-matrix.md) - Setting up the multi-provider environment, from environment variables to LLM-specific JSON configurations for generation and repository analysis.
- 03. [Data Pipeline & Analysis Logic](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/03-data-pipeline-analysis-logic.md) - Deep dive into the core processing logic: how repositories are cloned, split by tiktoken, embedded via AdalFlow, and indexed for RAG-powered wiki generation.
- 04. [The Multi-Provider Adapter Layer](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/04-the-multi-provider-adapter-layer.md) - Exploration of the BYOC (Bring Your Own Client) architecture, supporting Google Gemini, OpenAI, Azure, OpenRouter, and local Ollama instances.
- 05. [The Next.js Interface & Diagrams](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/05-the-next.js-interface-diagrams.md) - Architecture of the frontend application, focusing on real-time updates via WebSockets and automated Mermaid diagram rendering for repository visualization.
- 06. [Closing the Loop: DeepResearch & Q&A](https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/06-closing-the-loop-deepresearch-q-a.md) - Understanding the final stage of user interaction: the "Ask" feature and DeepResearch module for multi-turn repository investigations.

## Source File Index

- `api/api.py`
- `api/azureai_client.py`
- `api/bedrock_client.py`
- `api/config.py`
- `api/config/generator.json`
- `api/config/repo.json`
- `api/dashscope_client.py`
- `api/data_pipeline.py`
- `api/google_embedder_client.py`
- `api/logging_config.py`
- `api/main.py`
- `api/ollama_patch.py`
- `api/openai_client.py`
- `api/openrouter_client.py`
- `api/prompts.py`
- `api/rag.py`
- `api/simple_chat.py`
- `api/tools/embedder.py`
- `api/websocket_wiki.py`
- `docker-compose.yml`
- `Dockerfile`
- `next.config.ts`
- `Ollama-instruction.md`
- `package.json`
- `pytest.ini`
- `README.md`
- `run.sh`
- `src/app/globals.css`
- `src/app/layout.tsx`
- `src/app/page.tsx`
- `src/components/Ask.tsx`
- `src/components/ConfigurationModal.tsx`
- `src/components/Mermaid.tsx`
- `src/components/WikiTreeView.tsx`

---

## 01. Start Here: The Onboarding Map

> High-level system overview, fast read order, and core entry points for both the Python backend and Next.js frontend.

- Page Markdown: https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/01-start-here-the-onboarding-map.md
- Generated: 2026-05-19T19:05:22.078Z

### Source Files

- `README.md`
- `api/main.py`
- `api/api.py`
- `src/app/page.tsx`
- `docker-compose.yml`
- `run.sh`

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [README.md](README.md)
- [api/main.py](api/main.py)
- [api/api.py](api/api.py)
- [api/rag.py](api/rag.py)
- [api/config.py](api/config.py)
- [src/app/page.tsx](src/app/page.tsx)
- [docker-compose.yml](docker-compose.yml)
- [run.sh](run.sh)
</details>

# Start Here: The Onboarding Map

DeepWiki-Open is an automated, AI-driven documentation system designed to transform source code repositories into interactive, navigable wikis. By combining large language models (LLMs) with Retrieval Augmented Generation (RAG), the system provides deep insights into code architecture, data flows, and component relationships without manual intervention.

This page serves as the high-level map for new contributors and users, outlining the system's modular architecture, core logic flows, and the recommended sequence for exploring the codebase.

## System Overview

The project is structured as a decoupled full-stack application:
- **Backend (Python):** A FastAPI-based service that handles repository management (cloning, indexing), RAG operations, and LLM orchestration.
- **Frontend (Next.js):** A modern React application providing an intuitive interface for repository submission, configuration, and wiki browsing.
- **RAG Engine:** Powered by [AdalFlow](https://github.com/AsyncFuncAI/AdalFlow) and FAISS, it enables context-aware analysis and Q&A capabilities.

## Architecture & Data Flow

The following diagram illustrates the high-level interaction between the user, the frontend, and the backend services.

```mermaid
graph TD
    User([User])
    subgraph Frontend [Next.js App]
        Home[src/app/page.tsx]
        WikiView["src/app/[owner]/[repo]/page.tsx"]
    end
    subgraph Backend [FastAPI Server]
        Entry[api/main.py]
        API[api/api.py]
        RAG[api/rag.py]
        Pipeline[api/data_pipeline.py]
    end
    subgraph Persistence [Data Layer]
        Repos["~/.adalflow/repos/"]
        Vectors["~/.adalflow/databases/"]
        Cache["~/.adalflow/wikicache/"]
    end

    User -->|Enter URL| Home
    Home -->|Request Wiki| API
    API -->|Initialize RAG| RAG
    RAG -->|Clone/Process| Pipeline
    Pipeline -->|Source Files| Repos
    Pipeline -->|Embeddings| Vectors
    RAG -->|Generate Pages| Cache
    API -->|Serve Wiki| WikiView
```

## Fast Read Order

To quickly understand the system implementation, read the following files in order:

1.  **[README.md](README.md)**: Start here for a high-level overview of features, quick-start guides, and supported model providers.
    Sources: [README.md:1-30]()
2.  **[api/main.py](api/main.py)**: The backend entry point. It sets up environment variables, logging, and initializes the FastAPI server via Uvicorn.
    Sources: [api/main.py:62-78]()
3.  **[api/api.py](api/api.py)**: This file defines the core API routes, including wiki generation, caching logic, and model configuration retrieval.
    Sources: [api/api.py:20-33]()
4.  **[api/rag.py](api/rag.py)**: The "brain" of the application. It implements the RAG class which integrates vector search (FAISS) with LLM generation to produce grounded documentation.
    Sources: [api/rag.py:153-170]()
5.  **[src/app/page.tsx](src/app/page.tsx)**: The primary frontend landing page where users configure their repository and model selections.
    Sources: [src/app/page.tsx:45-116]()

## Core Entry Points

### Backend (Python/FastAPI)
The backend is a FastAPI application hosted in the `api/` directory. It uses `uvicorn` for high-performance serving.
- **Entry Point**: `api/main.py`
- **Main Logic**: `api/api.py` handles the HTTP endpoints, while `api/rag.py` manages the AI interactions.
- **Startup Command**: `uv run -m api.main` (or via `run.sh`).

### Frontend (Next.js)
The frontend is built with Next.js 13+ (App Router) and styled with Tailwind CSS.
- **Main Landing**: `src/app/page.tsx`
- **Wiki Display**: Dynamic routes located at `src/app/[owner]/[repo]/page.tsx`.
- **Interactions**: Uses `src/components/ConfigurationModal.tsx` for repository setup and `Mermaid.tsx` for diagram rendering.

## Configuration & Environment

DeepWiki-Open is designed to be provider-neutral, supporting various LLM and embedding providers via a flexible configuration system.

| Category | Primary Config File | Description |
| :--- | :--- | :--- |
| **Generators** | `api/config/generator.json` | LLM model providers (Google, OpenAI, OpenRouter, etc.) |
| **Embedders** | `api/config/embedder.json` | Vector embedding models and RAG retriever settings |
| **Repository** | `api/config/repo.json` | File filters and repository size limits |

Sources: [api/config.py:99-121](), [api/config.py:332-357]()

## Summary

DeepWiki-Open provides a robust, modular framework for automated repository documentation. By leveraging a decoupled Next.js/FastAPI stack and a sophisticated RAG pipeline, it offers a scalable solution for understanding complex codebases through interactive wikis and AI-powered Q&A.

Sources: [api/rag.py:345-380](), [README.md:116-126]()

---

## 02. Configuring Your AI Matrix

> Setting up the multi-provider environment, from environment variables to LLM-specific JSON configurations for generation and repository analysis.

- Page Markdown: https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/02-configuring-your-ai-matrix.md
- Generated: 2026-05-19T19:05:22.630Z

### Source Files

- `api/config.py`
- `api/config/generator.json`
- `api/config/repo.json`
- `next.config.ts`
- `package.json`
- `Dockerfile`

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [api/config.py](api/config.py)
- [api/config/generator.json](api/config/generator.json)
- [api/config/repo.json](api/config/repo.json)
- [api/config/embedder.json](api/config/embedder.json)
- [api/main.py](api/main.py)
- [Dockerfile](Dockerfile)
</details>

# Configuring Your AI Matrix

The DeepWiki AI Matrix is a modular configuration framework designed to support a multi-provider LLM environment. It enables the system to switch seamlessly between different AI providers for text generation, code analysis, and vector embeddings, while maintaining a consistent internal interface.

This configuration is primarily managed through a combination of environment variables and structured JSON files located in the `api/config/` directory. The system is built to be provider-neutral, allowing developers to bring their own keys (BYOK) and customize the behavior of each AI agent.

## Environment Variables

DeepWiki requires several environment variables to authenticate with AI providers and manage its runtime behavior. These variables are loaded at startup and can be provided via a `.env` file or directly in the shell environment.

| Variable | Description | Default |
|----------|-------------|---------|
| `GOOGLE_API_KEY` | API key for Google Gemini models (Required) | None |
| `OPENAI_API_KEY` | API key for OpenAI GPT models (Required) | None |
| `OPENROUTER_API_KEY` | API key for OpenRouter models | None |
| `DEEPWIKI_EMBEDDER_TYPE` | Type of embedder to use (`openai`, `google`, `ollama`, `bedrock`) | `openai` |
| `DEEPWIKI_CONFIG_DIR` | Custom directory for JSON configuration files | `api/config/` |
| `PORT` | The port for the backend API server | `8001` |

Sources: [api/config.py:19-26](api/config.py#L19-L26), [api/main.py:47-51](api/main.py#L47-L51), [Dockerfile:120-130](Dockerfile#L120-L130)

## JSON Configuration Matrix

The core logic of the AI Matrix resides in three primary JSON files. These files define the providers, models, and hyperparameters used across the application.

### 1. Text Generation (`generator.json`)
Defines the LLM providers and their specific models used for wiki generation and chat. It supports a wide range of providers including Dashscope, Google, OpenAI, OpenRouter, Ollama, Bedrock, and Azure.

- **Default Provider**: Set via `default_provider`.
- **Model Parameters**: Each model can have custom `temperature`, `top_p`, and `top_k` settings.

Sources: [api/config/generator.json:1-100](api/config/generator.json#L1-L100)

### 2. Embedding & Retrieval (`embedder.json`)
Configures the vectorization process for repository analysis. It supports multiple embedding engines and defines how text is split before indexing.

- **Client Classes**: Maps providers to their respective client implementations (e.g., `OpenAIClient`, `OllamaClient`).
- **Text Splitter**: Controls `chunk_size` and `chunk_overlap` for RAG optimization.

Sources: [api/config/embedder.json:1-40](api/config/embedder.json#L1-L40)

### 3. Repository Analysis (`repo.json`)
Manages the scope of the AI's "vision" by defining file filters and repository limits.

- **Excluded Directories**: Automatically ignores common build artifacts (`node_modules`, `.venv`, `.git`).
- **File Filters**: Excludes lock files and binary blobs to minimize token usage.

Sources: [api/config/repo.json:1-50](api/config/repo.json#L1-L50)

## Dynamic Configuration & Placeholders

The AI Matrix supports dynamic values within the JSON files using the `${ENV_VAR}` syntax. This allows you to keep sensitive keys out of the configuration files while maintaining flexibility.

```json
{
  "api_key": "${MY_CUSTOM_SECRET}"
}
```

The `replace_env_placeholders` function recursively traverses the configuration objects at runtime to inject the appropriate environment values.

Sources: [api/config.py:69-97](api/config.py#L69-L97)

## Configuration Architecture

The following diagram illustrates how configuration flows from environment sources into the DeepWiki runtime:

```mermaid
flowchart TD
    Env[Environment Variables/.env] --> ConfigPy[api/config.py]
    JsonFiles[JSON Config Files: generator, embedder, repo] --> ConfigPy
    ConfigPy --> Placeholders[Placeholder Replacement]
    Placeholders --> ClientInit[Client & Provider Initialization]
    
    subgraph "api/config/"
        JsonFiles
    end
    
    subgraph "Core Logic"
        ConfigPy
        Placeholders
    end
    
    ClientInit --> LLM[LLM Generation]
    ClientInit --> RAG[Vector Search/RAG]
```

## Summary

DeepWiki's AI Matrix provides a robust and extensible foundation for multi-provider AI integration. By separating model metadata into JSON files and sensitive credentials into environment variables, it ensures a secure and portable development environment.

Sources: [api/config.py:331-357](api/config.py#L331-L357)

---

## 03. Data Pipeline & Analysis Logic

> Deep dive into the core processing logic: how repositories are cloned, split by tiktoken, embedded via AdalFlow, and indexed for RAG-powered wiki generation.

- Page Markdown: https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/03-data-pipeline-analysis-logic.md
- Generated: 2026-05-19T19:05:20.612Z

### Source Files

- `api/data_pipeline.py`
- `api/rag.py`
- `api/websocket_wiki.py`
- `api/prompts.py`
- `api/logging_config.py`
- `api/tools/embedder.py`

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [api/data_pipeline.py](api/data_pipeline.py)
- [api/rag.py](api/rag.py)
- [api/websocket_wiki.py](api/websocket_wiki.py)
- [api/tools/embedder.py](api/tools/embedder.py)
- [api/prompts.py](api/prompts.py)
- [api/logging_config.py](api/logging_config.py)
</details>

# Data Pipeline & Analysis Logic

DeepWiki-Open implements a robust data pipeline designed to ingest, process, and index Git repositories for Retrieval-Augmented Generation (RAG). This system transforms raw source code and documentation into a queryable knowledge base by leveraging modular components for cloning, tokenization, embedding, and vector retrieval.

The pipeline is built on the `adalflow` library, providing a sequential workflow that handles everything from initial repository cloning to the final persistence of a FAISS-backed vector database.

## Repository Acquisition and Ingestion

The ingestion process begins with the `download_repo` function, which supports cloning from GitHub, GitLab, and Bitbucket. To ensure efficiency and minimize storage overhead, the system performs a shallow clone with a depth of 1.

Once the repository is available locally, the `read_all_documents` function recursively scans the directory structure. It applies a filtering mechanism based on file extensions, prioritizing implementation files (e.g., `.py`, `.js`, `.ts`, `.go`) while also including documentation (e.g., `.md`, `.txt`).

### File Filtering and Processing Logic
- **Exclusion/Inclusion**: Users can specify directories or file patterns to explicitly include or exclude during the scan.
- **Token Budgeting**: Files exceeding 10 times the `MAX_EMBEDDING_TOKENS` (8192) are skipped to prevent processing bottlenecks.
- **Metadata Tagging**: Each ingested document is tagged with metadata such as `file_path`, `type`, and whether it is considered an `is_implementation` file.

Sources: [api/data_pipeline.py:72-157](api/data_pipeline.py#L72-L157), [api/data_pipeline.py:161-388](api/data_pipeline.py#L161-L388)

## Data Pipeline & Analysis Flow

DeepWiki uses a modular architecture to transform raw files into searchable vectors. The following diagram illustrates the lifecycle of data from the initial repository URL to the indexed vector database.

```mermaid
graph TD
    A[Repo URL] --> B[git clone --depth 1]
    B --> C[Local Directory]
    C --> D[read_all_documents]
    D --> E[TextSplitter]
    E --> F[ToEmbeddings / OllamaProcessor]
    F --> G[LocalDB / FAISS]
    G --> H[.pkl Storage]
    
    subgraph Data Transformation
    E
    F
    end
    
    subgraph Vector Indexing
    G
    H
    end
```

## Tokenization and Embedding Strategy

Before embedding, text is split into manageable chunks using a `TextSplitter`. The system uses `tiktoken` to estimate token counts, ensuring chunks remain within the limits of the chosen embedding model.

DeepWiki supports multiple embedding providers via a centralized `get_embedder` utility. Depending on the configuration, it initializes an `adal.Embedder` for OpenAI, Google, Bedrock, or Ollama.

| Component | Responsibility | Provider Support |
| :--- | :--- | :--- |
| **Token Counter** | Estimates tokens using `cl100k_base` or model-specific encodings. | OpenAI, Google, Ollama, Bedrock |
| **Embedder** | Generates high-dimensional vectors for text chunks. | OpenAI, Google, Ollama, Bedrock |
| **Batch Processor** | Handles bulk embedding requests for API efficiency. | OpenAI, Google (via `ToEmbeddings`) |
| **Single Processor** | Handles per-document embedding for local models. | Ollama (via `OllamaDocumentProcessor`) |

Sources: [api/data_pipeline.py:27-70](api/data_pipeline.py#L27-L70), [api/tools/embedder.py:6-58](api/tools/embedder.py#L6-L58), [api/data_pipeline.py:390-432](api/data_pipeline.py#L390-L432)

## Vector Indexing and Retrieval Logic

The final stage of the pipeline involves indexing the transformed documents. DeepWiki utilizes `FAISSRetriever` to enable high-performance similarity searches. 

### Database Management
The `DatabaseManager` handles the persistence of the processed data. It saves the transformed documents and their corresponding vectors into a `.pkl` file within the `~/.adalflow/databases/` directory. This allows for quick loading in subsequent sessions without re-processing the entire repository.

### Retrieval Validation
A critical step in the retrieval preparation is the validation of embedding sizes. The `_validate_and_filter_embeddings` method ensures that all document vectors match the target size (e.g., 1536 for OpenAI `text-embedding-3-small`), filtering out any inconsistent or failed embeddings before the FAISS index is built.

Sources: [api/rag.py:345-414](api/rag.py#L345-L414), [api/rag.py:251-343](api/rag.py#L251-L343)

## WebSocket Integration for Real-time Analysis

DeepWiki exposes its analysis logic through a WebSocket interface, allowing for streaming RAG-powered chat interactions. When a request is received, the system:
1. **Initializes RAG**: Prepares the retriever for the specific repository.
2. **Context Retrieval**: Uses the `FAISSRetriever` to find relevant code snippets based on the user's query.
3. **Prompt Augmentation**: Injecting the retrieved snippets into a structured prompt (defined in `api/prompts.py`) to provide the LLM with repository-specific context.
4. **Streaming Response**: Generates and streams the answer back to the client in real-time.

Sources: [api/websocket_wiki.py:63-131](api/websocket_wiki.py#L63-L131), [api/rag.py:416-435](api/rag.py#L416-L435)

The entire pipeline ensures that the generated wiki content or chat responses are grounded in the actual implementation of the repository, providing high-fidelity technical insights.

Sources: [api/data_pipeline.py:434-458](api/data_pipeline.py#L434-L458)

---

## 04. The Multi-Provider Adapter Layer

> Exploration of the BYOC (Bring Your Own Client) architecture, supporting Google Gemini, OpenAI, Azure, OpenRouter, and local Ollama instances.

- Page Markdown: https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/04-the-multi-provider-adapter-layer.md
- Generated: 2026-05-19T19:07:26.054Z

### Source Files

- `api/openai_client.py`
- `api/google_embedder_client.py`
- `api/azureai_client.py`
- `api/bedrock_client.py`
- `api/openrouter_client.py`
- `api/dashscope_client.py`
- `api/ollama_patch.py`

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [api/config.py](api/config.py)
- [api/openai_client.py](api/openai_client.py)
- [api/openrouter_client.py](api/openrouter_client.py)
- [api/azureai_client.py](api/azureai_client.py)
- [api/bedrock_client.py](api/bedrock_client.py)
- [api/google_embedder_client.py](api/google_embedder_client.py)
- [api/dashscope_client.py](api/dashscope_client.py)
- [api/ollama_patch.py](api/ollama_patch.py)
</details>

# The Multi-Provider Adapter Layer

The Multi-Provider Adapter Layer is a core architectural component of the repository, implementing a **Bring Your Own Client (BYOC)** strategy. This layer abstracts the complexities of interacting with various Large Language Model (LLM) and embedding providers, offering a unified interface for the rest of the application. by leveraging the Adapter design pattern, the system can seamlessly switch between cloud providers like Google Gemini, OpenAI, Azure, and OpenRouter, as well as local instances via Ollama.

This modular architecture ensures that the application remains provider-neutral, allowing developers to deploy the system in diverse environments without significant code changes. The integration is built upon the `adalflow` framework, extending its `ModelClient` base class to provide specialized functionality for each supported platform.

Sources: [api/openai_client.py:43]()

## Core Abstraction: The ModelClient Interface

The foundation of the adapter layer is the `ModelClient` class from the `adalflow` library. Every provider-specific adapter must implement this interface, which standardizes how chat completions and embeddings are requested and processed.

### Supported Adapters

| Client Class | Provider | Primary Use Case |
| :--- | :--- | :--- |
| `OpenAIClient` | OpenAI | Chat & Embeddings |
| `GoogleEmbedderClient` | Google Cloud | Specialized Embeddings |
| `AzureAIClient` | Microsoft Azure | Enterprise OpenAI Instances |
| `BedrockClient` | AWS Bedrock | Enterprise Model Hosting |
| `OpenRouterClient` | OpenRouter | Unified API Gateway |
| `DashscopeClient` | Alibaba Cloud | Dashscope Models |
| `OllamaClient` | Local (Ollama) | Local Inference |

Sources: [api/config.py:58-67](), [api/openai_client.py:120](), [api/openrouter_client.py:19]()

## Configuration and Dynamic Dispatch

The application uses `api/config.py` to manage the lifecycle and selection of these adapters. It maps configuration strings to actual Python classes and handles the resolution of environment variables.

### Client Mapping
The `CLIENT_CLASSES` dictionary serves as the central registry for all available adapters. When a provider is specified in the configuration files (e.g., `embedder.json` or `generator_config`), the system uses this map to instantiate the correct client.

```python
# api/config.py:58-67
CLIENT_CLASSES = {
    "GoogleGenAIClient": GoogleGenAIClient,
    "GoogleEmbedderClient": GoogleEmbedderClient,
    "OpenAIClient": OpenAIClient,
    "OpenRouterClient": OpenRouterClient,
    "OllamaClient": OllamaClient,
    "BedrockClient": BedrockClient,
    "AzureAIClient": AzureAIClient,
    "DashscopeClient": DashscopeClient
}
```

Sources: [api/config.py:58-67](), [api/config.py:359-412]()

## Provider-Specific Enhancements

While the `ModelClient` provides a standard interface, individual adapters often include logic to handle provider-specific features or limitations.

### Local Ollama Integration
Local model support is enhanced via `api/ollama_patch.py`. Because the native `OllamaClient` does not always support batch embedding for all document types, the `OllamaDocumentProcessor` ensures that documents are processed individually with consistent embedding sizes.

### AWS Bedrock and Azure OpenAI
The `BedrockClient` and `AzureAIClient` are designed for enterprise environments, supporting complex authentication flows including AWS IAM roles and Azure-specific endpoint configurations.

Sources: [api/ollama_patch.py:62-105](), [api/azureai_client.py:118](), [api/bedrock_client.py:20]()

## Architecture Diagram

The following diagram illustrates how the configuration layer interacts with the Multi-Provider Adapter Layer to deliver requests to external services.

```mermaid
graph TD
    subgraph "Application Core"
        Config[api/config.py]
        API[api/api.py]
    end

    subgraph "Adapter Layer (BYOC)"
        Base[ModelClient Interface]
        OAI[OpenAIClient]
        GGL[GoogleEmbedderClient]
        AZR[AzureAIClient]
        OLM[OllamaClient + Patch]
        BDR[BedrockClient]
    end

    subgraph "External Providers"
        OpenAI_API((OpenAI API))
        Google_API((Google Cloud))
        Azure_API((Azure OpenAI))
        Local_Ollama((Local Ollama))
        AWS_API((AWS Bedrock))
    end

    Config -->|Resolves| Base
    Base -.-> OAI
    Base -.-> GGL
    Base -.-> AZR
    Base -.-> OLM
    Base -.-> BDR

    OAI --> OpenAI_API
    GGL --> Google_API
    AZR --> Azure_API
    OLM --> Local_Ollama
    BDR --> AWS_API

    API -->|Uses| Config
```

## Implementation Notes

- **Environment Variable Substitution**: The system supports `${VAR}` placeholders in configuration files, which are recursively replaced at runtime to protect sensitive API keys.
- **Error Handling**: Adapters like `OpenRouterClient` include complex retry logic and error generators to handle the diverse failure modes of aggregate API providers.
- **Refactoring TODOs**: The `AzureAIClient` currently contains significant overlap with `OpenAIClient`, with internal notes suggesting a future refactor to use subclassing for better code reuse.

In summary, the Multi-Provider Adapter Layer provides the necessary abstraction to ensure that the repository is truly provider-agnostic, enabling high portability and support for both cloud-native and local-first LLM deployments.

Sources: [api/config.py:69-97](), [api/openrouter_client.py:19](), [api/azureai_client.py:71]()

---

## 05. The Next.js Interface & Diagrams

> Architecture of the frontend application, focusing on real-time updates via WebSockets and automated Mermaid diagram rendering for repository visualization.

- Page Markdown: https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/05-the-next.js-interface-diagrams.md
- Generated: 2026-05-19T19:05:04.548Z

### Source Files

- `src/app/page.tsx`
- `src/components/Mermaid.tsx`
- `src/components/WikiTreeView.tsx`
- `src/components/ConfigurationModal.tsx`
- `src/app/globals.css`
- `src/app/layout.tsx`

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [src/app/page.tsx](src/app/page.tsx)
- [src/app/[owner]/[repo]/page.tsx](src/app/[owner]/[repo]/page.tsx)
- [src/components/Mermaid.tsx](src/components/Mermaid.tsx)
- [src/components/Markdown.tsx](src/components/Markdown.tsx)
- [src/components/WikiTreeView.tsx](src/components/WikiTreeView.tsx)
- [src/utils/websocketClient.ts](src/utils/websocketClient.ts)
- [src/app/globals.css](src/app/globals.css)
</details>

# The Next.js Interface & Diagrams

The **deepwiki-open** frontend is a high-performance Next.js application that provides a rich, interactive environment for exploring repository documentation. It features a unique Japanese-inspired aesthetic ("Wabi-sabi") characterized by warm paper textures, soft color palettes, and minimalist design. The interface is built for real-time engagement, utilizing WebSockets for streaming content generation and automated Mermaid diagram rendering to visualize complex codebase structures.

This architectural approach ensures that users can see the wiki take shape in real-time while providing deep insights into repository logic through dynamically generated visualizations.

## Frontend Architecture Overview

The application is structured using the Next.js App Router, with a clear separation between landing pages, repository-specific views, and shared components.

### Core Component Hierarchy

The interface is composed of several high-order components that manage state and rendering:

| Component | Responsibility |
| :--- | :--- |
| `RepoWikiPage` | The primary container for repository visualization, managing the wiki structure and page content state. |
| `WikiTreeView` | A hierarchical navigation sidebar that reflects the logical structure of the repository. |
| `Markdown` | The rendering engine for wiki pages, supporting GFM, math, and embedded diagrams. |
| `Mermaid` | A specialized component for rendering and interacting with Mermaid.js diagrams. |
| `Ask` | An AI-powered chat interface for querying the repository directly. |

Sources: [src/app/[owner]/[repo]/page.tsx:177-240](), [src/components/Markdown.tsx:16-196]()

## Real-time Communication via WebSockets

To provide a responsive "streaming" experience, the frontend utilizes WebSockets for communication with the backend agent. This replaces traditional polling or long-lived HTTP requests, allowing for instantaneous UI updates as the AI generates content.

### WebSocket Integration Pattern

The application uses a centralized `websocketClient` utility to manage connections. In `RepoWikiPage`, WebSockets are used for two primary tasks:
1.  **Structure Determination**: Analyzing the repository file tree and README to build the initial wiki hierarchy.
2.  **Content Generation**: Streaming the Markdown content for individual wiki pages.

```mermaid
sequenceDiagram
    participant UI as RepoWikiPage
    participant WS as WebSocketClient
    participant BE as Backend Agent
    
    UI->>WS: createChatWebSocket(request)
    WS->>BE: Open Connection (ws://.../ws/chat)
    BE-->>WS: Connection Established
    WS->>BE: Send JSON Request
    loop Streaming Content
        BE-->>WS: Message Chunk
        WS-->>UI: onMessage(data)
        UI->>UI: Update State (Live Preview)
    end
    BE-->>WS: Close Connection
    WS-->>UI: onClose()
```
Sources: [src/utils/websocketClient.ts:43-75](), [src/app/[owner]/[repo]/page.tsx:542-602]()

## Automated Mermaid Diagram Rendering

One of the standout features of deepwiki-open is its ability to automatically generate and render Mermaid diagrams. These diagrams are extracted from Markdown code blocks and rendered using a custom `Mermaid` component.

### Visualization Features

*   **Dynamic Theming**: Diagrams automatically switch between light and dark modes based on the application's global theme (e.g., using `--fuji` purple and `--washi` paper colors).
*   **Interactive Zooming**: Utilizing `svg-pan-zoom` and a custom `FullScreenModal`, users can inspect complex diagrams in detail.
*   **Japanese Aesthetic**: The `Mermaid` component injects custom CSS to ensure diagrams match the "Wabi-sabi" design system.

### Mermaid Component Logic

The `Mermaid` component handles the lifecycle of diagram rendering, from initialization to error handling.

```mermaid
stateDiagram-v2
    [*] --> Initializing: Chart Prop Received
    Initializing --> Rendering: mermaid.render()
    Rendering --> Success: SVG Generated
    Rendering --> Error: Syntax Error
    Success --> PanZoom: zoomingEnabled=true
    Success --> Clickable: zoomingEnabled=false
    Clickable --> FullScreen: On Click
    Error --> [*]
```
Sources: [src/components/Mermaid.tsx:6-170](), [src/components/Markdown.tsx:129-139]()

## Design System & Aesthetics

The interface adheres to a strict "Japanese Aesthetic" defined in `globals.css`. This system uses semantic CSS variables to maintain consistency across the application.

### Key Visual Tokens

| Token | Light Mode (`--washi`) | Dark Mode (`--charcoal`) |
| :--- | :--- | :--- |
| `--background` | `#f8f4e6` (Warm Paper) | `#1a1a1a` (Deep Charcoal) |
| `--accent-primary` | `#9b7cb9` (Fuji Purple) | `#9370db` (Soft Lavender) |
| `--border-color` | `#e0d8c8` (Soft Beige) | `#2c2c2c` (Dark Grey) |

Sources: [src/app/globals.css:6-32]()

The interface uses modern typography (Noto Sans JP and Geist Mono) and subtle paper textures to create a premium, calm reading experience that encourages deep exploration of code architecture.

Sources: [src/app/globals.css:39-68]()

## Summary

The **deepwiki-open** interface is a sophisticated Next.js application that leverages WebSockets for real-time reactivity and Mermaid.js for automated architectural visualization. By combining a unique Japanese design philosophy with robust engineering patterns, it transforms raw repository data into a beautiful, interactive knowledge base.

Sources: [src/app/[owner]/[repo]/page.tsx:177-280](), [src/components/Mermaid.tsx:306-487]()

---

## 06. Closing the Loop: DeepResearch & Q&A

> Understanding the final stage of user interaction: the "Ask" feature and DeepResearch module for multi-turn repository investigations.

- Page Markdown: https://grok-wiki.com/public/wiki/asyncfuncai-deepwiki-open-4d1f22320747/pages/06-closing-the-loop-deepresearch-q-a.md
- Generated: 2026-05-19T19:17:21.534Z

### Source Files

- `api/websocket_wiki.py`
- `api/simple_chat.py`
- `src/components/Ask.tsx`
- `Ollama-instruction.md`
- `api/logging_config.py`
- `pytest.ini`

<details>
<summary>Relevant source files</summary>
The following files were used as context for generating this wiki page:
- [api/websocket_wiki.py](api/websocket_wiki.py)
- [api/simple_chat.py](api/simple_chat.py)
- [src/components/Ask.tsx](src/components/Ask.tsx)
- [api/rag.py](api/rag.py)
- [api/prompts.py](api/prompts.py)
- [Ollama-instruction.md](Ollama-instruction.md)
</details>

# Closing the Loop: DeepResearch & Q&A

The final stage of user interaction in DeepWiki is centered around the **Ask** feature and the **DeepResearch** module. While the initial wiki generation provides a broad architectural overview, these features enable developers to perform surgical, multi-turn investigations into specific components, logic flows, or bug root causes. By "closing the loop," DeepWiki transforms from a static documentation generator into an active, conversational research partner.

This page explores the technical implementation of the retrieval-augmented Q&A engine and the autonomous multi-iteration research loop that powers complex repository investigations.

## The Q&A Engine: "Ask" Feature

The **Ask** feature provides a direct interface for querying the repository using natural language. It leverages a Retrieval-Augmented Generation (RAG) pipeline to ground every answer in the actual source code, ensuring technical accuracy and citing specific files.

### RAG Integration and Retrieval
When a user submits a query, the backend utilizes the `RAG` component to identify the most relevant code snippets. This process involves:
1.  **Vector Search**: Using `FAISSRetriever` to find documents whose embeddings align with the query.
2.  **Context Construction**: Grouping retrieved snippets by file path to provide structured context to the LLM.
3.  **Memory Management**: Maintaining a `DialogTurn` history to support follow-up questions within the same session.

Sources: [api/rag.py:153-243](api/rag.py#L153-L243), [api/websocket_wiki.py:192-234](api/websocket_wiki.py#L192-L234)

### Streaming and WebSockets
To provide a responsive user experience, the Ask feature uses real-time streaming. The primary communication channel is WebSockets, which allows the AI to stream its reasoning and final answer as it is generated. An HTTP streaming fallback is implemented in `api/simple_chat.py` for environments where WebSockets are restricted.

Sources: [api/websocket_wiki.py:63-131](api/websocket_wiki.py#L63-L131), [src/components/Ask.tsx:578-620](src/components/Ask.tsx#L578-L620)

## DeepResearch: Multi-Turn Investigations

The **DeepResearch** module is an advanced mode that automates multi-iteration investigations. Instead of a single-shot answer, it executes a structured research loop to "dive deep" into complex topics.

### The Iterative Research Loop
DeepResearch follows a programmed progression, typically spanning up to 5 iterations. Each iteration is guided by a specific system prompt tailored to the research stage:

1.  **Research Plan**: The first iteration identifies key aspects to investigate and outlines an approach.
2.  **Research Updates**: Intermediate iterations (2-4) explore gaps, verify findings, and provide new technical insights.
3.  **Final Conclusion**: The final iteration synthesizes all previous findings into a comprehensive, definitive answer.

Sources: [api/prompts.py:60-151](api/prompts.py#L60-L151), [api/websocket_wiki.py:258-357](api/websocket_wiki.py#L258-L357)

### Autonomous Continuation logic
The frontend component (`Ask.tsx`) manages the state of the research process. It detects when the AI has finished an iteration and, if the research is not yet complete, automatically triggers the next iteration by sending a hidden `[DEEP RESEARCH] Continue the research` prompt to the backend.

```mermaid
sequenceDiagram
    participant User
    participant UI as Ask.tsx
    participant API as websocket_wiki.py
    participant LLM as LLM Provider

    User->>UI: Enable DeepResearch & Ask Question
    UI->>API: [DEEP RESEARCH] Question
    API->>LLM: Apply "Research Plan" Prompt
    LLM-->>UI: Streaming "## Research Plan"
    Note over UI: Check completion markers
    UI->>API: [DEEP RESEARCH] Continue
    API->>LLM: Apply "Research Update" Prompt
    LLM-->>UI: Streaming "## Research Update"
    Note over UI: Repeat up to Iteration 5
    UI->>API: [DEEP RESEARCH] Continue (Final)
    API->>LLM: Apply "Final Conclusion" Prompt
    LLM-->>UI: Streaming "## Final Conclusion"
```
Sources: [src/components/Ask.tsx:281-403](src/components/Ask.tsx#L281-L403), [src/components/Ask.tsx:483-498](src/components/Ask.tsx#L483-L498)

## Architecture and Data Flow

The Ask and DeepResearch features are designed to be provider-neutral, supporting local models like Ollama alongside cloud providers such as Google Gemini, OpenAI, and AWS Bedrock.

### Provider Neutrality
The system abstracts model interactions through a modular client architecture. Users can switch between providers in the UI, and the backend handles the specific API requirements (token counting, context window management, and response parsing) for each.

| Component | Responsibility | Key Files |
| :--- | :--- | :--- |
| **Frontend UI** | State management, research navigation, and markdown rendering. | [Ask.tsx](src/components/Ask.tsx) |
| **Orchestration** | Handling WebSocket/HTTP sessions and DeepResearch iteration logic. | [websocket_wiki.py](api/websocket_wiki.py) |
| **RAG Pipeline** | Embedding generation and document retrieval from FAISS. | [rag.py](api/rag.py) |
| **Prompt Library** | Standardized templates for Q&A and DeepResearch stages. | [prompts.py](api/prompts.py) |

Sources: [api/websocket_wiki.py:440-562](api/websocket_wiki.py#L440-L562), [Ollama-instruction.md:135-152](Ollama-instruction.md#L135-L152)

## Summary

The Q&A and DeepResearch modules provide the final, most granular layer of the DeepWiki investigation process. By combining a robust RAG pipeline with an autonomous multi-turn research loop, DeepWiki enables developers to transition from high-level understanding to deep technical mastery of a codebase. The system's provider-neutral architecture ensures this capability is accessible whether running locally with Ollama or leveraging massive cloud context windows.

Sources: [api/websocket_wiki.py:156-188](api/websocket_wiki.py#L156-L188), [src/components/Ask.tsx:712-723](src/components/Ask.tsx#L712-L723)

---