# The Plugs & Sockets (Database, Cache & Files) > Why OpenWA is like a modular power strip: how it lets you plug in SQLite or PostgreSQL for data, local disks or S3 for files, and RAM or Redis for caching without rewiring the code. - Repository: rmyndharis/OpenWA - GitHub: https://github.com/rmyndharis/OpenWA - Human wiki: https://grok-wiki.com/public/wiki/rmyndharis-openwa-2c9996a09a22 - Complete Markdown: https://grok-wiki.com/public/wiki/rmyndharis-openwa-2c9996a09a22/llms-full.txt ## Source Files - `src/common/storage/storage.service.ts` - `src/common/cache/cache.service.ts` - `src/modules/infra/infra.controller.ts` - `src/config/configuration.ts` ---
Relevant source files The following files were used as context for generating this wiki page: - [src/common/storage/storage.service.ts](src/common/storage/storage.service.ts) - [src/common/cache/cache.service.ts](src/common/cache/cache.service.ts) - [src/modules/infra/infra.controller.ts](src/modules/infra/infra.controller.ts) - [src/config/configuration.ts](src/config/configuration.ts) - [src/app.module.ts](src/app.module.ts)
# The Plugs & Sockets (Database, Cache & Files) Imagine you have a premium modular power strip. Depending on what country you are in, or how much power you need, you can plug in different kinds of plugs. You don't rebuild your house's electrical wiring just because you plugged in a laptop charger instead of a lamp. OpenWA is designed exactly like this modular power strip. It divides the complex job of running a WhatsApp automation server into three distinct "sockets": **Database (Data Persistence)**, **Cache (Speed & Session Status)**, and **Files (Media Storage)**. Through simple environment variables, you can plug in high-performance enterprise systems like PostgreSQL, Redis, and Amazon S3 when deploying in production, or stick to simple, zero-configuration local options like SQLite, local memory, and your local hard drive when running on your personal computer—all without changing a single line of the server's codebase. ```mermaid flowchart TD subgraph Client ["Client Interface & REST API"] IC["infra.controller.ts
(Infrastructure API & Migrations)"] end subgraph Logic ["Application Layer"] appModule["app.module.ts
(Dynamic Modules)"] storageService["storage.service.ts
(Agnostic Streams)"] cacheService["cache.service.ts
(Resilient Connections)"] end subgraph Persistence ["Sockets: Plug-and-Play Infrastructure"] subgraph Database ["Database Socket"] sqliteDB[("Local SQLite
(main.sqlite / openwa.sqlite)")] postgresDB[("PostgreSQL
(External/Built-in Container)")] end subgraph Caching ["Cache Socket"] ramCache["Memory Cache
(Graceful Fallback)"] redisCache[("Redis Cache
(ioredis Client)")] end subgraph Storage ["File Storage Socket"] localDisk["Local File System
(./data/media)"] s3Bucket[("S3 / MinIO Cloud Storage
(AWS S3 Client)")] end end IC --> appModule IC --> storageService IC --> cacheService appModule -.-> |"TypeOrmModule (data connection)"| sqliteDB appModule -.-> |"TypeOrmModule (data connection)"| postgresDB storageService -.-> |"STORAGE_TYPE = local"| localDisk storageService -.-> |"STORAGE_TYPE = s3"| s3Bucket cacheService -.-> |"REDIS_ENABLED = false (Fallback)"| ramCache cacheService -.-> |"REDIS_ENABLED = true"| redisCache ``` --- ## Pluggable Data Persistence (Database) OpenWA maintains a clever separation of concerns using two distinct database connections inside NestJS: 1. **The Main Boot Database (`main`)**: This connection is dedicated strictly to low-write credentials (like API keys) and security logs. It *always* uses SQLite (`./data/main.sqlite`), ensuring that the system can boot and authenticate users even if external network systems are temporarily down. 2. **The Data Storage Database (`data`)**: This is the pluggable "socket." It houses all dynamic, high-churn application data: session records, active webhooks, messages, and message batches. Depending on the `DATABASE_TYPE` environment variable, NestJS instantiates the correct connector: * **SQLite (Local Development)**: Highly portable and file-based (`./data/openwa.sqlite`). To make the newcomer experience effortless, SQLite is set to `synchronize: true`, meaning the database tables are auto-generated on first boot without any manual migration commands. * **PostgreSQL (Production)**: When scaled up, it connects to an external or built-in Postgres database using connection pooling. To guarantee data integrity in production, auto-synchronization is strictly disabled (`synchronize: false`). Instead, OpenWA automatically detects and runs safe migration scripts sequentially on startup (`migrationsRun: true`) to preserve production safety. ```typescript // From src/app.module.ts if (dbType === 'postgres') { return { ...baseConfig, type: 'postgres' as const, host: configService.get('dataDatabase.host'), port: configService.get('dataDatabase.port'), username: configService.get('dataDatabase.username'), password: configService.get('dataDatabase.password'), database: 'openwa', // Never auto-sync Postgres in production; rely on migrations. synchronize: configService.get('dataDatabase.synchronize', false), migrationsRun: true, retryAttempts: 10, retryDelay: 3000, extra: { max: configService.get('dataDatabase.poolSize', 10), }, }; } ``` Sources: [src/app.module.ts:49-110](), [src/config/configuration.ts:21-46]() --- ## Swappable Cache & Session Store (Cache) WhatsApp automation demands extremely fast reads and writes. For instance, the system needs to immediately fetch whether a session is `CONNECTED` or check an active QR code stream. Reading this from a persistent database every second would create a heavy performance bottleneck. The `CacheService` acts as the cache socket. It checks if `REDIS_ENABLED` is set to `true`: * **Redis Mode**: If enabled, the service uses `ioredis` to manage highly efficient key-value lookups in a dedicated Redis container. The service features built-in resilience: it will attempt to connect up to 5 times (`maxConnectionAttempts = 5`), rate-limits reconnect attempts to protect network sockets, and handles connections lazily so a slow-starting container won't crash the server. * **RAM Fallback (Offline Mode)**: If `REDIS_ENABLED` is `false`, the cache layer acts as a silent, non-blocking pass-through. Instead of throwing runtime errors, it gracefully returns `null` or skips writing cache keys, allowing the server to continue running directly off the database. Each stored cache item relies on strict Time-To-Live (TTL) constants: ```typescript // From src/common/cache/cache.service.ts const TTL = { SESSION_STATUS: 300, // 5 min SESSION_INFO: 600, // 10 min SESSION_QR: 60, // 1 min SESSIONS_LIST: 30, // 30 sec SESSIONS_STATS: 15, // 15 sec }; ``` These values ensure that highly volatile data (like QR codes and statistics) is refreshed frequently, while static details (like session info) persist longer, maximizing response speed. Sources: [src/common/cache/cache.service.ts:21-48](), [src/common/cache/cache.service.ts:119-128](), [src/config/configuration.ts:4-19]() --- ## Flexible File Management (Files) When automation bots receive images, documents, or voice notes from WhatsApp, they must be stored reliably. The `StorageService` abstracts this under a unified interface, supporting two pluggable backends selected via `STORAGE_TYPE`: * **Local Disk (`local`)**: Media files are stored inside a local folder structure (defaults to `./data/media`). The service automatically handles path routing, recursive directory creation (`fs.mkdirSync`), and reads/writes using standard Node.js `fs` streams. * **AWS S3 or MinIO (`s3`)**: In containerized or distributed deployments, media files are pushed directly to an S3-compatible cloud storage bucket. On boot, `StorageService` runs a `HeadBucketCommand` via the AWS S3 SDK. If the target bucket doesn't exist, the system creates it automatically (`CreateBucketCommand`), offering a fully hands-free setup. ```typescript // From src/common/storage/storage.service.ts async putFile(filePath: string, data: Buffer): Promise { if (this.storageType === 's3' && this.s3Client && this.s3Available) { return this.putS3File(filePath, data); } return this.putLocalFile(filePath, data); } ``` ### Portability & Tarball Migrations A major risk of using different file stores is vendor lock-in. OpenWA solves this by incorporating a high-performance backup/migration system directly inside the `StorageService`. Using the `archiver` and `tar-stream` libraries, the service can compress all files from the active storage backend (whether S3 or local disk) into a gzip-compressed tarball stream (`.tar.gz`), or extract an incoming stream and write files to the active backend. Sources: [src/common/storage/storage.service.ts:36-67](), [src/common/storage/storage.service.ts:112-124](), [src/common/storage/storage.service.ts:148-220](), [src/config/configuration.ts:80-91]() --- ## The Management Console (Infrastructure Control API) Connecting these sockets is the `InfraController`, a dedicated dashboard API which acts as the main system plugboard. It allows developers to check status, adjust environment settings, backup databases, or even trigger service restarts on the fly: * **System Health Monitor (`GET /infra/status`)**: Aggregates connectivity states across all modules—reporting whether the primary and secondary databases are initialized, if Redis is reachable, the active storage engine type, and Puppeteer engine details. * **Configuration Writer (`PUT /infra/config`)**: Receives payload data containing new credentials and builds a formatted `.env.generated` file in the persistent `/data` directory, persisting settings across container lifecycles. * **Service Orchestration (`POST /infra/restart`)**: Communicates with the local Docker daemon via `DockerService`. If profiles have changed (e.g., PostgreSQL enabled, SQLite disabled), it orchestrates starting new service containers, removing unused ones, and scheduling a clean graceful shutdown (`shutdownService.shutdown(3000)`) so the app can reboot with the new profiles. * **Neutral Data Portability (`/infra/export-data` and `/infra/import-data`)**: To migrate data between SQLite and PostgreSQL, this controller performs a complete transaction-safe export/import of all user tables (`sessions`, `webhooks`, `messages`, `message_batches`) using plain environment-neutral JSON. Sources: [src/modules/infra/infra.controller.ts:152-190](), [src/modules/infra/infra.controller.ts:206-425](), [src/modules/infra/infra.controller.ts:438-664]() --- ## Configuration Reference Matrix To easily plug and unplug services, use the following environment variable matrix: | Socket Type | Environment Variable | Accepted Values | Default Value | Description | | :--- | :--- | :--- | :--- | :--- | | **Database** | `DATABASE_TYPE` | `sqlite`, `postgres` | `sqlite` | Selects which persistence connector to load for user data. | | **Database** | `POSTGRES_BUILTIN` | `true`, `false` | `false` | Instructs the orchestrator to spin up the local PostgreSQL container. | | **Database** | `DATABASE_SYNCHRONIZE` | `true`, `false` | `false` | Toggles automatic schema sync (enabled by default for local SQLite). | | **Cache** | `REDIS_ENABLED` | `true`, `false` | `false` | Activates Redis connection for queueing and status caching. | | **Cache** | `REDIS_BUILTIN` | `true`, `false` | `false` | Launches a built-in Redis container via the docker profile orchestrator. | | **Files** | `STORAGE_TYPE` | `local`, `s3` | `local` | Directs how media attachments are stored (hard drive vs cloud). | | **Files** | `MINIO_BUILTIN` | `true`, `false` | `false` | Spins up a built-in local S3-compatible MinIO object store container. | | **Files** | `STORAGE_LOCAL_PATH` | File System Path | `./data/media` | Target folder for media file writes in `local` mode. | | **Files** | `S3_ENDPOINT` | URL | *None* | Connection endpoint for S3 / MinIO storage APIs. | Sources: [src/config/configuration.ts:4-46](), [src/config/configuration.ts:80-91](), [src/modules/infra/infra.controller.ts:206-336]() --- ## Summary & Design Philosophy OpenWA's architecture treats third-party systems as secondary adapters rather than foundational dependencies. By strictly wrapping databases behind NestJS data-source scopes, caching under connection-resilient abstraction services, and file I/O behind path-agnostic streams, the application remains incredibly portable, developer-friendly, and simple to maintain. This design guarantees that whether you are developing offline on a train or scaling to millions of messages in a cloud cluster, your software remains exactly the same. Sources: [src/app.module.ts:62-110](), [src/common/cache/cache.service.ts:39-48](), [src/common/storage/storage.service.ts:36-67]()