# DDL staging > .create/.modify/.delete staging dirs, sql/.test/.commit control files, index DDL via .indexes/, grace period, and errno on validation failures. - Repository: timescale/tigerfs - GitHub: https://github.com/timescale/tigerfs - Human docs: https://grok-wiki.com/public/docs/timescale-tigerfs-60719456a5c3 - Complete Markdown: https://grok-wiki.com/public/docs/timescale-tigerfs-60719456a5c3/llms-full.txt ## Source Files - `internal/tigerfs/fs/ddl.go` - `internal/tigerfs/fs/staging.go` - `internal/tigerfs/fuse/control_files.go` - `docs/adr/003-ddl-staging-pattern.md` - `test/integration/ddl_test.go` - `docs/spec.md` --- --- title: "DDL staging" description: ".create/.modify/.delete staging dirs, sql/.test/.commit control files, index DDL via .indexes/, grace period, and errno on validation failures." --- TigerFS applies schema changes through in-memory DDL staging sessions exposed as directories under `/.create/`, `//.modify/`, `/
/.delete/`, and related paths. Each session holds staged SQL plus trigger files (`sql`, `.test`, `test.log`, `.commit`, `.abort`); `fs.DDLManager` validates with `BEGIN`/`ROLLBACK` and commits with a real `Exec`, while FUSE and NFS adapters map `fs.FSError` codes to POSIX errno and log hints to stderr. DDL staging is not row staging. `StagingManager` in `internal/tigerfs/fs/staging.go` tracks partial `INSERT`s when you `mkdir` a new primary-key directory under a table. DDL staging uses `DDLManager` in `internal/tigerfs/fs/ddl.go` and only affects schema objects. ## Supported operations and paths All DDL types share the same control-file workflow (ADR-003). Path resolution is `PathDDL` in `internal/tigerfs/fs/path.go`. | Object | Create | Modify | Delete | |--------|--------|--------|--------| | Table | `/.create//` | `/
/.modify/` | `/
/.delete/` | | Index | `/
/.indexes/.create//` | — | `/
/.indexes//.delete/` | | Schema | `/.schemas/.create//` | — | `/.schemas//.delete/` | | View | `/.views/.create//` | — | `/.views//.delete/` | Existing indexes also expose read-only metadata at `/
/.indexes//.schema` (generated `CREATE INDEX` DDL, not a staging directory). ```text /mnt/db/ ├── .create/orders/ sql .test test.log .commit .abort ├── users/ │ ├── .modify/ (auto-session on first access) │ ├── .delete/ │ └── .indexes/ │ ├── .create/email_idx/ │ └── email_idx/.delete/ ├── .schemas/.create/app/ └── .views/.create/active_users/ ``` ## Control files | File | Type | Read | Write / touch | |------|------|------|----------------| | `sql` | Content | Staged SQL, or generated template if empty | Stage DDL; resets validation | | `.test` | Trigger | Empty | Run `ExecInTransaction` (rolled back) | | `test.log` | Content | Last validation output | Read-only (`ENOENT` until first test) | | `.commit` | Trigger | Empty | Execute staged SQL | | `.abort` | Trigger | Empty | Mark session completed (cancel) | Trigger files must be opened for **write** (for example `touch` or `os.Create`), not read-only open — integration tests document that read-only open does not fire the write path. Comment stripping uses `ExtractSQL` / `IsEmptyOrCommented`: lines starting with `--` and block comments `/* … */` are removed before test or commit. Comment-only `sql` is rejected at test/commit with a message in `test.log` or commit error text. ## Workflow For **new** root-level objects (`/.create//`, `/.schemas/.create/`, `/.views/.create/`, index create under `.indexes/.create/`), run `mkdir`: ```bash mkdir /mnt/db/.create/orders mkdir /mnt/db/users/.indexes/.create/email_idx ``` For **modify**, **delete**, and index delete paths on existing objects, TigerFS **auto-creates** a session on first `ls`, `stat`, or read of `sql` — no `mkdir` required. ```bash cat /mnt/db/.create/orders/sql # template with commented examples vi /mnt/db/.create/orders/sql # or echo/redirection ``` Writing `sql` clears the `Validated` flag. Vim/emacs swap files are stored in-memory on the session (`ExtraFiles`), not on disk. ```bash touch /mnt/db/.create/orders/.test cat /mnt/db/.create/orders/test.log ``` A failed validation updates `test.log` but the touch itself **succeeds** at the filesystem layer (no errno for bad SQL). Check `test.log` and stderr JSON logs (`logging.Warn` / `logging.Info`). ```bash touch /mnt/db/.create/orders/.commit # applies DDL, invalidates metadata cache # or touch /mnt/db/.create/orders/.abort # cancels; sql reverts to template on read ``` On commit failure the session stays active so you can fix `sql` and retry. On success the session is marked `Completed` and kept in memory for the grace period. Script-style one-liner (no `.test`): ```bash mkdir /mnt/db/.create/orders && \ echo 'CREATE TABLE orders (id serial PRIMARY KEY)' > /mnt/db/.create/orders/sql && \ touch /mnt/db/.create/orders/.commit ``` ## Session lifecycle and grace period ```mermaid stateDiagram-v2 [*] --> Active: mkdir or auto-create Active --> Active: write sql Active --> Active: touch .test Active --> Completed: touch .commit success Active --> Completed: touch .abort Completed --> Reaped: grace period elapsed Reaped --> [*] Active --> Active: commit failure ``` | State | Behavior | |-------|----------| | Active | SQL and control files fully functional | | Completed | After successful commit or abort; trigger touches are **no-ops** | | Reaped | Lazy removal on `ListSessionEntries` / `FindSessionByName` after grace elapsed | How long completed sessions remain visible so NFS/FUSE post-close `stat`/`readdir` do not fail. Set in `~/.config/tigerfs/config.yaml` or `TIGERFS_DDL_GRACE_PERIOD`. Zero means use the 30s default (`config.DDLGracePeriod` → `NewDDLManager`). Staged SQL lives **in memory per mount**. Unmount drops all sessions. This is intentional — no stale DDL on disk. ## Creating sessions: mkdir rules | Situation | errno | Hint (stderr) | |-----------|-------|----------------| | Duplicate active session | `EEXIST` | Abort existing session first | | Empty name `/.create/` | `EINVAL` | Use `mkdir /.create/` | | Unknown operation in path | `EINVAL` | Malformed DDL path | | Access `sql` without session (root create) | `ENOENT` | `mkdir /.create/` | Replacing a **completed** root-level create session: `mkdir` removes the old session and starts fresh. Completed table-level auto-sessions are replaced silently on next access. ## errno and logging on validation failures FUSE/NFS only return errno codes; details go to **stderr** via structured zap logs (`logging.Error` with `hint`). | Operation | Condition | errno | Where detail appears | |-----------|-----------|-------|-------------------| | Touch `.test` | Bad SQL | *(success)* | `test.log`, stderr warn | | Touch `.test` | No / comment-only SQL | *(success)* | `test.log` | | Read `test.log` | Before any test | `ENOENT` | Hint: run `touch …/.test` | | Touch `.commit` | Empty / comment-only SQL | `EIO` | stderr + error message | | Touch `.commit` | PostgreSQL error | `EIO` | stderr; session preserved | | Write `sql` | Session already completed | `EPERM` | Hint: new `mkdir` | | `mkdir` duplicate active | `EEXIST` | stderr hint | | `rm` control file or staging dir | — | `EACCES` | Use `.abort` | | Read `test.log` open for write | — | `EACCES` | Read-only file | Do not infer validation success from the exit code of `touch .test`. Always read `test.log` or stderr. Commit failures use `EIO` — read TigerFS logs when the shell only prints `Input/output error`. Example stderr after failed commit: ```json {"message":"DDL commit failed for create/orders","hint":"session=… error: …"} ``` ## Index DDL Indexes use the same control files under the owning table: ```bash mkdir /mnt/db/users/.indexes/.create/email_idx echo 'CREATE INDEX email_idx ON users (email)' > /mnt/db/users/.indexes/.create/email_idx/sql touch /mnt/db/users/.indexes/.create/email_idx/.test cat /mnt/db/users/.indexes/.create/email_idx/test.log touch /mnt/db/users/.indexes/.create/email_idx/.commit # Drop echo 'DROP INDEX email_idx' > /mnt/db/users/.indexes/email_idx/.delete/sql touch /mnt/db/users/.indexes/email_idx/.delete/.commit ``` `DDLParentTable` is set from path context so templates reference the correct table. Primary key indexes are omitted from `.indexes/` listings (row paths remain the access model). ## Implementation map | Layer | Role | |-------|------| | `fs/path.go` | Parses `PathDDL`, `DDLOp`, `DDLFile`, object type | | `fs/ddl.go` | `DDLManager`: sessions, test, commit, templates, grace reaper | | `fs/write.go` | `mkdirDDL`, `writeDDLFile`, metadata cache invalidation on commit | | `fs/operations.go` | `ensureDDLSession`, directory listings, stable mtimes | | `fuse/adapter.go` | `ErrorToErrno` + hint logging | | `fuse/control_files.go` | Legacy FUSE-only nodes when `legacy_fuse: true` | Default mounts route through `fs.Operations`; both Linux FUSE and macOS NFS use the same DDL semantics. ## Templates Reading `sql` before writing returns object-specific commented templates (`generateCreateTemplate`, `generateModifyTemplate`, `generateDeleteTemplate` in `ddl.go`). Templates encourage uncommenting real statements rather than executing comments. ## Verification Integration coverage in `test/integration/ddl_test.go` exercises table create/delete, `ALTER` via `.modify`, index and schema cycles, view create/delete, valid/invalid `.test`, abort clearing staged content, and script workflow. ```bash go test -run TestDDL_ ./test/integration/... go test -run 'TestMkdir_PathDDL|TestWriteDDLFile' ./internal/tigerfs/fs/... ``` After create/delete, allow a short pause (or cache TTL) before `stat` on new table paths — tests sleep ~500ms for metadata visibility. ## Related pages Path model, dot-directories, and how `fs.Operations` maps paths to SQL. `fs.ErrorCode` to errno mapping and reading stderr hints. `ddl_grace_period` and mount-level settings. Build, `go test`, and integration testcontainers workflow.