# Troubleshooting

> Stale mounts, force unmount, NFS monotonicity warnings in stress runs, logging --debug, .refresh metadata clear, and common errno recovery.

- Repository: timescale/tigerfs
- GitHub: https://github.com/timescale/tigerfs
- Human docs: https://grok-wiki.com/public/docs/timescale-tigerfs-60719456a5c3
- Complete Markdown: https://grok-wiki.com/public/docs/timescale-tigerfs-60719456a5c3/llms-full.txt

## Source Files

- `internal/tigerfs/cmd/unmount.go`
- `internal/tigerfs/cmd/stop.go`
- `test/stress/README.md`
- `internal/tigerfs/logging/logging.go`
- `docs/spec.md`
- `CHANGELOG.md`

---

---
title: "Troubleshooting"
description: "Stale mounts, force unmount, NFS monotonicity warnings in stress runs, logging --debug, .refresh metadata clear, and common errno recovery."
---

TigerFS surfaces failures through POSIX errno on the mount, structured JSON logs on stderr, and a JSON mount registry under `~/.config/tigerfs/mounts.json`. Most operational issues—stale registry rows, busy unmounts, metadata that lags external DDL, stress-run NFS warnings, and opaque `EINVAL`/`EACCES` from shell tools—are diagnosed from those three channels plus the `tigerfs` lifecycle commands (`status`, `list`, `unmount`, `stop`).

## Symptom map

| Symptom | First checks | Typical fix |
|---------|--------------|-------------|
| `tigerfs status` shows a mount you killed | Registry PID stale | `tigerfs status` or `list` (auto-`Cleanup`) |
| `unmount: device is busy` | Open files under mountpoint | Close processes, then `tigerfs unmount --force` |
| `ls` missing a new table | Metadata cache TTL | Wait for `metadata_refresh_interval`, or remount |
| Tool reports `EINVAL` with no detail | TigerFS stderr | Re-run with `--log-level debug` |
| `[warn iter N] readLatestLogID regressed` | Stress run on macOS NFS | Expected; see monotonicity section |

```text
  shell tool (mv, cat, ls)
        │
        ▼
  FUSE / NFS adapter  ──► errno (ENOENT, EINVAL, …)
        │
        ▼
  fs.Operations       ──► FSError + zap hint on stderr
        │
        ▼
  PostgreSQL
```

## Stale mount registry entries

The mount registry records mountpoint, PID, database label, and start time. Entries survive crashes when TigerFS exits without `unmount` or `stop`.

`Register` replaces duplicate mountpoints when a prior process died without cleanup. `Cleanup()` removes entries whose PID is no longer running. `ListActive()` returns only live processes.

| Command | Registry behavior |
|---------|-------------------|
| `tigerfs status` | Calls `Cleanup()`, then lists active mounts |
| `tigerfs list` | Same cleanup; prints mountpoints only (scripting) |
| `tigerfs status /path/to/mount` | Shows `active` or `stale` for that entry’s PID |
| `tigerfs unmount MOUNT` | Unmounts, then `Unregister` (best-effort) |

<Steps>
<Step title="Confirm registry vs reality">

```bash
tigerfs status
# or
tigerfs status /mnt/db
```

`active` means the PID responds to a signal probe. `stale` means the registry row outlived the process.

</Step>
<Step title="Refresh the registry">

```bash
tigerfs list    # cleanup + active mountpoints
tigerfs status  # human-readable table
```

Both commands invoke `registry.Cleanup()` before listing.

</Step>
<Step title="Remove a ghost mountpoint path">

If the directory still exists but nothing is mounted:

```bash
# macOS
diskutil unmount force /mnt/db 2>/dev/null || true
# Linux FUSE
fusermount -uz /mnt/db 2>/dev/null || true
```

`unmount` treats “not currently mounted” / “not mounted” as success.

</Step>
</Steps>

<Note>
A stale registry row does not mean the filesystem is still mounted. Check the mount with `mount | grep tigerfs` (Linux) or `mount` (macOS) before force-unmounting.
</Note>

## Force unmount and stuck mounts

`unmount` tries graceful shutdown first: look up the registry entry, send `SIGTERM`, poll every 100ms until the context timeout (default 30s), then fall back to OS unmount.

| Platform | Normal | Force (`-f` / `--force`) |
|----------|--------|---------------------------|
| macOS | `umount MOUNT` | `diskutil unmount force MOUNT` |
| Linux | `fusermount -u MOUNT` | `fusermount -uz MOUNT` (lazy) |
| Other | `umount MOUNT` | `umount -f MOUNT` |

<ParamField body="timeout" type="int" default="30">
Seconds to wait for the TigerFS process to exit after `SIGTERM` before system unmount.
</ParamField>

<RequestExample>

```bash
# Graceful (registry SIGTERM, then system unmount)
tigerfs unmount /mnt/db

# Busy mountpoint
tigerfs unmount --force /mnt/db

# Known PID instead of path
tigerfs stop 12345
```

</RequestExample>

`stop PID` sends `SIGTERM`, waits for exit, and unregisters the mountpoint if found in the registry. Use when you have the PID from `tigerfs status` but not the path.

<Warning>
Force unmount can strand open file handles in client processes. Close editors, shells with `cd` into the mount, and long-running jobs before `--force`.
</Warning>

### Stress-test teardown

`tigerfs-stress` uses the same platform rules: SIGTERM with a 5s grace window, then `Kill`, then `diskutil unmount force` (macOS) or `umount` (Linux). For a stuck stress run from another terminal:

```bash
bin/tigerfs-stress stop
# Docker FUSE mode (match info file env):
TIGERFS_STRESS_INFO_FILE=/out/tigerfs-stress.info \
  docker compose -f test/stress/docker/docker-compose.yml \
  exec -e TIGERFS_STRESS_INFO_FILE=/out/tigerfs-stress.info \
  stress /usr/local/bin/tigerfs-stress stop
```

## Debug logging (`--log-level debug`)

FUSE and NFS adapters return errno only. TigerFS logs the underlying reason (and optional `hint` fields) to stderr before returning.

| Level | Config / flag | Output style |
|-------|---------------|--------------|
| `warn` (default) | `log_level: warn` | Minimal production JSON |
| `debug` | `--log-level debug` | Development: colors, timestamps, caller |

<RequestExample>

```bash
# Foreground mount with full diagnostics
tigerfs mount --foreground --log-level debug postgres://host/db /mnt/db

# Any subcommand inherits persistent flags
tigerfs --log-level debug status

# Config file or env (viper)
# ~/.config/tigerfs/config.yaml: log_level: debug
# export TIGERFS_LOG_LEVEL=debug
```

</RequestExample>

At `debug` or `info`, logging uses `zap.NewDevelopmentConfig` (colored levels, ISO8601 timestamps). At `warn` and `error`, output is compact production JSON with `message` as the primary key.

Stress runs pass debug through to the child TigerFS process:

```bash
bin/tigerfs-stress start --debug --iterations 10
# equivalent: --log-level debug on the tigerfs binary
```

TigerFS process stdout/stderr from stress land in `tigerfs.log` in the working directory.

<Tip>
Reproduce the failing operation with debug enabled on a foreground mount. Search stderr for `"hint"` — synth validation, DDL staging, and undo paths emit structured guidance there.
</Tip>

## Metadata cache refresh

Metadata (schemas, tables, views, PKs, permissions) is cached in-process with tiered TTLs. Row **content** is not cached; stale symptoms usually affect directory listings or stat sizes, not `cat` on a known path.

| Cache tier | Config key | Default TTL |
|------------|------------|-------------|
| Catalog (schemas, tables, views) | `metadata_refresh_interval` | 10s |
| Structural (row counts, permissions, PKs) | `structural_metadata_refresh_interval` | 5m |
| Stat / path (per-table listings) | Fixed in code | 2s |
| Undo preview | Fixed in code | 2s |

`MetadataCache.Invalidate()` clears all tiers and forces a DB refresh on next access. It runs after DDL commits, workspace build operations, and similar write paths.

### Manual clear via `.refresh`

`docs/spec.md` documents a mount-root control file:

```bash
echo 1 > /mnt/db/.refresh
```

Writing to `.refresh` is specified to clear cached metadata (table lists, columns, indexes, permissions). The active `fs.Operations` root listing currently exposes `.build`, `.create`, `.info`, `.schemas`, `.tables`, and `.views` — not `.refresh`. If `ls -a` at the mount root does not show `.refresh`, use the workarounds below instead of expecting that path to exist.

<Steps>
<Step title="Wait for catalog TTL">

After external `CREATE TABLE` / `DROP` outside TigerFS, new names appear after `metadata_refresh_interval` (default 10s) unless a TigerFS write path already called `Invalidate()`.

</Step>
<Step title="Tune refresh interval">

```yaml
# ~/.config/tigerfs/config.yaml
metadata_refresh_interval: 5s
structural_metadata_refresh_interval: 2m
```

Or `TIGERFS_METADATA_REFRESH_INTERVAL` / `TIGERFS_STRUCTURAL_METADATA_REFRESH_INTERVAL` where supported by your config loader.

</Step>
<Step title="Remount for a hard reset">

```bash
tigerfs unmount /mnt/db
tigerfs mount postgres://host/db /mnt/db
```

</Step>
</Steps>

<Info>
Direct file access (`cat /mount/table/row/col`) always queries PostgreSQL. If `cat` works but `ls` on the parent directory is wrong, suspect metadata or kernel directory cache (often 1–2s on FUSE), not row content staleness.
</Info>

## NFS monotonicity warnings (stress runs)

On macOS, TigerFS serves the mount through an in-process NFS v3 server (`go-nfs`). The stress harness (`bin/tigerfs-stress`) reads `.log/.last/N/.export/json` over NFS. After heavy undo operations, the macOS NFS client can return a **stale** snapshot where the newest `log_id` is older than one already observed (UUIDv7 ordering makes that impossible for a true latest value).

The runner wraps reads in `readLatestLogIDMonotonic`:

- Up to **25** retries, **100ms** apart (~2.5s budget)
- On recovery: stderr warning `[warn iter N] readLatestLogID regressed after …; recovered after …`
- On exhaustion: keeps the prior known-good `lastLogID` and logs that recovery failed

<Warning>
These warnings are **expected** during stress runs. They indicate NFS/kernel attribute caching below TigerFS, not application data loss. Do not treat them as TigerFS bugs to fix; the run continues with the last known-good log ID.
</Warning>

End-of-run summary (stdout):

```text
=== Monotonicity Warnings ===
Total:     N regressions across M ops (x.xx%)
Recovered: … / … (… kept prior lastLogID after retry exhaustion)
```

To compare NFS vs FUSE behavior, run the same seed natively on macOS (NFS) and via `./scripts/stress-docker.sh` (Linux FUSE in Docker).

| Observation | Interpretation |
|-------------|----------------|
| Warnings only on macOS native | NFS client / go-nfs path |
| Same seed fails validation | Use failure dump under `/tmp/tigerfs-stress-failure-*` or `test/stress/docker/host-out/` |
| Seed does not replay | NFS timing can diverge PRNG consumption; rely on dump artifacts |

## Common errno recovery

Shell tools show only errno. Map symptoms using stderr JSON and this table.

### `fs.ErrorCode` → POSIX (FUSE adapter)

| `FSError` code | errno | Typical cause |
|----------------|-------|----------------|
| `ErrNotExist` | `ENOENT` | Missing path, row, or staging entry |
| `ErrPermission` | `EACCES` | Grants, constraints, reserved dotfile |
| `ErrInvalidPath`, `ErrInvalidFormat`, `ErrInvalidArgument` | `EINVAL` | Bad path, malformed PATCH/JSON, synth numbering |
| `ErrInvalidOperation` | `EPERM` | Operation not allowed on path type |
| `ErrReadOnly` | `EROFS` | Read-only mount |
| `ErrNotEmpty` | `ENOTEMPTY` | `rmdir` on non-empty directory |
| `ErrAlreadyExists` | `EEXIST` | Create when path exists |
| `ErrIO`, `ErrInternal` | `EIO` | DB connection, timeout, unexpected SQL |
| `ErrNotImplemented` | `ENOSYS` | Unimplemented feature |

### PostgreSQL errors → errno (user-visible)

| Database condition | errno | Where detail lives |
|--------------------|-------|-------------------|
| Row not found | `ENOENT` | stderr at `debug` |
| Permission / NOT NULL / FK violation | `EACCES` | stderr `ERROR` with SQL detail |
| Connection refused / timeout | `EIO` | stderr + `tigerfs test-connection` |

<RequestExample>

```bash
# User sees only:
echo 'x' > /mnt/db/users/1/email
# email: Permission denied

# Check exit code
echo $?    # 13 = EACCES on Linux

# TigerFS stderr (run mount with --log-level debug):
# {"message":"NOT NULL constraint violation",...,"hint":"..."}
```

</RequestExample>

### Recovery patterns

| errno | Check | Action |
|-------|-------|--------|
| `ENOENT` | Path spelling, PK segment, pipeline depth | `ls` parent; read stderr hint for synth paths |
| `EINVAL` | Format suffix (`.json`, `.tsv`), task numbers, undo order | stderr `hint`; undo **newest first** |
| `EACCES` | Table grants, NOT NULL, reserved `.name` | Fix SQL or pick non-reserved filename |
| `EPERM` | Read-only mount, history path, DDL policy | `tigerfs mount` without `--read-only` |
| `EIO` | Network, `sslmode`, pool exhaustion | `tigerfs test-connection URL`; TLS flags |
| `EBUSY` (unmount) | Processes using mount | `lsof +D /mnt/db`; `unmount --force` |

<Note>
`ErrNotExist` is logged at **debug** level to avoid noise from tab completion and speculative `stat`. For missing-file investigations, always use `--log-level debug`.
</Note>

### Cross-mount consistency

If mount A wrote data and mount B still shows wrong directory entries within ~2s, suspect stat/path cache TTL or metadata catalog TTL—not stale row reads. Writes should call `statCache.invalidate(schema, table)` and `pathCache.invalidate(schema, table)` with the **user** schema (not internal `tigerfs` schema keys). A schema-key mismatch leaves stale cache entries for up to 2 seconds.

## Quick reference

| Goal | Command |
|------|---------|
| List live mounts | `tigerfs status` |
| Scriptable mountpoints | `tigerfs list` |
| Graceful stop | `tigerfs unmount /mnt/db` |
| Force stop | `tigerfs unmount -f /mnt/db` |
| Stop by PID | `tigerfs stop PID` |
| Verbose server logs | `tigerfs --log-level debug mount --foreground …` |
| Stress debug | `bin/tigerfs-stress start --debug` |
| Stop stress infra | `bin/tigerfs-stress stop` |

## Related pages

<CardGroup>
<Card title="Error codes" href="/error-codes">
Full errno mapping, constraint behavior, and reading stderr hints.
</Card>
<Card title="Consistency and caching" href="/consistency-and-caching">
Cross-mount freshness rules, cache TTLs, and invalidation on writes.
</Card>
<Card title="CLI reference" href="/cli-reference">
`mount`, `unmount`, `stop`, `status`, `list`, and global flags.
</Card>
<Card title="Configuration reference" href="/configuration-reference">
`metadata_refresh_interval`, `log_level`, and env precedence.
</Card>
<Card title="Platform backends" href="/platform-backends">
Linux FUSE vs macOS NFS write/commit behavior.
</Card>
<Card title="Develop and test" href="/develop-and-test">
Stress runner, integration tests, and pre-commit checks.
</Card>
</CardGroup>