token_thief/docs/compose/reports/db-log-reliability.md

feature, status, specs, plans, branch, commits

feature	status	specs	plans	branch	commits
db-log-reliability	delivered		docs/compose/plans/2026-07-12-db-log-reliability.md	main	uncommitted

Database And Log Reliability - Final Report

What Was Built

Temporary ClickHouse outages no longer cause worker-held log batches to be immediately discarded. Each worker retains at most one configured-size batch while the backend is unhealthy, preserving the existing global entry and byte budgets and the non-blocking submission policy.

ClickHouse connections now use generation-bound leases for batch writes and health checks. Replaced or closed connections remain alive until active users release them. Existing proxy_logs tables can gain missing known columns through fixed additive DDL after table-level compatibility checks.

Architecture

db.Pool.Acquire returns the current connection, its generation, and an idempotent release function. Retired connections are tracked until all leases are released and the underlying close operation completes. Pool.Close prevents new leases and waits within its context for active and in-progress closes.

logger.Queue retains a full local batch when Backend.Healthy is false and stops consuming further channel entries until recovery or shutdown. The pool adapter acquires one connection lease per flush attempt and releases it after the attempt.

db/migrate.go validates the table engine, partition key, sorting key, and types of existing required columns before executing static ADD COLUMN IF NOT EXISTS statements. It permits unrelated extra columns and revalidates after migration.

Design Decisions

We kept outage buffering in memory because the existing bounded queue already defines memory ownership and overload behavior; adding a durable WAL would substantially expand scope. Ambiguous Send failures remain non-retryable to avoid duplicate records.

We use static column definitions rather than metadata-derived SQL so migration input cannot introduce identifiers or DDL fragments.

Usage

No configuration or API changes are required. Existing queue size, byte budget, batch size, and ClickHouse settings continue to control operation.

Verification

go test -count=1 ./db ./tests/logger, go vet ./..., go build ./..., and git diff --check pass. go test -count=1 ./... has one pre-existing failure in tests/deployment: the unchanged Compose file publishes ClickHouse host ports. Race detection remains unavailable because this Windows environment has CGO disabled.

Independent final review found no new or unresolved high/medium-risk issues in the changed reliability paths.

Journey Log

• [lesson] Connection leases must cover health checks as well as database writes.
• [pivot] Pool shutdown now tracks in-progress connection closes so repeated close calls cannot report completion early.
• [lesson] Retaining an unhealthy batch must also stop channel consumption at batchSize to prevent recovery spikes.
• [pivot] Migration validates table-level invariants before any additive DDL to avoid modifying incompatible tables.

Source Materials

File	Role	Notes
docs/compose/plans/2026-07-12-db-log-reliability.md	Implementation plan	Complete