---
feature: db-log-reliability
status: delivered
specs: []
plans:
  - docs/compose/plans/2026-07-12-db-log-reliability.md
branch: main
commits: uncommitted
---

# Database And Log Reliability - Final Report

## What Was Built

Temporary ClickHouse outages no longer cause worker-held log batches to be immediately discarded. Each worker retains at most one configured-size batch while the backend is unhealthy, preserving the existing global entry and byte budgets and the non-blocking submission policy.

ClickHouse connections now use generation-bound leases for batch writes and health checks. Replaced or closed connections remain alive until active users release them. Existing `proxy_logs` tables can gain missing known columns through fixed additive DDL after table-level compatibility checks.

## Architecture

`db.Pool.Acquire` returns the current connection, its generation, and an idempotent release function. Retired connections are tracked until all leases are released and the underlying close operation completes. `Pool.Close` prevents new leases and waits within its context for active and in-progress closes.

`logger.Queue` retains a full local batch when `Backend.Healthy` is false and stops consuming further channel entries until recovery or shutdown. The pool adapter acquires one connection lease per flush attempt and releases it after the attempt.

`db/migrate.go` validates the table engine, partition key, sorting key, and types of existing required columns before executing static `ADD COLUMN IF NOT EXISTS` statements. It permits unrelated extra columns and revalidates after migration.

### Design Decisions

We kept outage buffering in memory because the existing bounded queue already defines memory ownership and overload behavior; adding a durable WAL would substantially expand scope. Ambiguous `Send` failures remain non-retryable to avoid duplicate records.

We use static column definitions rather than metadata-derived SQL so migration input cannot introduce identifiers or DDL fragments.

## Usage

No configuration or API changes are required. Existing queue size, byte budget, batch size, and ClickHouse settings continue to control operation.

## Verification

`go test -count=1 ./db ./tests/logger`, `go vet ./...`, `go build ./...`, and `git diff --check` pass. `go test -count=1 ./...` has one pre-existing failure in `tests/deployment`: the unchanged Compose file publishes ClickHouse host ports. Race detection remains unavailable because this Windows environment has CGO disabled.

Independent final review found no new or unresolved high/medium-risk issues in the changed reliability paths.

## Journey Log

- [lesson] Connection leases must cover health checks as well as database writes.
- [pivot] Pool shutdown now tracks in-progress connection closes so repeated close calls cannot report completion early.
- [lesson] Retaining an unhealthy batch must also stop channel consumption at `batchSize` to prevent recovery spikes.
- [pivot] Migration validates table-level invariants before any additive DDL to avoid modifying incompatible tables.

## Source Materials

| File | Role | Notes |
|------|------|-------|
| `docs/compose/plans/2026-07-12-db-log-reliability.md` | Implementation plan | Complete |