token_thief/docs/compose/plans/2026-07-12-db-log-reliability.md

Database And Log Reliability Implementation Plan

Note

This document may not reflect the current implementation. See the final report for up-to-date state: Final Report

For agentic workers: REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Preserve queued logs during temporary database outages, prevent active ClickHouse connections from being closed during replacement, and migrate missing log columns safely.

Architecture: Add reference-counted connection leases to db.Pool, consume those leases from the logger adapter, and retain an in-memory worker batch while the backend is unhealthy. Extend migration with fixed, additive DDL for missing known columns while retaining strict validation of existing columns and table keys.

Tech Stack: Go, ClickHouse Go driver, standard library concurrency primitives, Go tests.

Global Constraints

• Change only findings 2, 3, and 4 from the review.
• Keep Submit non-blocking and retain existing queue entry and byte budgets.
• Do not add disk persistence or retry ambiguous Send failures.
• Do not destructively modify existing ClickHouse columns, engine, partition key, or sorting key.

---

Task 1: Additive Schema Migration

Files:

• Modify: db/migrate.go
• Modify: db/clickhouse_test.go

Interfaces:

• Produces: migration that adds missing entries from proxyLogsColumns using fixed ALTER TABLE proxy_logs ADD COLUMN IF NOT EXISTS statements.

• Add tests proving a missing known column executes its fixed additive DDL and succeeds after refreshed metadata; extra columns are accepted; wrong existing types and table keys remain rejected.

• Run go test ./db -run 'Test.*Migration|TestValidateProxyLogsSchema' -count=1 and confirm the new tests fail.

• Implement a fixed column-definition map, detect missing columns by name, execute additive DDL, re-query metadata, and validate required columns by name and type without rejecting extras.

• Run go test ./db -count=1 and confirm it passes.

Task 2: Connection Leases

Files:

• Modify: db/clickhouse.go
• Modify: db/clickhouse_test.go
• Modify: logger/queue.go

Interfaces:

• Produces: Pool.Acquire() (clickhouse.Conn, uint64, func()); release is idempotent and closes a retired connection only after its final lease ends.

• Consumes: logger poolBackend acquires one lease per flush snapshot and releases it after the flush attempt.

• Add tests proving replacement publishes the new connection without closing a leased old connection, then closes the old connection after release; Close rejects new leases and waits within its context for active leases.

• Run go test ./db -run 'Test.*Lease|TestClose' -count=1 and confirm failure.

• Add per-generation connection state with active count and retired flag; implement Acquire; retire instead of immediately closing on replacement; update Close to wait for lease drain under context.

• Update poolBackend snapshot/release handling so every acquired connection is released after a flush attempt.

• Run go test ./db ./tests/logger -count=1 and confirm passing output.

Task 3: Retain Batches While Database Is Unhealthy

Files:

• Modify: logger/queue.go
• Modify: tests/logger/queue_test.go

Interfaces:

• Consumes: existing Backend.Healthy() and leased backend snapshots.

• Produces: worker batches remain reserved and retry after health recovery; shutdown cancellation records and releases unsubmitted entries.

• Add a test that queues one entry while unhealthy, verifies no prepare call and no failed count, restores health, and verifies exactly one successful prepare/send.

• Add a test that shutdown deadline releases a retained unhealthy batch and records it as failed.

• Run the two focused tests and confirm they fail.

• Change worker flushing so an unhealthy backend returns a retained outcome; wait on a bounded timer or cancellation without consuming additional entries; release only after success/final failure or shutdown.

• Run go test ./tests/logger -count=1 and confirm passing output.

Task 4: Verification And Security Review

Files:

• Review only: all changed files and their tests.

Interfaces:

• Produces: fresh verification evidence and a list of any new correctness or security issues introduced by the changes.

• Run gofmt on changed Go files.

• Run go test -count=1 ./...; expect only the pre-existing deployment test concerning ClickHouse host ports to fail.

• Run go test -count=1 ./db ./tests/logger, go vet ./..., and go build ./...; require success.

• Review lease acquisition/release paths, cancellation, lock ordering, migration identifier construction, and queue budget accounting for new vulnerabilities.