# Database And Log Reliability Implementation Plan

> [!NOTE]
> This document may not reflect the current implementation.
> See the final report for up-to-date state:
> [Final Report](../reports/db-log-reliability.md)

> **For agentic workers:** REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Preserve queued logs during temporary database outages, prevent active ClickHouse connections from being closed during replacement, and migrate missing log columns safely.

**Architecture:** Add reference-counted connection leases to `db.Pool`, consume those leases from the logger adapter, and retain an in-memory worker batch while the backend is unhealthy. Extend migration with fixed, additive DDL for missing known columns while retaining strict validation of existing columns and table keys.

**Tech Stack:** Go, ClickHouse Go driver, standard library concurrency primitives, Go tests.

## Global Constraints

- Change only findings 2, 3, and 4 from the review.
- Keep `Submit` non-blocking and retain existing queue entry and byte budgets.
- Do not add disk persistence or retry ambiguous `Send` failures.
- Do not destructively modify existing ClickHouse columns, engine, partition key, or sorting key.

---

### Task 1: Additive Schema Migration

**Files:**
- Modify: `db/migrate.go`
- Modify: `db/clickhouse_test.go`

**Interfaces:**
- Produces: migration that adds missing entries from `proxyLogsColumns` using fixed `ALTER TABLE proxy_logs ADD COLUMN IF NOT EXISTS` statements.

- [ ] Add tests proving a missing known column executes its fixed additive DDL and succeeds after refreshed metadata; extra columns are accepted; wrong existing types and table keys remain rejected.
- [ ] Run `go test ./db -run 'Test.*Migration|TestValidateProxyLogsSchema' -count=1` and confirm the new tests fail.
- [ ] Implement a fixed column-definition map, detect missing columns by name, execute additive DDL, re-query metadata, and validate required columns by name and type without rejecting extras.
- [ ] Run `go test ./db -count=1` and confirm it passes.

### Task 2: Connection Leases

**Files:**
- Modify: `db/clickhouse.go`
- Modify: `db/clickhouse_test.go`
- Modify: `logger/queue.go`

**Interfaces:**
- Produces: `Pool.Acquire() (clickhouse.Conn, uint64, func())`; release is idempotent and closes a retired connection only after its final lease ends.
- Consumes: logger `poolBackend` acquires one lease per flush snapshot and releases it after the flush attempt.

- [ ] Add tests proving replacement publishes the new connection without closing a leased old connection, then closes the old connection after release; `Close` rejects new leases and waits within its context for active leases.
- [ ] Run `go test ./db -run 'Test.*Lease|TestClose' -count=1` and confirm failure.
- [ ] Add per-generation connection state with active count and retired flag; implement `Acquire`; retire instead of immediately closing on replacement; update `Close` to wait for lease drain under context.
- [ ] Update `poolBackend` snapshot/release handling so every acquired connection is released after a flush attempt.
- [ ] Run `go test ./db ./tests/logger -count=1` and confirm passing output.

### Task 3: Retain Batches While Database Is Unhealthy

**Files:**
- Modify: `logger/queue.go`
- Modify: `tests/logger/queue_test.go`

**Interfaces:**
- Consumes: existing `Backend.Healthy()` and leased backend snapshots.
- Produces: worker batches remain reserved and retry after health recovery; shutdown cancellation records and releases unsubmitted entries.

- [ ] Add a test that queues one entry while unhealthy, verifies no prepare call and no failed count, restores health, and verifies exactly one successful prepare/send.
- [ ] Add a test that shutdown deadline releases a retained unhealthy batch and records it as failed.
- [ ] Run the two focused tests and confirm they fail.
- [ ] Change worker flushing so an unhealthy backend returns a retained outcome; wait on a bounded timer or cancellation without consuming additional entries; release only after success/final failure or shutdown.
- [ ] Run `go test ./tests/logger -count=1` and confirm passing output.

### Task 4: Verification And Security Review

**Files:**
- Review only: all changed files and their tests.

**Interfaces:**
- Produces: fresh verification evidence and a list of any new correctness or security issues introduced by the changes.

- [ ] Run `gofmt` on changed Go files.
- [ ] Run `go test -count=1 ./...`; expect only the pre-existing deployment test concerning ClickHouse host ports to fail.
- [ ] Run `go test -count=1 ./db ./tests/logger`, `go vet ./...`, and `go build ./...`; require success.
- [ ] Review lease acquisition/release paths, cancellation, lock ordering, migration identifier construction, and queue budget accounting for new vulnerabilities.