84 lines
4.8 KiB
Markdown
84 lines
4.8 KiB
Markdown
# Database And Log Reliability Implementation Plan
|
|
|
|
> [!NOTE]
|
|
> This document may not reflect the current implementation.
|
|
> See the final report for up-to-date state:
|
|
> [Final Report](../reports/db-log-reliability.md)
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Preserve queued logs during temporary database outages, prevent active ClickHouse connections from being closed during replacement, and migrate missing log columns safely.
|
|
|
|
**Architecture:** Add reference-counted connection leases to `db.Pool`, consume those leases from the logger adapter, and retain an in-memory worker batch while the backend is unhealthy. Extend migration with fixed, additive DDL for missing known columns while retaining strict validation of existing columns and table keys.
|
|
|
|
**Tech Stack:** Go, ClickHouse Go driver, standard library concurrency primitives, Go tests.
|
|
|
|
## Global Constraints
|
|
|
|
- Change only findings 2, 3, and 4 from the review.
|
|
- Keep `Submit` non-blocking and retain existing queue entry and byte budgets.
|
|
- Do not add disk persistence or retry ambiguous `Send` failures.
|
|
- Do not destructively modify existing ClickHouse columns, engine, partition key, or sorting key.
|
|
|
|
---
|
|
|
|
### Task 1: Additive Schema Migration
|
|
|
|
**Files:**
|
|
- Modify: `db/migrate.go`
|
|
- Modify: `db/clickhouse_test.go`
|
|
|
|
**Interfaces:**
|
|
- Produces: migration that adds missing entries from `proxyLogsColumns` using fixed `ALTER TABLE proxy_logs ADD COLUMN IF NOT EXISTS` statements.
|
|
|
|
- [ ] Add tests proving a missing known column executes its fixed additive DDL and succeeds after refreshed metadata; extra columns are accepted; wrong existing types and table keys remain rejected.
|
|
- [ ] Run `go test ./db -run 'Test.*Migration|TestValidateProxyLogsSchema' -count=1` and confirm the new tests fail.
|
|
- [ ] Implement a fixed column-definition map, detect missing columns by name, execute additive DDL, re-query metadata, and validate required columns by name and type without rejecting extras.
|
|
- [ ] Run `go test ./db -count=1` and confirm it passes.
|
|
|
|
### Task 2: Connection Leases
|
|
|
|
**Files:**
|
|
- Modify: `db/clickhouse.go`
|
|
- Modify: `db/clickhouse_test.go`
|
|
- Modify: `logger/queue.go`
|
|
|
|
**Interfaces:**
|
|
- Produces: `Pool.Acquire() (clickhouse.Conn, uint64, func())`; release is idempotent and closes a retired connection only after its final lease ends.
|
|
- Consumes: logger `poolBackend` acquires one lease per flush snapshot and releases it after the flush attempt.
|
|
|
|
- [ ] Add tests proving replacement publishes the new connection without closing a leased old connection, then closes the old connection after release; `Close` rejects new leases and waits within its context for active leases.
|
|
- [ ] Run `go test ./db -run 'Test.*Lease|TestClose' -count=1` and confirm failure.
|
|
- [ ] Add per-generation connection state with active count and retired flag; implement `Acquire`; retire instead of immediately closing on replacement; update `Close` to wait for lease drain under context.
|
|
- [ ] Update `poolBackend` snapshot/release handling so every acquired connection is released after a flush attempt.
|
|
- [ ] Run `go test ./db ./tests/logger -count=1` and confirm passing output.
|
|
|
|
### Task 3: Retain Batches While Database Is Unhealthy
|
|
|
|
**Files:**
|
|
- Modify: `logger/queue.go`
|
|
- Modify: `tests/logger/queue_test.go`
|
|
|
|
**Interfaces:**
|
|
- Consumes: existing `Backend.Healthy()` and leased backend snapshots.
|
|
- Produces: worker batches remain reserved and retry after health recovery; shutdown cancellation records and releases unsubmitted entries.
|
|
|
|
- [ ] Add a test that queues one entry while unhealthy, verifies no prepare call and no failed count, restores health, and verifies exactly one successful prepare/send.
|
|
- [ ] Add a test that shutdown deadline releases a retained unhealthy batch and records it as failed.
|
|
- [ ] Run the two focused tests and confirm they fail.
|
|
- [ ] Change worker flushing so an unhealthy backend returns a retained outcome; wait on a bounded timer or cancellation without consuming additional entries; release only after success/final failure or shutdown.
|
|
- [ ] Run `go test ./tests/logger -count=1` and confirm passing output.
|
|
|
|
### Task 4: Verification And Security Review
|
|
|
|
**Files:**
|
|
- Review only: all changed files and their tests.
|
|
|
|
**Interfaces:**
|
|
- Produces: fresh verification evidence and a list of any new correctness or security issues introduced by the changes.
|
|
|
|
- [ ] Run `gofmt` on changed Go files.
|
|
- [ ] Run `go test -count=1 ./...`; expect only the pre-existing deployment test concerning ClickHouse host ports to fail.
|
|
- [ ] Run `go test -count=1 ./db ./tests/logger`, `go vet ./...`, and `go build ./...`; require success.
|
|
- [ ] Review lease acquisition/release paths, cancellation, lock ordering, migration identifier construction, and queue budget accounting for new vulnerabilities.
|