修复日志持久化可靠性与数据库安全默认值

2026-07-13 11:27:34 +08:00
parent db325ec413
commit 6f37165256
10 changed files with 612 additions and 75 deletions
@@ -0,0 +1,54 @@
+---
+feature: db-log-reliability
+status: delivered
+specs: []
+plans:
+  - docs/compose/plans/2026-07-12-db-log-reliability.md
+branch: main
+commits: uncommitted
+---
+
+# Database And Log Reliability - Final Report
+
+## What Was Built
+
+Temporary ClickHouse outages no longer cause worker-held log batches to be immediately discarded. Each worker retains at most one configured-size batch while the backend is unhealthy, preserving the existing global entry and byte budgets and the non-blocking submission policy.
+
+ClickHouse connections now use generation-bound leases for batch writes and health checks. Replaced or closed connections remain alive until active users release them. Existing `proxy_logs` tables can gain missing known columns through fixed additive DDL after table-level compatibility checks.
+
+## Architecture
+
+`db.Pool.Acquire` returns the current connection, its generation, and an idempotent release function. Retired connections are tracked until all leases are released and the underlying close operation completes. `Pool.Close` prevents new leases and waits within its context for active and in-progress closes.
+
+`logger.Queue` retains a full local batch when `Backend.Healthy` is false and stops consuming further channel entries until recovery or shutdown. The pool adapter acquires one connection lease per flush attempt and releases it after the attempt.
+
+`db/migrate.go` validates the table engine, partition key, sorting key, and types of existing required columns before executing static `ADD COLUMN IF NOT EXISTS` statements. It permits unrelated extra columns and revalidates after migration.
+
+### Design Decisions
+
+We kept outage buffering in memory because the existing bounded queue already defines memory ownership and overload behavior; adding a durable WAL would substantially expand scope. Ambiguous `Send` failures remain non-retryable to avoid duplicate records.
+
+We use static column definitions rather than metadata-derived SQL so migration input cannot introduce identifiers or DDL fragments.
+
+## Usage
+
+No configuration or API changes are required. Existing queue size, byte budget, batch size, and ClickHouse settings continue to control operation.
+
+## Verification
+
+`go test -count=1 ./db ./tests/logger`, `go vet ./...`, `go build ./...`, and `git diff --check` pass. `go test -count=1 ./...` has one pre-existing failure in `tests/deployment`: the unchanged Compose file publishes ClickHouse host ports. Race detection remains unavailable because this Windows environment has CGO disabled.
+
+Independent final review found no new or unresolved high/medium-risk issues in the changed reliability paths.
+
+## Journey Log
+
+- [lesson] Connection leases must cover health checks as well as database writes.
+- [pivot] Pool shutdown now tracks in-progress connection closes so repeated close calls cannot report completion early.
+- [lesson] Retaining an unhealthy batch must also stop channel consumption at `batchSize` to prevent recovery spikes.
+- [pivot] Migration validates table-level invariants before any additive DDL to avoid modifying incompatible tables.
+
+## Source Materials
+
+| File | Role | Notes |
+|------|------|-------|
+| `docs/compose/plans/2026-07-12-db-log-reliability.md` | Implementation plan | Complete |