修复日志持久化可靠性与数据库安全默认值

docs/compose/plans/2026-07-12-db-log-reliability.md

@@ -0,0 +1,83 @@
+# Database And Log Reliability Implementation Plan
+
+> [!NOTE]
+> This document may not reflect the current implementation.
+> See the final report for up-to-date state:
+> [Final Report](../reports/db-log-reliability.md)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use compose:subagent (recommended) or compose:execute to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Preserve queued logs during temporary database outages, prevent active ClickHouse connections from being closed during replacement, and migrate missing log columns safely.
+
+**Architecture:** Add reference-counted connection leases to `db.Pool`, consume those leases from the logger adapter, and retain an in-memory worker batch while the backend is unhealthy. Extend migration with fixed, additive DDL for missing known columns while retaining strict validation of existing columns and table keys.
+
+**Tech Stack:** Go, ClickHouse Go driver, standard library concurrency primitives, Go tests.
+
+## Global Constraints
+
+- Change only findings 2, 3, and 4 from the review.
+- Keep `Submit` non-blocking and retain existing queue entry and byte budgets.
+- Do not add disk persistence or retry ambiguous `Send` failures.
+- Do not destructively modify existing ClickHouse columns, engine, partition key, or sorting key.
+
+---
+
+### Task 1: Additive Schema Migration
+
+**Files:**
+- Modify: `db/migrate.go`
+- Modify: `db/clickhouse_test.go`
+
+**Interfaces:**
+- Produces: migration that adds missing entries from `proxyLogsColumns` using fixed `ALTER TABLE proxy_logs ADD COLUMN IF NOT EXISTS` statements.
+
+- [ ] Add tests proving a missing known column executes its fixed additive DDL and succeeds after refreshed metadata; extra columns are accepted; wrong existing types and table keys remain rejected.
+- [ ] Run `go test ./db -run 'Test.*Migration|TestValidateProxyLogsSchema' -count=1` and confirm the new tests fail.
+- [ ] Implement a fixed column-definition map, detect missing columns by name, execute additive DDL, re-query metadata, and validate required columns by name and type without rejecting extras.
+- [ ] Run `go test ./db -count=1` and confirm it passes.
+
+### Task 2: Connection Leases
+
+**Files:**
+- Modify: `db/clickhouse.go`
+- Modify: `db/clickhouse_test.go`
+- Modify: `logger/queue.go`
+
+**Interfaces:**
+- Produces: `Pool.Acquire() (clickhouse.Conn, uint64, func())`; release is idempotent and closes a retired connection only after its final lease ends.
+- Consumes: logger `poolBackend` acquires one lease per flush snapshot and releases it after the flush attempt.
+
+- [ ] Add tests proving replacement publishes the new connection without closing a leased old connection, then closes the old connection after release; `Close` rejects new leases and waits within its context for active leases.
+- [ ] Run `go test ./db -run 'Test.*Lease|TestClose' -count=1` and confirm failure.
+- [ ] Add per-generation connection state with active count and retired flag; implement `Acquire`; retire instead of immediately closing on replacement; update `Close` to wait for lease drain under context.
+- [ ] Update `poolBackend` snapshot/release handling so every acquired connection is released after a flush attempt.
+- [ ] Run `go test ./db ./tests/logger -count=1` and confirm passing output.
+
+### Task 3: Retain Batches While Database Is Unhealthy
+
+**Files:**
+- Modify: `logger/queue.go`
+- Modify: `tests/logger/queue_test.go`
+
+**Interfaces:**
+- Consumes: existing `Backend.Healthy()` and leased backend snapshots.
+- Produces: worker batches remain reserved and retry after health recovery; shutdown cancellation records and releases unsubmitted entries.
+
+- [ ] Add a test that queues one entry while unhealthy, verifies no prepare call and no failed count, restores health, and verifies exactly one successful prepare/send.
+- [ ] Add a test that shutdown deadline releases a retained unhealthy batch and records it as failed.
+- [ ] Run the two focused tests and confirm they fail.
+- [ ] Change worker flushing so an unhealthy backend returns a retained outcome; wait on a bounded timer or cancellation without consuming additional entries; release only after success/final failure or shutdown.
+- [ ] Run `go test ./tests/logger -count=1` and confirm passing output.
+
+### Task 4: Verification And Security Review
+
+**Files:**
+- Review only: all changed files and their tests.
+
+**Interfaces:**
+- Produces: fresh verification evidence and a list of any new correctness or security issues introduced by the changes.
+
+- [ ] Run `gofmt` on changed Go files.
+- [ ] Run `go test -count=1 ./...`; expect only the pre-existing deployment test concerning ClickHouse host ports to fail.
+- [ ] Run `go test -count=1 ./db ./tests/logger`, `go vet ./...`, and `go build ./...`; require success.
+- [ ] Review lease acquisition/release paths, cancellation, lock ordering, migration identifier construction, and queue budget accounting for new vulnerabilities.

docs/compose/reports/db-log-reliability.md

@@ -0,0 +1,54 @@
+---
+feature: db-log-reliability
+status: delivered
+specs: []
+plans:
+  - docs/compose/plans/2026-07-12-db-log-reliability.md
+branch: main
+commits: uncommitted
+---
+
+# Database And Log Reliability - Final Report
+
+## What Was Built
+
+Temporary ClickHouse outages no longer cause worker-held log batches to be immediately discarded. Each worker retains at most one configured-size batch while the backend is unhealthy, preserving the existing global entry and byte budgets and the non-blocking submission policy.
+
+ClickHouse connections now use generation-bound leases for batch writes and health checks. Replaced or closed connections remain alive until active users release them. Existing `proxy_logs` tables can gain missing known columns through fixed additive DDL after table-level compatibility checks.
+
+## Architecture
+
+`db.Pool.Acquire` returns the current connection, its generation, and an idempotent release function. Retired connections are tracked until all leases are released and the underlying close operation completes. `Pool.Close` prevents new leases and waits within its context for active and in-progress closes.
+
+`logger.Queue` retains a full local batch when `Backend.Healthy` is false and stops consuming further channel entries until recovery or shutdown. The pool adapter acquires one connection lease per flush attempt and releases it after the attempt.
+
+`db/migrate.go` validates the table engine, partition key, sorting key, and types of existing required columns before executing static `ADD COLUMN IF NOT EXISTS` statements. It permits unrelated extra columns and revalidates after migration.
+
+### Design Decisions
+
+We kept outage buffering in memory because the existing bounded queue already defines memory ownership and overload behavior; adding a durable WAL would substantially expand scope. Ambiguous `Send` failures remain non-retryable to avoid duplicate records.
+
+We use static column definitions rather than metadata-derived SQL so migration input cannot introduce identifiers or DDL fragments.
+
+## Usage
+
+No configuration or API changes are required. Existing queue size, byte budget, batch size, and ClickHouse settings continue to control operation.
+
+## Verification
+
+`go test -count=1 ./db ./tests/logger`, `go vet ./...`, `go build ./...`, and `git diff --check` pass. `go test -count=1 ./...` has one pre-existing failure in `tests/deployment`: the unchanged Compose file publishes ClickHouse host ports. Race detection remains unavailable because this Windows environment has CGO disabled.
+
+Independent final review found no new or unresolved high/medium-risk issues in the changed reliability paths.
+
+## Journey Log
+
+- [lesson] Connection leases must cover health checks as well as database writes.
+- [pivot] Pool shutdown now tracks in-progress connection closes so repeated close calls cannot report completion early.
+- [lesson] Retaining an unhealthy batch must also stop channel consumption at `batchSize` to prevent recovery spikes.
+- [pivot] Migration validates table-level invariants before any additive DDL to avoid modifying incompatible tables.
+
+## Source Materials
+
+| File | Role | Notes |
+|------|------|-------|
+| `docs/compose/plans/2026-07-12-db-log-reliability.md` | Implementation plan | Complete |