m1saka/dsh-web
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: m1saka/dsh-web:main
Branches Tags
m1saka/dsh-web:main m1saka/dsh-web:dockerfile m1saka/dsh-web:android-app
..
compare: m1saka/dsh-web:ab76be422ddc9aaea6c3e494923e1cd685d47533
Branches Tags
m1saka/dsh-web:dockerfile m1saka/dsh-web:main m1saka/dsh-web:android-app
1 Commits
main .. ab76be422d
Author SHA1 Message Date
m1saka ab76be422d chore: initialize empty dockerfile branch 2026-10-14 15:14:25 +08:00
193 changed files with 0 additions and 31557 deletions
Expand all files Collapse all files
.dockerignore
-12
View File
@@ -1,12 +0,0 @@
.mimocode
.git
**/node_modules
web/dist
data
bin
.env
output
**/output
**/*.tsbuildinfo
.playwright-cli
**/*.sqlite*
.env.example
-27
View File
@@ -1,27 +0,0 @@
# Copy to .env (ignored by git). Use an absolute dedicated data directory.
DSH_HOST_DATA_ROOT=/var/lib/dsh-mgr
DSH_ADMIN_USERNAME=admin
# Supply a unique strong password before FIRST start; the panel reads it only
# while the account table is empty. Remove the value after the administrator
# account exists, then run `docker compose up -d --force-recreate panel`
# to remove it from the container environment and `docker inspect` output.
# Restarting the existing container does not update its environment.
DSH_ADMIN_PASSWORD=
# Numeric group from: stat -c %g /var/run/docker.sock
DOCKER_GID=
# Exact docker info --format {{.DockerRootDir}}; mounted read-only for statfs.
DSH_DOCKER_ROOT_DIR=
# Optional exact extension allowlist, e.g. .txt,.md,.json,.yaml,.go
# Empty permits arbitrary workspace files; this is not malware/content scanning.
UPLOAD_ALLOWED_EXTENSIONS=
# Public origin of the panel exactly as browsers see it (scheme://host, no
# path); controls Origin/CSRF validation and Secure cookies.
PANEL_PUBLIC_URL=http://127.0.0.1:8088
# Optional proxy networks. X-Forwarded-For is ignored unless the peer is in these CIDRs.
TRUSTED_PROXY_CIDRS=
# Absolute login lifetime (Go duration: 24h, 168h for 7 days). Applies to new logins.
# Refresh does not extend this deadline. Invalid/non-positive values prevent startup.
SESSION_TTL=24h
# Trusted executable runtime build source. Do not embed credentials in this URL.
DSH_RELEASE_REPOSITORY=https://git.misaka.ren/m1saka/dsh-web.git
DSH_RELEASE_BRANCH=dockerfile
.gitignore
-38
View File
@@ -1,38 +0,0 @@
.mimocode/
node_modules/
web/dist/
data/
bin/
.env
output/
.playwright-cli/
*.sqlite*
# Local IDE files
.idea/
*.iml
# Generated files left when switching from android-app to main
/.dart_tool/
/.flutter-plugins-dependencies
/build/
/android/.gradle/
/android/.kotlin/
/android/build/
/android/app/build/
/android/local.properties
/android/gradlew
/android/gradlew.bat
/android/gradle/wrapper/gradle-wrapper.jar
/android/app/src/main/java/io/flutter/plugins/GeneratedPluginRegistrant.java
# Android packages and signing credentials
*.apk
*.aab
*.jks
*.keystore
*.p12
*.pfx
keystore.properties
signing.properties
/android/key.properties
ACCEPTANCE.md
-287
View File
@@ -1,287 +0,0 @@
# DESIGN.md 验收记录
> 发布架构变更说明：下文为各阶段的历史验收证据，保留当时命令、测试数量及结论，不作为当前操作指南。main 已移除内置 runtime 构建资产；`release.json`、完整 `image/` 和初始化测试仅在 `dockerfile` 发布分支维护，新构建仅允许管理员通过 Git 发布检查并确认精确 commit。旧 npm/实验构建及构建后创建预约流程已被取代，这是 breaking change：移除 `/api/latest`、`/api/compatibility`、`POST /api/builds` 及 `/api/intents` 的全部查询、创建和取消接口，不提供 410 兼容入口。仅保留 `GET /api/builds` 和 `GET /api/builds/{id}` 构建历史读取。不迁移或恢复旧构建及旧创建意图，不在后台续跑或完成旧预约；这不意味着删除已有实例或数据。旧 `scripts/build-image.sh` 与 `scripts/test-intent-live.mjs` 已删除；`scripts/test-panel.mjs` 现在复用预先验证且准入的本地镜像。历史通过记录不等于新 Git 发布架构已通过真实部署验收，当前使用方式见 README.md、API.md、OPERATIONS.md。
## 当前实现与验收状态（2026-10-16 文档核对）
本节优先于后面的历史记录。已完整阅读两份文档并对照当前源码；本地验证由本次维护任务的独立验证执行者完成并回报，不将代码存在、模拟测试或旧部署证据冒充当前真实环境通过。
### 本次实际验证
- `bash scripts/check-trial.sh`：退出0；Go普通测试/race各5包通过，vet/build通过；前端25文件198测试通过，TypeScript/Vite生产构建通过；`node --check scripts/test-panel.mjs`通过。
- `node --test scripts/check-trial.test.mjs`：10/10通过。
- 两项Docker live测试因未启用`DSH_DOCKER_LIVE_TEST`跳过，**不计真实Docker通过**。本次未执行真实浏览器、公网入口或完整Git发布部署验收，未调用模型或搜索。
- 历史测试数量、镜像ID、截图及/tmp路径仅证明当时记录；本轮未逐一重新获取这些临时证据，不代表当前线上状态。
### 当前能力矩阵
| 范围 | 当前代码与本地验证 | 当前版本仍缺的真实证据 |
|---|---|---|
| 前端与架构 | React19/TS/Vite、Tailwind4/Radix、Monaco；Go标准库Docker HTTP client、SQLite；本轮自动化/构建通过 | 当前UI浏览器全流程与真机；Flutter/Android仅历史工作区残留，无当前原生发布验收 |
| Git发布与准入 | 管理员检查发布/精确commit、本机构建验证/历史；模板7与当前平台身份检查；本地测试通过 | 真实发布→构建→验证→准入→创建完整链路、故障及重启恢复；不能沿用旧npm/实验构建结论 |
| 实例与存储 | 两实例事务名额、每实例bridge/home，/workspace为DSH_HOME；删除清理实例数据及备份；无预约或共享入口 | 当前删除语义、并发名额与不确定创建恢复的真实Docker复验；旧“删除保留数据”证据不适用 |
| 代理与会话 | 归属鉴权、服务端token/cookie桥接、WS/SSE与续期；本地测试通过 | 当前部署iframe、独立页、长时/后台/多页续期与真实WS/SSE；历史615秒结果不是当前整体重验 |
| 文件与配额 | 完整home、上传下载/ZIP/文本编辑/图片预览、账号软配额；本地测试通过 | 当前浏览器编辑冲突/图片显示/敏感确认、跨用户访问、真实配额耗尽及容器直接写盘边界 |
| 配置与升级 | Profile校验/预览、密码加密配置备份/导入/恢复、快照/回滚；本地测试通过 | 当前跨版本/插件重建/离线失败/旧包兼容及回滚数据保留；同版本替换不证明跨版本迁移 |
| 部署与安全 | 登录/refresh/CSRF/审计/改密、非root/只读/资源限制；本地测试通过 | 实际域名HTTPS/代理、socket权限、真实浏览器；不宣称不可信公网多租户隔离或生产SLA |
| 可选增强 | XFS/ZFS硬配额、配额sidecar、socket-proxy未作为当前默认交付 | 未实现/未验收，不标通过；不列为默认软配额流程失败项 |
关键纠偏：除usage外文件API不带instance返回410，新容器不挂/shared；准入布局仅支持模板7，旧模板3–6不能靠说明正向准入。实例删除会清理自身目录和备份，孤儿回收是另一流程。创建不确定时不凭超时404清除依据。见`internal/api/files.go`、`runtime.go`、`instance_data.go`、`admission.go`、`resource_recovery.go`。
结论：**本次本地自动化与构建通过；当前完整真实部署/浏览器验收未完成，不作M1–M7整体结案。**
## 历史阶段记录（以下均非当前版本整体验收）
各节“当前”“本轮”“最终”“已部署”均指该节当时版本；旧命令可能已移除。旧结案不覆盖文首矩阵；各历史阶段之间的覆盖关系仅适用于对应迭代。
## 恢复宽限期与轮询缓存修复（历史，已被后续修复替代）
范围：本轮为对上述工作树的独立代码审计与全量重测后的修复。审计确认八项修复中六项正确，发现三处残留：创建结果不确定的挂起标志永不过期导致实例永久无法删除（高）、构建校验不确定行无限重试并持续新增审计（中）、意图轮询每秒对每个成功镜像发起 Docker inspect（中）。修复未改变凭据文件管理边界或准入策略；未部署、未操作真实 Docker 实例。
- `runtime_creates` 增加 `container_pending_at`/`network_pending_at` 置位时间（列名白名单，禁用未校验列名拼接）。确定性名称 404 在宽限窗口（默认1分钟）内仍视为不确定；窗口过后判定资源从未创建，标志落账清除，实例可正常删除，删除成功后总是清理 `runtime_creates`/`runtime_networks` 行。无时间戳的遗留行视为已过期立即解析。
- `image_build_checks` 增加 `created_at`。无已记录容器 ID 且名称 404 的不确定行在宽限窗口过后删除行停止重试；窗口内保持降频重试与 `build.cleanup_error` 审计（≤1/分钟不变）。
- 意图轮询改用 `verifiedImagesPoll`：成功构建行每秒重读，逐镜像准入评估按 `imageDeleteGeneration`+5秒TTL缓存；镜像删除或准入写入使缓存立即失效。创建/升级入口仍走未缓存的权威校验。
- 前端：无 Web Locks 浏览器的会话恢复重试窗口从约0.85秒延长至约3.85秒；`useResource` 对60秒不返回的请求自动放弃并重新武装轮询（序号守卫防止迟到结果覆盖新状态）；`runtimeSession` 测试改为 TypeScript 并纳入 tsc 检查；`test-panel.mjs` 构建详情轮询对瞬时 5xx 容错续轮。
回归覆盖：宽限窗口内删除被拒、窗口过后删除成功且无残留行（TestUncertainPendingResolvesAfterGrace）；校验行窗口内保持不确定、窗口过后与遗留行均解析（TestBuildCheckUncertainRowResolvesAfterGrace）；轮询三次仅一次 Docker inspect、代际提升后缓存失效（TestIntentPollerCachesImageEvaluations）；悬挂请求逃生阀（ui.test.ts）。Go共有192个顶层测试（另有子测试）。前端15个测试文件、95项测试通过；`@types/node` 新增为 devDependency，官方源 npm audit 0 漏洞。
## 完整home与镜像准入策略实施
本节落实用户确认的完整home所有者访问、兼容清单及管理员实验构建策略，取代下方上一轮“尚未改变这两项策略”的历史边界。未部署，未操作真实Docker实例或为实际镜像写入准入。
- 所有者可读取/打包凭据，普通文件管理不强制使用专用Profile流程；匿名、跨用户和路径穿越仍拒绝。下载及敏感修改增加确认，取消不触发操作；上传同名拒绝不覆盖。专用Profile页面仍要求停机，文案与通用文件管理明确区分。
- 默认普通构建支持linux/amd64、模板7的rc.1/rc.2；未知版本只能管理员实验构建。实验完整执行原检查且success不自动可选。准入绑定精确imageID/version/recipe/platform与成功验证记录，管理员填写依据，策略和审计同事务，审计失败回滚。
- 历史模板3–6默认仅存量，经明确验收可准入并使用旧双挂载；未知布局拒绝正向准入，模板7向旧共享布局降级拒绝。已准入未知版本只通过本地imageID使用，不自动扩大版本构建白名单。
- blocked拒绝新建、升级目标、start/restart及受控配置非GET操作，避免恢复/备份自动启动绕过；不自动停止既有容器，普通文件管理保持可用。队列恢复/执行重查版本、平台及实验请求者管理员权限。
- 独立复核修正了准入与生命周期锁顺序、受控配置启动遗漏及两处提示矛盾；最终静态复核未发现剩余阻塞项。
回归覆盖精确版本/模板/平台匹配、旧模板实际创建挂载、实验success仍不可选、未知请求零预留、普通用户和CSRF拒绝、审计回滚、并发降级等待创建完成、blocked启停/配置拒绝，以及前端准入确认/取消和选择器过滤。Go共有189个顶层测试（另有子测试）；本轮Go普通全量和全量race禁缓存通过，API包分别约17秒与97秒。前端15个测试文件、94项测试通过。浏览器使用模拟API验证实验警告、准入表单、创建筛选及unknown latest禁止预留，临时服务和浏览器已关闭；非真实Docker或新DSH版本兼容验收。
完整 `scripts/check-trial.sh` 最终退出码0；`go vet`、Go构建、TypeScript与Vite生产构建、profile初始化3项Node测试、验收脚本自身11项Node测试和脚本语法检查均通过。仍有既有Vite大chunk提示，非构建失败；`git diff --check`通过。
## 2026-10-14 真实环境补充验收与构建拆包
- 全量 race：`go test -race -count=1 -timeout=180s ./...` 退出 0，全部包通过、无 race 报告；API 包 166.645 秒，总耗时约 170 秒。
- 真实 Docker：新增默认跳过、需 `DSH_DOCKER_LIVE_TEST=1` 启用的 `TestDockerLiveRecovery`。经故障代理向真实 `/var/run/docker.sock` 转发，真实 create/delete 成功后丢失响应，或断开客户端并延迟创建，先取得真实 404 再放行。六个子场景连续两轮通过，另一次 race 模式通过。覆盖 runtime/network/build-check 恢复、面板未连接/完整 ID/名称连接及删除响应丢失。仅使用专用唯一标签资源、本地 node:24-alpine；按精确 ID 清理后标签查询零残留。测试调用恢复方法，不模拟整机断电，也不重启共享 daemon。
- 真实浏览器续期：当前 Go 源码、隔离临时数据库、真实 DSH rc.2 模板7容器、Chromium 独立 Cookie 上下文，原始 5 分钟 access / 4 分钟续期间隔。2026-10-14T03:49:07.767Z 至 03:59:22.828Z 共 615.061 秒；初次、第 240.023 秒及第 480.020 秒 refresh 均 200；每 15 秒原生 fetch 检查 `/api/me` 和真实 runtime，共 41 轮全部 200（含第 301 秒和第 601 秒）。没有加速时钟、手动续期或共享 Cookie 的其他页面。证据 `output/playwright/renewal-acceptance.json`；不代表休眠/后台节流或多小时压力验收。
- Monaco 改为 React.lazy 按需加载，依模块分层拆包；主入口从约 2589 kB 降至 284.92 kB，最大 chunk 411.21 kB，worker 232.35 kB，保留默认 500 kB 阈值且最终构建无大 chunk 或循环 chunk 告警。新增构建测试约束体积、入口不静态依赖 Monaco 及 chunk 无环；16 文件/114 项前端测试通过。真实浏览器验证编辑输入、运行中保存禁用、只读预览及 worker 初始化成功，截图 `output/playwright/editor-chunks-verified.png`。
- 最终 `go vet ./...`、`go build ./...`、TypeScript/Vite 构建和 `git diff --check` 通过。专用面板、两个浏览器已关闭，验收容器/网络及临时数据和凭据已清理；原四个运行容器仍在，未部署到现有面板、未更改共享镜像。
复现真实故障测试：`DSH_DOCKER_LIVE_TEST=1 go test ./internal/api -run TestDockerLiveRecovery -v -count=2 -timeout=3m`；需可访问 Docker socket 和本地 node:24-alpine，不会自动拉取镜像。
## 再次审查修复（本地回归）
- runtime 注入脚本每 4 分钟主动续期，不再因 `/api/me` 仍有效而跳过；保留 Web Locks 串行刷新、同页请求合并及冲突复验。模拟时钟覆盖连续续期，补测锁等待期间 Cookie 更新。
- 资源加载超时会结束调用方等待链、清除 loading 并提示重试；覆盖无轮询、旧请求和后续请求均悬挂、迟到成功/失败、超时配置变化，不再只清空 in-flight 引用。
- Docker 创建结果不确定时，不凭一分钟后的名称 404 清除 pending/验证容器记录；覆盖旧/缺失/非法时间戳及迟到资源回收。网络清理只断开确实连接的面板端点，保留身份校验和真实错误处理。持续未确认的创建需按 OPERATIONS.md 人工确认。
- 准入依据前端按实际提交内容计算 UTF-8 字节数，超出 4096 字节时提示并禁止提交；后端保持字节上限及请求体限制，长度错误明确返回。覆盖中文、ASCII、多字节、表情及边界。
本轮通过：`go test -count=1 -timeout=180s ./...`、准入与资源恢复相关 API 测试的禁缓存 race 检查、`go vet ./...`、`go build ./...`；前端 15 文件/113 测试及 TypeScript/Vite 生产构建；profile 初始化与验收脚本自身共 14 项 Node 测试、集成脚本语法检查、`git diff --check`。本轮未执行全量 race、真实浏览器长时续期或真实 Docker 故障注入；未部署、未操作现有实例。Vite 仍有既有大 chunk 提示。
## 八项审查问题闭环（本地回归）
范围：普通创建响应丢失、验证容器回收、构建终态持久化、多页面续期、慢请求轮询、当前密码重认证、配额更新竞态、构建历史分页与按需日志。未改变凭据文件管理边界或历史镜像兼容策略；未部署、未操作真实 Docker 实例。
- 资源恢复：创建前保存恢复依据；校验资源身份后按持久 ID 清理。覆盖迟到资源（首次404随后出现）、网络创建/删除响应丢失、验证容器重命名、外部同名网络拒删、清理失败重试、重开数据库后无排队任务自动回收。个别预约待恢复不会阻止面板启动。
- 构建终态：任务与镜像索引事务提交，故障注入验证无半提交；独立重试不阻挡后续构建，Store关闭后退出。持续不确定创建保留恢复记录，不凭单次404释放资源依据；人工确认流程见 OPERATIONS.md。
- 认证：备份、升级、改密共用失败计数与锁定；旧备份密码不参与登录失败计数。并发refresh冲突返回409且不签发cookie，前端及runtime复验后保留可重试错误，真正撤销仍退出；覆盖慢升级不阻塞其他账号登录。
- 文件与前端：上传提交锁内读取最新额度；管理员修改额度共用用户锁。慢轮询不重叠、不废弃有效慢响应，显式刷新保证额外读取，key切换隔离旧结果。构建列表默认20条、最多100条且不读取日志正文，详情按需加载；参数、路由鉴权、整页边界和当前页镜像对账均有回归。
最终验证：
- 完整 `scripts/check-trial.sh` 通过，最终退出码0；最终Go测试通过 `GOFLAGS=-count=1` 禁用测试结果缓存。Go共有182个顶层测试（另含子测试），API普通测试约18秒，全量race中API包约110秒；无race报告。
- `go vet ./...`、`go build ./...`、前端类型检查与生产构建通过；Vite仍提示原有大chunk，不影响构建成功。
- 前端10个文件、57项测试通过；profile初始化3项Node测试、验收脚本自身11项Node测试通过；真实runtime注入脚本的401/403/409/500行为已自动测试。
- 浏览器配合模拟API验证每页20条、末页和禁用下一页、日志打开/关闭及终态停止轮询。临时浏览器和开发服务器已关闭。
- `git diff --check`通过。Docker故障场景使用httptest模拟，并非真实daemon断电/断网验收；未运行指向真实部署的test-panel脚本，仅更新其单任务查询并检查语法。
升级注意：已打开的旧页面需刷新；外部 `/api/builds` 消费者需迁移分页和单任务日志接口。持续数据库故障或身份冲突不能承诺自动成功，保留记录并按运维流程处理。
## 2026-10-12 可信团队试运行加固（本地自动化验收）
范围：本机或仅可信账号的团队，可由管理员通过公网 HTTPS 反代访问。未修改 Nginx/Caddy，未部署面板或操作现有实例。本节仅说明当前代码的本地自动化验收，不将历史部署证据当作本次公网实测。
已修复：
- 登录来源30次/分钟、有界1024来源桶、8个并发槽；Origin/严格JSON校验先于额度消耗；保留账号失败锁定。默认不信任XFF，仅配置的可信代理CIDR可提供来源链，非法链回退peer。非法公网URL/CIDR在创建数据目录前拒绝；URL规范化大小写与默认端口。
- 实例文件传输改为短锁登记lease，不再持全局生命周期锁传输；同实例启停/删除/修改/升级/恢复冲突返回409。配置JSON/multipart锁外有界接收（并发4、30秒读deadline），JSON响应在解锁后发送。
- runtime token按容器ID/StartedAt加密持久化，Host分别缓存；日志流式扫描64MiB上限、完整帧校验、逐行选最新启动token。上游401/403失效cookie，下次请求重新交换，不重放写请求。
- 恢复快照复制失败清理本次半成品，保留有效快照；清理失败明确500及审计。目录列表10000项上限，用量和快照分批遍历。
- 构建容量测试注入可控探测，生产2GiB阈值不变；更新模板7文档、实例范围集成脚本和一键验收入口。
实际通过：
- `bash scripts/check-trial.sh` 退出0：Go全量测试、全量race、vet、build；前端32测试及生产构建；初始化3测试；集成脚本语法检查。全量race在此脚本中使用刚刚分包成功运行的Go测试缓存；API包单独完整race约54秒通过。此前一次全量并行race被工具60秒上限终止，不计通过。
- `node --test scripts/check-trial.test.mjs`：11项通过（包括每阶段失败即停）。
- `TestSecurityTLSCookieJarLifecycle`：真实本地TLS HTTP服务/cookiejar登录、me、refresh轮换、logout、旧会话拒绝；来源隔离、伪造头、非法配置、并发限流回归通过。不是公网证书/反代验收。
- 慢上传/下载/配置body/错误JSON不持锁、同实例busy、失败恢复三次不耗名额回归及race通过。
- runtime专项race覆盖9MiB raw/framed日志、逐字节分片、截断帧拒绝、面板对象重建、Host切换、启动时间变化、磁盘权限/加密/篡改、写盘失败不缓存、cookie拒绝后重交换。Docker及上游由测试HTTP服务模拟。
- Compose用占位环境变量只读解析，公网URL及可信代理CIDR正确展开。
剩余边界与部署前检查：
- 未重新构建/启动真实runtime镜像，未执行真实Docker生命周期、浏览器或实际公网域名验收；未调用模型、搜索或收费服务。必须在部署入口确认登录/刷新、iframe、WS/SSE和插件设置工作。
- 必须设置实际 `PANEL_PUBLIC_URL` 和准确的 `TRUSTED_PROXY_CIDRS`；代理及后端可达范围由管理员负责。公网并不意味着支持不可信租户。
- 旧实例首次成功连接前若启动token已轮转丢失，需维护窗口重启该实例再连接；任意上游token主动轮转不保证自动恢复。软配额、同源插件、已建立WS/SSE不即时撤销等既有边界保留。
- 前端仍有Monaco大chunk警告。本轮未补许可证、依赖漏洞全量扫描或开源发布材料，不作开源发布就绪声明。
结论：本地代码与自动化验收通过，可进入本机/可信团队受控试运行的部署验证；不宣称实际公网部署已验收或长期生产SLA已达标。
## Free Search 模板5（部署及真实iframe验收完成）
- 最终面板已部署：6067f9e8bd12aff649c3b8cc662c37a7a14eebec2d686b84a0d965ee599a4dc5，镜像sha256:6f397fba50a9c3469c1e5ae1ed685e6a4fe90f08ed91d947f38eca9c0b3a4932；原配置/挂载保留，旧面板停止留作回滚。正常API构建记录617457ffdd4c168c58f9cb21e7a591cc成功，模板5可用。
- 专用新实例5dfabbdde3339c8df7bb616e1c65dce9真实iframe：Installed2、Free Search0.8.5 Running、Configure打开完整设置；前缀describe/credentials-status/mutate均200。Bing地区及缓存时长修改后保存，再恢复zh-CN/5并确认；普通DSH页面重新加载200。截图output/playwright/free-search-settings.png。未执行搜索/引擎测试/在线更新/密钥配置。
- 实际HTTP匿名401、恶意Origin403、缺失Origin403，根路径请求拒绝；跨用户/public Host由代理自动化测试覆盖，本轮未另做实际公开域名或真实跨用户浏览器验证。全量race TEST_EXIT=0（/tmp/dsh-free-search-race.log）。
- 上游0.8.5已确认UI限制：单改Bing地区不会置dirty，保存按钮不启用；修改缓存时长后可一起保存。未擅改上游发布包，不把此缺陷视为已修复。
- 测试实例已经正常API删除，bind数据保留。旧63b实例仍为recipe4，容器ID/StartedAt13:01:01.657475588Z/RestartCount0保持，未安装或启用free-search。
- 固定内置dsh-free-search0.8.5及三项普通依赖原始tarball，SHA-512校验；新profile正式依赖及bundle默认启用，旧profile不注入，mobile旧路径保留。runtime镜像dsh-web:0.2.0-rc.2-5，ID sha256:aea9e1ed8614b30457c1d60274a68175e85c1716734e4bca396205bffa885eb6。
- 断网/非root/只读rootfs启动、两个插件inventory及listBundles、settings describe200、无重复宿主peers或绝对symlink、离线恢复和禁用状态保留、recipe4 mobile-only恢复通过。证据output/recipe5-build.log及recipe5-smoke.log。未调用搜索、引擎测试、收费服务或模型。
- 面板精确适配8个settings路径，先登录/实例归属/Origin检查再转换loopback Host及Origin，并重新交换Host绑定cookie；客户端fetch仅重写同源精确路径。真实HTTP upstream测试涵盖公开Host、body/query、普通资源、匿名/跨用户/恶意或缺失Origin拒绝，Node测试覆盖Request/URL/string及流语义。Go全量/vet、初始化3测试和聚焦代理race通过。
- 上游会接管等于deepseek-official的搜索及默认http抓取（包含显式同名值）；其他自定义provider保留。默认Bing但自动回退可能使用已有收费API凭据，不承诺零费用。
- 早期部署曾被权限规则拒绝，后续用户授权继续，主会话接手完成上述部署和验收；实际公网域名未单独验收。
## 管理员重设与用户自助修改密码验收
- 已部署 `dsh-mgr:password-complete`（sha256:83189ad42e35b5a7db3ba74e27f62adcd35f4c45635be35d4f5e568b37c54c75）至127.0.0.1:40411。面板Config除Image外、HostConfig及Mounts一致，旧面板停止保留供回滚；DSH runtime ID/启动时间/重启计数未变。
- 管理员入口：设置→用户与审计→用户列表→重设密码。普通用户及管理员自助入口：设置→账号→修改密码，必须验证当前密码。
- 专用成员pwdqa_1791643434738真实Playwright CLI验收：管理员重设、确认不一致错误、新密码登录；成员错误当前密码保留登录、确认不一致错误、成功修改返回登录页及新密码登录均通过。没有修改既有用户密码。
- 两次改密后旧密码及旧access/session请求401，携带原CSRF的旧refresh请求403（session已删除），新session刷新200。审计显示user.reset_password和user.change_password，不含密码。
- Go全量race通过（/tmp/dsh-password-race-final.log，TEST_EXIT=0），vet、前端21测试及生产构建通过；既有Monaco大chunk警告仍在。
- 证据：output/playwright/password-member-empty-form.png、password-admin-reset-empty-form.png；/tmp/dsh-password-browser-evidence.txt。测试会话已退出；因无删除用户API，专用成员保留，随机测试密码未持久化。
- 限制：已建立的WS/SSE不会被主动即时断开，后续请求及重连拒绝旧会话。保留的旧面板是部署回滚副本，不是数据库时间点备份。
## 模板4：修复插件列表不可见（取代模板3管理可用性结论）
- 根因：仅host依赖和bundle可使插件运行，但DSH管理UI按profile dependencies判断installed，模板3未登记而被过滤。模板4固定tarball正式依赖，补缺失workspace的no-auto-peers配置，保留已有manifest/bundle禁用状态。
- runtime `dsh-web:0.2.0-rc.2-4`：sha256:d967800c73888cefba67b8ea14b923c14032f96058311630a37210219877fcdd。面板已部署模板4并经正常API登记镜像。
- 当前实例63b337c848bf23156270068ff9e5cf38已备份、升级并补mobile依赖，容器2e2d6c75946a65eb7b39a11d72e2e3d23394c3d92d6508cfe25a8fa827a450fb，配置/数据保留。备份73e27b58f421868de00b07958f88113d及此前备份保留。
- 实际面板iframe触屏浏览器：Plugins新增Installed1，dsh-web-mobile v3.0.5，详情组件Running；真实开关关闭显示Off、再次开启恢复运行，最终保持启用。截图output/playwright/plugin-touch/recipe4-installed-detail.png。UI显示Uninstall但未执行卸载；无独立参数配置表单。
- 初始化3测试、Go全量/race/vet通过。断网非root只读冒烟验证正式inventory、上游listBundles、无重复peers/绝对symlink，以及仅manifest/patch/lockfile恢复后离线安装并保留disabled状态。当前实例未另做disabled后容器重启测试。
- 迁移曾因旧package.json.lock及.plugin-manager/run.json阻塞并回滚；锁PID16与新容器安装进程PID16相同。确认实例完全停止后将两份记录改名保留，停机直接升级成功。未删除配置数据；该上游纯PID锁风险未通过自动删锁规避，后续遇到需停所有profile写入者再人工核验恢复。
## 默认内置 dsh-web-mobile 增量验收
- 模板修订3固定内置 `[email protected]`，校验发布包SHA-512与宿主peer版本，不执行插件生命周期脚本。当前限定DSH rc.1/rc.2；真实镜像测试使用rc.2。
- 新建空web profile默认加入mobile bundle；已有manifest或其他profile文件不覆盖、不强制启用。初始化拒绝符号链接目录，幂等/已有配置保留及符号链接测试均通过。
- 发布包原始权限导致非root加载失败，已通过镜像内 `chmod -R a+rX` 修复。修复后 `node scripts/test-mobile-image.mjs` 通过：断网、非root、只读rootfs启动，manifest与实际dump-config均含mobile插件。专用冒烟容器及其tmpfs测试数据已删除。
- 验收面板 `127.0.0.1:40411` 已更新至 `dsh-mgr:plugin3-active`（sha256:06bca1aae39accbac0721af723082e8a35a0c9cb2129e4cb942015b48fbffd85），HTTP 200。正常构建API成功登记rc.2模板3，镜像API状态available。旧面板停止保留供回滚，现有模板2实例未重启或自动加入插件。
- 本轮通过：初始化2测试、Go全量测试/race/vet、前端19测试及生产构建（仅既有Monaco大chunk警告）。
- 后续浏览器验收发现单纯放入node_modules仍会导入失败：DSH运行时只解析安装依赖图。已将内置包加入宿主package.json dependencies，且冒烟增加stdout/stderr激活失败检查（不输出token）。最终runtime ID：sha256:30ca773ce5d70a3a540a8f4aae149c83733102f8dfcd6a77ef9794bd992bde8e。此前只检查dump-config的成功结果不等于运行时激活成功，以下复验取代它。
- Playwright CLI独立loopback runtime触屏复验：coarse pointer=true、maxTouchPoints=1；含dsh-web-mobile/client.js的实际模块请求200；四个data-mobile-nav标记生效。首页及tap展开目录后viewport/client/scroll/body宽度均390。截图：output/playwright/plugin-touch/rc2-3-fixed-default-390.png、rc2-3-fixed-drawer-390.png。临时runtime已删除。
- 边界：本次插件复验未覆盖面板iframe、实体手机或模型交互；额外目录关闭tap超时，未计通过。前轮专用临时runtime token曾在工具输出回显，已删除该容器使服务不可访问，未涉及旧实例凭据。插件还包含响应压缩、会话删除入口和推理档位补全，不是纯CSS；未执行模型调用。
## 2026-10-10 移动端内嵌工作台增量验收
用户追加需求：登录默认 iframe 内嵌 DSH，设置切换容器，移动端底部导航。已实现并部署至原验收地址，镜像 dsh-mgr:mobile-renew（sha256:392c8bb64c822572005625df5d662774e1f780a1c83121caa249fafe6a16d01f）。
- 真实浏览器390×844及320×640：外层scrollWidth等于viewport宽度；iframe底边分别780/576，与底栏顶边一致，无底栏遮挡。桌面1440×900保留侧栏。
- 设置切换第二个真实容器、整页刷新后选择保留；停止所选实例显示启动提示而非自动切换；删除测试实例后回到可用实例。测试实例已删除，bind数据保留。
- 设置往返iframe DOM标记保留，未重载；退出登录iframe卸载；重新登录默认iframe已验证。
- 清除dsh_access模拟过期，真实iframe续期refresh200并回到原DSH地址，无加载错误。修复iframe导航续期及SPA X-Frame-Options冲突；使用SAMEORIGIN与frame-ancestors self保留跨源嵌入限制。
- Go全量测试、race及vet通过；前端19测试/生产构建通过。截图：output/playwright/design-final/mobile-settings.png及page-2026-10-10T09-29-51-590Z.png、page-2026-10-10T09-34-19-112Z.png。
- 验证为Chrome小屏视口，不代表iOS/Android真机软键盘兼容已全测。同源iframe不是安全沙箱；保留受信任团队部署边界。
以 DESIGN.md 为目标，不通过修改设计降低标准。以下最新结论覆盖后文早期进度记录。
## 2026-10-10 最终验收结案
**结论：按用户确认的默认路径代理、软配额及功能等价替代范围，验收通过并结案。** 用户在最终结构化确认中明确选择“接受功能等价替代”：标准库 Docker HTTP 客户端替代 SDK、每实例独立 bridge 替代共享 bridge、服务端保管 DSH cookie 替代下发浏览器、React+CSS 替代 shadcn（保留 Monaco）、部分模块集中于 internal/api。上述选型不再是待确认项；并未修改 DESIGN.md 来降低要求。
结案时再次确认 /healthz 返回 ok，dsh-design-check 正在运行 dsh-mgr:design-final，用户1000:1000、只读根文件系统、内存限制1073741824字节。此次结案仅更新验收文档，没有新增代码变更；沿用下列已完成测试证据。
默认路径代理、软配额方案的 M1–M7 功能已实现，真实主要流程通过；故障边界结合自动化注入测试验收，不声称所有故障均在宿主实际制造。可选 XFS/ZFS、socket-proxy、公网 TLS 未实测。
| 范围 | 最新证据 |
|---|---|
| M1 镜像 | rc.1/rc.2 recipe2实际构建，rc.2 smoke通过，精确版本/非root/只读/pnpm10.11.0 |
| M2 生命周期 | 最终面板完整API通过；test-intent-live.mjs验证预约自动创建、两个真实运行实例、第三创建与预约409、删除释放 |
| M3 代理 | 新浏览器DSH/空白会话/插件资源/SSE200；最终WS 1连接、5发送、5接收；面板替换自动重新接入网络；独立DSH页超过5分钟refresh200且PTY可操作 |
| M4 工作区 | 33项真实HTTP+43项多用户断言；ZIP/Range/穿越；PTY读取上传文件；直接调用镜像内DSH read/write实现在/workspace往返（无模型调用）；扩展名白名单.txt成功/.exe415 |
| M5 配置备份 | Monaco键盘编辑、保存启动、bundle/脱敏dump-config；浏览器备份创建/下载/错误密码拒绝/正确密码导入恢复 |
| M6 升级回滚 | rc.1→rc.2；manifest加入is-number@7.0.0验证依赖安装/恢复/迁移；不可达依赖触发失败，原ID/版本/停止状态恢复，数据marker保留。fixture验证安装机制，不代表任意第三方插件兼容 |
| M7 管理安全 | JWT/refresh/隔离/配额/审计；最终面板1000:1000、只读、1GiB、socket组989；管理员显式回收停止孤儿容器通过且bind目录保留 |
故障注入回归覆盖：构建去重/限频、磁盘不足/未知磁盘、npm精确版本、镜像缺失、重启恢复、失败释放预约、镜像引用保护、备份资源与旧包、路径/符号链接/配额并发和回滚。见internal/api/*_test.go及internal/backup/*_test.go；没有填满宿主磁盘或删除共享镜像。
真实验收发现并修复：dump-config顶层YAML列表误拒绝；代理日志旧token污染启动token提取。新增测试并容器复验通过。
最终镜像：
- panel dsh-mgr:design-final：sha256:d2da2f35e00e15fc0da7afc5183c49e40be6215bd4c5d89145e48f1034617cc5
- rc.2 recipe2：sha256:7bd09f12b4b60fa9d13358e99bae14caafc79d3a0936e70cf590fdd1af64998e
- rc.1 recipe2：sha256:1bb553deb3cef25c4529f93526d8cd3282d218df22787be2dc5d5762defaf0c4
通过命令：go test ./... -timeout 45s；go test -race ./... -timeout 90s；go vet ./...；npm --prefix web test（12项）；npm --prefix web run build。前端有Monaco大chunk警告。
证据：output/playwright/design-final/；/tmp/dsh-design-check.8ekWaC/{api-check-results.txt,isolation-results.json}；/tmp/dsh-design-plugin-rollback-verified.log；/tmp/dsh-design-cross-version.log。保留专用面板及一个浏览器实例；临时实例删除但数据保留，未推送镜像、未调用付费模型。
用户确认的孤儿策略：管理员二次确认回收身份/挂载核验通过、无实例/升级引用的停止容器；不自动纳管，不触碰运行中或来源不明容器，不删数据/网络。其他架构替代见下表；不声称逐字遵循SDK、组件库或目录示意。
## 后续UI改版记录（历史独立阶段，不由2026-10-10结案覆盖）
## UI 改版与功能回归（2026-10-15）
本节为本轮证据，后文保留历史记录；不代表所有历史待验收项均已关闭。
- 前端改为 Tailwind v4、Radix 原语的 shadcn 风格组件及 lucide 图标；桌面侧栏、移动导航、响应式卡片、确认弹窗和备份操作统一。
- 最终前端 20 个测试文件、131 个用例通过，TypeScript/Vite 生产构建通过；Go 全量测试、race 测试、vet、build 均通过；check-trial 脚本的 10 个测试通过，git diff --check 通过。
- 浏览器验证桌面及 320/375/667px 移动视口：无页面横向溢出、主要控件触控高度至少 44px、弹窗关闭焦点恢复、准入编辑器只挂载一份且跨断点保留草稿。配额读回、重设密码后登录、自助修改密码后退出、删除取消/确认、重命名冲突在弹窗内提示并可修正重试、错误备份密码在当前弹窗提示均通过。截图位于本地 .playwright-cli/。
- 隔离真实 Docker 面板执行 scripts/test-panel.mjs，TEST_UPGRADE=1，最终退出码 0：登录/刷新、镜像选择、创建/代理、文件夹/上传/下载/ZIP/重命名/删除、Profile 保存、加密备份/下载/导入/恢复、启停、同镜像替换升级、实例删除通过。补充验证未认证 401、缺少 CSRF/外部 Origin 403、普通用户不能访问用户管理、配额更新与负值拒绝。
- 真实回归发现并修复两个后端缺陷：空 Profile 规范化为合法空数组 YAML；回滚停止容器接受 Docker 304，但必须继续核验确实停止。新增回归用例，修复后完整集成流程重新通过。
- 本次升级仅覆盖 0.2.0-rc.2 到相同版本的容器替换；跨版本升级、插件重建、离线失败回滚、长时会话续期，以及真实配额耗尽/跨用户实例访问未在本轮完整实测，不计为通过。未调用付费模型。
- 一次性测试实例及其容器/网络均已清理；原有 DSH 实例未重启。测试日志及隔离备份保留于 /tmp/dsh-isolated-integration-BFUCkK/（本地临时证据，不随提交分发）。
## 早期实现与待验收表（已归档）
原表描述模板2、npm构建及预约阶段，已由文首当前矩阵替代。早期未闭项须按各阶段证据解释，不能推导当前Git发布通过。配置备份为白名单包，不含完整工作区、依赖或会话；旧密钥包仍须保管原key。操作中断或回滚失败不保证自动灾难恢复。
## 历史合理替代方案（不改变验收目标）
| DESIGN选型 | 当前实现 | 等价目标与边界 |
|---|---|---|
| 官方Docker SDK | stdlib net/http Docker Engine client，API v1.47 | 仍须验证编排、构建流、错误处理和daemon兼容性，不能以未用SDK认定功能缺失 |
| 共享bridge | 每实例独立bridge，面板接入 | 减少实例直接互通；不等于完整出站或宿主网络隔离 |
| DSH cookie改写后下发浏览器 | 服务端交换/缓存cookie，反代注入并移除上游Set-Cookie | 保留面板登录与实例归属鉴权；WS/SSE、重启失效与续期仍需实测 |
| shadcn/ui | 当时为普通React/CSS；现为Tailwind v4及Radix原语的shadcn风格组件，保留Monaco | 历史确认不替代当前UI浏览器验收 |
代码主要按internal/api、internal/docker、internal/backup、internal/store组织，与设计目录示意不同；不宣称逐字一致。默认软配额符合§9/§18路线，XFS/ZFS硬quota依赖运维选择，不列为默认必过项。npm 0.2.0-rc.2不支持--public-url，当前使用路径代理而不传该参数。
## 早期历史证据与当时边界（非当前待办结论）
本轮已取得真实证据：
- 非root面板 `dsh-design-check`（1000:1000、只读rootfs、socket附加组989）已启动；33项HTTP认证/刷新/配额/ZIP/Range检查通过，报告 `/tmp/dsh-design-check.8ekWaC/api-check-results.txt`。
- 两普通用户隔离与预约43项断言通过，包括越权拒绝、两名额及取消释放；报告同目录 `isolation-results.json`。未把预约名额测试算作真实runtime并发测试。
- 浏览器基础UI登录、配额、ZIP、镜像状态、版本确认与预约页面通过，证据 `output/playwright/design-final/`。实例相关Monaco/备份/长JWT待runtime模板2就绪。
- 模板2实际构建进行中，详细安装日志持续有进度；尚无runtime2启动通过结论。
- 既有记录包含旧版真实Docker文件/实例/备份恢复及同镜像替换流程；同镜像替换不能证明跨DSH版本迁移。
- 旧浏览器证据目录output/playwright/browsercheck18091/记录插件加载、空白会话及WS/SSE，不代表模型生成、PTY或真实插件安装通过，也不能替代模板2和新增UI。
- 上轮记录的go test -race ./... -timeout 90s、go vet ./...、前端12项测试及前后端构建通过，仅为当时检查结果；冻结后须重跑，不能据此宣称全完成。
- **仍需关闭：模板2非root真实部署、新浏览器新增功能、插件跨版本/离线失败、长时JWT续期**。主代理进行中，结论与证据待回填；本文件不将此前权限拒绝误写为当前持续阻塞。
- 真实测试仅限已授权的本项目测试资源；工具再次拒绝时不绕过。不得隐式使用实际模型密钥或付费调用，不自动删除未知孤儿容器。
# 默认工作区 /workspace（模板 6，已由下述模板 7 数据根方案取代）
- 当前用户实例已通过 DSH 界面添加并选择 `/workspace`，刷新后仍保留；原 `normal` 工作区和历史会话未移动。
- 空实例初始化通过 DSH workspaceRegistry 注册准确的 `/workspace`，重复启动复用默认 ID；有历史工作区或会话时不强制迁移。
- 真实断网非 root 容器首次初始化、插件加载、备份恢复及旧模板兼容验证通过，详见 `/tmp/dsh-recipe6-test.log`。
- 面板已部署 `dsh-mgr:workspace6`；正常构建 API 记录 `e9a368bbb66e51e6c8f2439a8e94698a` 成功，recipe_rev=6，镜像 `sha256:38e04308da602da22e8df49d54166da1835922d50a3fa485711b7c5f6479d287`。
- 面板部署保留配置和挂载，旧面板停止留作回滚；现有 DSH 实例未重启，构建前后实例记录一致。
## 历史模板7迁移：完整 DSH_HOME 改为 /workspace
> 本节的/shared挂载、共享入口与迁移结果仅描述当时方案；当前共享入口和新容器共享挂载均已移除，实例删除会清理自身数据及备份，详见文首矩阵。
### 管理交互优化验收
- 配额静态展示与模态编辑、常用预设、未改变禁提交；密码明确按钮、原生模态、显示隐藏、即时UTF-8校验、取消清空及焦点恢复。账号创建规则对齐，移动用户卡片和退出分区。
- 前端26测试、生产构建通过。Playwright真实测试账号 uxqa_1791711946466：配额10→20→10 GiB；管理员重设→登录→自助修改→退出→新密码登录均通过；取消清空、焦点恢复、确认不一致禁提交、320/390px对话框不越界通过。
- 测试账号最终重新随机化密码、撤销会话，账号保留；未修改真实账号配额或密码。
- 正式面板 dsh-mgr:ux-polish 已部署，挂载集合核验相同（Docker返回顺序不同），DSH容器953ed34d034f未重启。正式入口重新加载后配额/密码对话框检查通过。
- 截图：output/playwright/ux-before-settings.png、ux-before-admin.png、ux-deployed-settings.png、ux-deployed-quota.png、ux-mobile-quota.png、ux-mobile-password-320.png。浏览器密码管理器用户名提示尚存；不声称全面无bug或完整无障碍合规。
- 依用户澄清，实例 home 直接挂载 /workspace，四个 HOME 环境变量同步；/data/dsh-home 为兼容软链接。旧共享目录保留为 /shared，不合并、不覆盖历史文件。
- 面板文件接口以 instance 参数校验归属、跟随所选实例；旧共享文件保留显式入口。账号总配额不变，恢复期间禁止写入，实例文件请求与升级互斥。
- 当前实例 63b337c848bf23156270068ff9e5cf38 正常升级成功，容器 953ed34d034faed9e0f8c7373964dd0156450b1d65c2965c73a539a83a226934，运行镜像 sha256:bd541770aaaf5a846ecb898ec80863c8a356524eb7f4b8053b0eb646d6e10fad。
- 构建 6c3589998c431acc809b5307c30e8e1e 成功，真实离线 smoke 验证环境变量、旧路径链接及同文件读写。升级中修复宿主路径转换和 pnpm 非交互日志边界；早期失败自动回滚，快照和备份保留。
- 最终验证：DSH 代理 GET 200；实例文件 API 可见 profiles/storages，storages/normal 可见四项历史报告/笔记；旧共享接口可访问。未调用模型或搜索。
- 前端 23 测试/build 通过，后端全量及 vet 曾通过，race 通过。最终定向 Workspace/Upgrade/Orphan 测试通过；最后一次全量受 Docker 磁盘不足 2 GiB 策略影响，3 个构建测试失败，不记为全绿。浏览器交互验证被权限拒绝，未绕过；HTTP 验证不替代浏览器验收。
API.md
-168
View File
@@ -1,168 +0,0 @@
# DSH Web 面板 API（MVP）
默认监听 127.0.0.1:8088。面板处理器通过 fail 返回的错误为 JSON {error:string}；未注册路由的标准 404、静态文件/下载错误及运行时上游响应不保证此格式。面板非 GET 请求（包括 HEAD/OPTIONS）需 X-CSRF-Token 和精确同源 Origin，登录仅要求 Origin（未配置 PANEL_PUBLIC_URL 时按请求 TLS 与 Host 校验）。运行时代理使用下文独立规则。access JWT 有效期5分钟，refresh轮换但不延长 session 的绝对到期时间，session 默认24小时，可用 SESSION_TTL 修改。
除另有说明，成功返回 200；创建用户、实例、备份也不是 201。下述处理器状态码以通过前置认证、Origin/CSRF 和资源检查为前提，前置失败可能优先返回 401/403/409。普通业务接口需要同时有效的 dsh_session 与 dsh_access，不支持以 Authorization Bearer 替代。
- /healthz：无需认证，返回 200 和纯文本 ok；当前实现不限制方法，也不检查 Docker/数据库健康。
- GET /api/runtime-session.js：无需登录，返回 JavaScript、Cache-Control: no-store，用于运行时页面续期。
## 配置
- UPLOAD_ALLOWED_EXTENSIONS：可选逗号分隔扩展名白名单（例如 .txt,.md）；空表示不限制。大小写不敏感，非内容/恶意软件检测。
## 实例备份与快照管理
### 完整 home 快照恢复
`GET /api/instances/:id/snapshots` 的每项另含 `restorable:boolean`、`restore_reason:string`、可选 `source_version` 与 `source_image`。来源记录在独立的 `snapshot_metadata` 表中，不会从当前实例推断旧快照版本。已知镜像或版本不一致时不可恢复；来源未知时 `restorable:true` 但 `restore_reason` 为 `snapshot source image/version unknown; explicit acknowledgement required`，客户端必须显示风险警告并要求额外确认。保护状态及大小读取错误时不可恢复。列表只是预检提示，最终恢复还会检查实时配额、容器布局、镜像准入与读者占用。
`POST /api/instances/:id/snapshots` 接收 `{id,password,confirm:true,allow_unknown_version?:true}`（JSON 最大4096字节，在获取锁前读取）。`password` 是当前登录密码，不是旧备份密码；需正常登录、同源 Origin 与 CSRF。来源未知必须明确传入 `allow_unknown_version:true`；此开关不能绕过已知不兼容。成功返回 `200 {ok:true,snapshot_id:"pre-restore-<32hex>"}`，源快照保持不变。
恢复替换整个 home，不是仅覆盖配置。仅支持经核验的模板7独立 /workspace bind 挂载与当前实例镜像；拒绝旧共享布局、被阻止镜像、恢复保护状态及活跃文件读者。先进行配额和保留数量检查，再使用 rooted 路径验证复制到同文件系统暂存目录；外部/绝对/悬空符号链接及特殊文件拒绝。停止运行时后重新检查配额，将当前 home 重命名保留为 pre-restore 快照，再安装暂存目录并验证就绪；原来停止的实例会临时启动验证后再次停止。失败时独立超时上下文执行回滚，失败的新 home 保留为 failed-restore 快照；无法安全回滚时保持 error，要求管理员恢复。
非法请求或不安全内容400，当前密码错误403，目标不存在/所有权不符404，保护/读者占用/不兼容/缺少未知版本确认/保留上限409，请求或配额超限413，存储/数据库/回滚故障500，容器检查/停止失败502。正常恢复前会记录 snapshot.restore.started，完成或回滚分别记录 snapshot.restore.completed / snapshot.restore.rolled_back。此接口不改变实例镜像。
- `DELETE /api/instances/:id/backups?id=<32hex>` 删除当前用户实例的加密 `.dshbak` 包及其索引，记录 `backup.delete` 审计。ID 为32位小写十六进制；支持无索引的旧包，以及文件已不存在的残留索引。
- `GET /api/instances/:id/snapshots` 返回数组 `{id,size,created_at,protected,protection_reason,size_error?}`。`size` 为普通文件逻辑字节总和（与配额口径相同，硬链接逐路径计数，不跟随符号链接）；扫描失败为 `-1` 并提供 `size_error`，不能显示为零。`created_at` 为目录修改时间的 RFC3339 字符串，不保证真实创建时间。未保护时 `protection_reason` 是空字符串。
- `DELETE /api/instances/:id/snapshots?id=<id>` 删除选定快照。先持久化 `snapshot.delete.started`，删除成功后记录 `snapshot.delete.completed`，删除失败则记录 `snapshot.delete.failed`；target 为实例 ID，detail 为快照 ID。初始审计失败时不会删除，终态审计失败返回500并保留 started 记录。快照另支持 GET 下载，见下方接口清单及 OPERATIONS.md 的归档限制。
仅识别实例目录直属的 `pre-restore-<32hex>`、`failed-restore-<32hex>`、`upgrade-<32hex>-snapshot`、`failed-home-<32hex>` 真实目录；不列出或删除 `home`、其它命名或符号链接目录。删除成功返回 `200 {ok:true}`，非法ID/不安全路径400，不存在/所有权不符404，恢复保护409，存储/索引/审计错误500。不支持的方法405。删除仍要求同源 Origin 与 CSRF，不要求备份密码，不调用 Docker inspect/stop/restart。
实例 `error`、未完成升级（仅 `complete` / `rolled_back` 为终态）、恢复日志查询错误时拒绝两类删除。快照列表仍可读取并报告保护原因。所有权在实例锁与用户锁内重新检查。无自动清理；文件系统和数据库无法原子提交，包删除后的索引提交失败会返回500，重试同一删除可清理残留索引。
快照递归删除期间保留实例锁与用户锁，但不持有 SQL 连接或事务。仅明确识别的非法/不安全路径返回400；权限、I/O 等操作系统故障返回500。快照目录不存在时列表返回空数组，删除返回404。
## 管理员孤儿回收
GET /api/orphans 返回可回收的完整容器ID数组。DELETE /api/orphans?id=<id>&confirm=<相同完整id> 二次核验后非强制删除，保留bind数据及网络。仅接受本项目身份/挂载匹配、已停止、无实例或升级引用的容器；普通用户403，确认不匹配400，状态不安全409。
## 其他配置
- LISTEN_ADDR（默认 127.0.0.1:8088）
- DATA_ROOT（默认 ./data）
- HOST_DATA_ROOT（默认等于实际 CONTAINER_DATA_ROOT 的绝对路径，用于 Docker bind source；容器部署必须显式设置宿主绝对路径）
- CONTAINER_DATA_ROOT（默认 DATA_ROOT）
- DOCKER_HOST（默认 unix:///var/run/docker.sock）
- PANEL_PUBLIC_URL（可选 http/https origin，可带根路径 /，不可带其它路径、用户信息、查询或片段；非法值拒绝启动）
- TRUSTED_PROXY_CIDRS（可选逗号分隔 CIDR；仅信任这些代理的 X-Forwarded-For 来计算登录来源限流，不替代 Origin 校验；非法值拒绝启动）
- SESSION_TTL（默认 24h；Go duration，至少1秒且必须为整秒；非法值拒绝启动，只影响新建 session）
- ADMIN_USERNAME / ADMIN_PASSWORD（首次启动创建管理员；密码必须 12–72 字节，不设置则拒绝启动，不打印密码）
- WEB_DIR（静态前端目录，默认 web/dist）
- PANEL_CONTAINER_ID（面板容器名/ID，用于加入实例网络；宿主进程运行时留空）
- DEFAULT_QUOTA_BYTES（默认10GiB）
- DSH_DOCKER_ROOT_VISIBLE（与DockerRootDir相同路径的明确映射，用于构建前容量检查；未知容量拒绝新构建）
- DSH_RELEASE_REPOSITORY（默认 https://git.misaka.ren/m1saka/dsh-web.git；仅接受无用户信息、查询或片段的 HTTPS 仓库地址）
- DSH_RELEASE_BRANCH（默认 dockerfile；Git 发布检查和构建使用的分支）
## 认证
- POST /api/login {username,password} → {user,csrf_token}；user 字段为 id,username,admin,quota_bytes,created_at
- 登录请求体最多4096字节，非法/多段 JSON 返回400，凭据错误401，Origin错误403；每来源每分钟最多30次、账户累计5次失败锁定15分钟，限流/锁定429，认证不可用或并发槽耗尽503。
- cookie 均为 SameSite=Lax：HttpOnly 的 dsh_session/dsh_access 路径 /，dsh_refresh 路径 /api；另有可供脚本读取的 dsh_csrf（路径 /，非 HttpOnly）。请求使用 TLS 或 PANEL_PUBLIC_URL 为 HTTPS 时设置 Secure。
- POST /api/logout → {ok:true}（不要求有效 access，但仍需有效 session、Origin 和 CSRF；撤销当前会话并清除四个 cookie，不是无条件幂等的匿名退出接口）
- POST /api/refresh → {user,csrf_token}（需 session、refresh cookie、Origin 和 CSRF，不要求 access 尚有效；接受当前 refresh 或紧邻的上一代 refresh，上一代重试幂等返回同一个后继 token 并签发 cookie，不返回轮换冲突409。更旧/无效 refresh 返回401；过期/撤销 session 通常先由 guard 返回403，处理器复查失败也可能401；存储异常在处理器返回500/503，但 guard 的 session 查询失败也会403）
- GET /api/me → {user,csrf_token}
- POST /api/me/password {current_password,password} → {ok:true,reauthenticate:true}（普通用户及管理员仅修改本人；不接收目标 ID，未知 JSON 字段返回 400）。需有效 access/session、同源 Origin 和 X-CSRF-Token；新密码为 UTF-8 12–72 字节，非法 JSON、缺少当前密码或长度不符返回 400。错误当前密码返回 400 而非 401，客户端不应因此退出登录。未登录 401，Origin/CSRF 不符 403（失效 session 在 guard 中也可能返回 403），已认证的非 POST 请求返回 405。
- 与登录、管理员重设共用 authLoginMu，等待后复验会话与当前密码；沿用账户登录失败计数，累计 5 次错误锁定 15 分钟，锁定期间返回 429（即使当前密码正确），已登录不能绕过。认证存储故障返回 503。
- 同一事务保存 bcrypt hash、撤销本人全部 session/refresh、清除失败计数，并记录 user.change_password 审计（操作者和目标均为本人 ID，detail 为空，无密码/hash/token）；事务失败返回 500 并完整回滚，其他用户不受影响。成功后客户端应清理登录状态并重新登录；旧密码、旧 access/session/refresh 不再可用。已建立的 runtime WS/SSE 不主动断开，下一次请求或重连重新鉴权。
## 管理
- GET /api/users → User[]（管理员，按创建时间排序）
- POST /api/users {username,password,admin?,quota_bytes?} → User（管理员；username为3–40个字母/数字/_/-，password为12–72字节；admin默认false，quota_bytes省略或0使用默认配额，负数400；用户名重复或创建存储失败409）
- PATCH /api/users {id,quota_bytes} → {ok:true}（管理员修改正数配额；非法输入400，用户不存在404）
- POST /api/users/reset-password {id,password} → {ok:true,reauthenticate:boolean}（仅管理员；需有效 access/session、同源 Origin 和 X-CSRF-Token）。密码按 UTF-8 字节计为 12–72 字节，以 bcrypt 保存；缺少 ID、非法 JSON 或长度不符返回 400，目标不存在 404，未认证 401，非管理员或 Origin/CSRF 不符 403（失效 session 在 guard 中也可能返回 403），已认证的非 POST 请求返回 405。
- 同一事务更新密码、删除目标全部 session/refresh、清除目标登录失败锁定，并记录 user.reset_password 审计（操作者 ID、目标 ID、时间，detail 为空，不含密码/hash/token）。任何数据库失败返回 500 并回滚，不影响其他用户会话。与登录串行，等待后重新检查管理员会话和权限。
- 允许重设自身，此时 reauthenticate:true，当前会话也立即失效，客户端应清理登录状态并回到登录页；重设他人返回 false。旧密码不能再登录，旧 access/refresh 不能发起新的认证请求。已建立的 runtime WS/SSE 不会被主动立即断开：代理仅在请求/握手时鉴权，下一次请求或重连会被拒绝；现有 runtime 页面刷新认证脚本不是即时断连保证。
- GET /api/audit → [{id,user_id,action,target,detail,created_at}]（管理员；按id倒序最多500条，无分页）
## 镜像与构建
这是 breaking change：`/api/latest`、`/api/compatibility`、`POST /api/builds` 及 `/api/intents`（包括子路径的查询、创建、取消）已移除，不提供 410 兼容入口。`GET /api/builds` 与 `GET /api/builds/{id}` 仅用于构建历史读取。旧构建及旧创建意图不迁移、不恢复执行，不在后台续跑或完成旧预约；不因此删除已有实例或数据。管理员改用 Git 发布接口构建，用户以已验证且准入的精确本地镜像 ID 创建实例。
已认证请求中，`POST /api/builds` 返回 405；`/api/latest`、`/api/compatibility` 和 `/api/intents`（含子路径）返回 404。历史意图记录留在数据库中，但不再提供存储操作、轮询或恢复流程，也不计入配额或作为镜像删除引用。
- GET /api/releases/latest（仅管理员；获取并校验配置的 Git 发布分支，返回 release_id、version、commit、repository、branch、recipe_rev、platforms、compatible、reason，以及可选的 build。检查会下载构建文件，但不执行 Dockerfile；未知部署协议或平台不允许构建）
- POST /api/releases/build {commit}（仅管理员、同源和 CSRF；提交检查时看到的精确 commit，分支已变化返回 409，需重新检查。返回 202 构建任务；不自动升级实例）
- Git 构建任务附带 release_id、source_commit、source_repository 来源信息；发布身份独立于 DSH 版本及部署布局，允许同一 DSH 版本构建新的脚本修订。可信发布通过现有镜像身份和启动验证后可准入，显式禁用仍优先。新构建仅允许管理员通过 Git 发布接口提交；main 不再包含内置 runtime 构建资产。
- GET /api/images（仅返回 Docker 本地实际存在的镜像；每项含 admission、admission_reason、selectable；只有 status=available 且 selectable=true 才可新建/升级。inspect 确认 404 后原子删除镜像索引行并将成功构建记录标为 missing，保留构建历史、显式准入策略和实例；missing 历史不会重建索引。Docker 列表或检查失败时请求失败，不删除索引行）
- POST /api/images/admission {image_id,state,evidence}（管理员、同源和CSRF；state为recommended/compatible/existing_only/blocked，evidence必填。正向准入要求精确镜像身份、成功验证记录、受支持挂载布局与平台；策略与审计持久化。单镜像准入不授权新构建）
- DELETE /api/images?image_id=sha256:...（管理员；保护实例、活动任务、升级及全部Docker容器引用，不force）
- GET /api/builds?limit=20&offset=0（摘要数组，不含日志正文；limit默认20、最大100，offset非负；X-Has-More响应头表示后续是否有记录）
- GET /api/builds/{id}（按需获取单个任务及脱敏日志；仅已登录用户，日志最多64 KiB）
镜像策略：recommended/compatible可在实际验证与身份检查通过后用于新建/升级；existing_only允许已有实例运行和启停，不能新建/作为升级目标；blocked还拒绝新的start/restart及受控配置非GET操作（含Profile保存、备份和恢复），但不会自动停止已运行容器或断开工作台，通用文件管理仍允许。experimental是成功实验镜像未准入时的展示状态，不是兼容性结论。missing/unverified不可选。
历史兼容基线为 linux/amd64、模板7、0.2.0-rc.1（compatible）和0.2.0-rc.2（recommended）。其他平台/版本不继承这些验收结论。当前正向准入仅支持模板7且平台必须匹配面板的 GOOS/GOARCH；模板3–6的历史成功验证记录保留，但不能靠管理员 evidence 正向准入。未知模板布局不可正向准入，模板7向旧共享布局降级仍拒绝。
镜像项字段为 id,image_id,version,status,created_at,usage_count,admission,admission_reason,selectable，以及可省略的 tag,recipe_rev,platform。构建项字段为 id,version,status,created_at,experimental，以及非空时返回的 image_id,error,requested_by,recipe_rev,platform,finished_at,release_id,source_repository,source_commit；详情另有非空时返回的 logs。
发布构建 commit 必须为40位小写十六进制，未知字段或多段 JSON 返回400；相同发布已有 queued/running/validating/success 任务时复用，仍返回202。每管理员一小时最多3个新构建请求，超限429。准入 evidence 不得为空白，原始字符串最多4096 UTF-8字节，成功返回镜像项。镜像删除成功返回 {image_id,status:"missing"}；未索引镜像404，引用/活动任务冲突409，Docker异常502；活动构建或未完成升级会全局阻止镜像删除，不仅限于目标镜像引用。构建分页参数重复、非整数或越界返回400，并非自动截断至100。
## 实例
- GET /api/instances → Instance[]；字段 id,uid,name,version,image_id,status,container_id,created_at
- POST /api/instances {name,image_id} → Instance（每用户最多 2 个，超限409；image_id 必须为 sha256: 加64位小写十六进制且指向已验证、准入的精确本地镜像。version 由镜像标签推导，提交 version 或其他未知字段返回400，不接受构建预约。name 去除首尾空白后非空，原始长度最多200字节；请求体最多64 KiB）
- GET /api/instances/{id} → Instance
- POST /api/instances/{id}/start|stop|restart → Instance
- DELETE /api/instances/{id} → {ok:true}
实例列表仅包含当前用户的实例；管理员也不能通过实例接口访问其他用户实例（404）。列表 Docker inspect 失败时该项 status 为 unavailable，详情 inspect 失败则502。生命周期操作遇到恢复状态或文件读租约冲突409，Docker操作失败通常502。
列表型接口返回顶层数组（不是 data 包装），复合对象内的列表除外。实现与验收仍在进行，不应仅凭接口列在此处认定已验收。
- GET /api/instances/{id}/logs → {logs:string}（一次性脱敏日志，不是 SSE）
## 运行时 HTTP / WebSocket / SSE
- /dsh/{id}/...：校验有效 session/access 和 owner，需同 Docker 网络；不存在或非本人404，恢复状态409，运行时未启动、网络或认证不可用502。/dsh/{id} 重定向至带尾斜杠路径（307）。
- WS/SSE 通过相同前缀透明代理，不是 /api/ws/（后者是文件管理）。WS握手必须携带精确同源 Origin；普通 GET/HEAD 可省略 Origin，但提供时必须匹配；其他方法必须同源。这里不校验 X-CSRF-Token，也不将该头转发给上游。
- 认证只在请求/握手时检查，不因 access 到期、退出或改密主动关闭已建立流。未认证的普通请求/WS返回401；满足浏览器 document/iframe 导航条件的 GET 会303跳转到面板 /#runtime-renew 续期桥。
- 代理剥离面板 Cookie、Authorization、Proxy-Authorization 和查询中的运行时 token，使用服务端交换得到的运行时 Cookie；上游 Set-Cookie 不下发，Location 重写到实例前缀。WS帧及 SSE 事件结构由运行时决定，面板未定义独立消息协议。上游认证失败不会自动重放写请求。
## 工作区
除 usage 外，所有 /api/ws/ 请求必须带查询参数 instance={id}，缺少或空值返回410（共享文件入口已移除）；实例不存在或不属本人返回404，管理员无跨用户豁免。文件根为该实例的完整home；允许下载凭据及目录ZIP，不承诺write-only。下载提示仅在界面防误操作，API不按敏感文件名过滤。通用文件修改不强制走专用Profile的停机/YAML校验流程；归属、路径安全、配额、恢复状态和现有文件操作并发保护仍有效。上传不覆盖同名文件（409）。下列简写路径同样必须补上 instance 查询参数。
- GET /api/ws/tree?path=. → [{name,path,is_dir,size,mod_time}]（按名称排序，目录 size 为0）
- GET /api/ws/usage → {used_bytes,quota_bytes}（当前用户总用量，不是单实例用量；可不带 instance，带上时仍检查归属）
- POST /api/ws/mkdir {path}
- POST /api/ws/rename {path,new_path}
- DELETE /api/ws/delete?path=...
- POST /api/ws/upload multipart file,path
- mkdir/rename/delete/upload 成功均返回 {ok:true}；upload 的 path 是目标目录，instance 仍须在 URL 查询中提供。配额/文件大小限制413，扩展名白名单拒绝415，并发上传限流429；multipart解析失败（含部分超限请求）400。
- GET /api/ws/image?instance=...&path=...：只读图片预览，沿用工作区认证、实例归属检查和实例读租约；仅支持 GET。
- path 与 text 接口一致：保留空白和反斜杠，必须是非空、规范的相对文件路径。Linux openat2 限制在实例 home 内，拒绝符号链接、跨挂载点、多硬链接及特殊文件；不写入文件。
- 有界读取最多 16 MiB 的内存快照，读取期间检测到文件变化返回 409，超限返回 413。PNG/JPEG/GIF/WebP 按 magic 识别；SVG 必须有 .svg 扩展名（不区分大小写），完整 XML 的唯一根元素必须是 SVG 命名空间的 svg，拒绝 HTML、XML directives/DOCTYPE 和未知实体，不加载外部实体。其他格式或目录返回 415，缺失/不安全文件返回 404，非法路径返回 400。
- 返回原始图片字节及 image MIME；固定 `X-Content-Type-Options: nosniff`、`Content-Security-Policy: sandbox; default-src 'none'; style-src 'unsafe-inline'`、`Content-Disposition: attachment`、`Cache-Control: no-store`。SVG 不做内容改写，依靠图片上下文和 CSP 限制脚本及外部资源；直接导航下载附件。
- GET /api/ws/text?instance=...&path=... → {content:string,version:string}
- PUT /api/ws/text?instance=...&path=... JSON {content:string,version:string} → {version:string}
- 仅编辑已有普通文件；沿用工作区认证、实例归属和恢复保护。必须提交 GET 返回的不透明 version；缺少 content/version 返回 400，版本不匹配或保存前文件已删除返回 409，不创建新文件。GET 不存在/不安全路径返回 404；非法相对路径返回 400。
- 内容最多 1 MiB（UTF-8 字节数，含边界），超限 413；无效 UTF-8、二进制/控制字符返回 415（仅允许 tab、LF、CR 控制字符）。JSON 请求上限为 6 MiB + 4096 字节以容纳转义，超限 413。
- Linux openat2 限定访问工作区内路径，拒绝符号链接、跨挂载点和多硬链接文件；不支持 openat2 时安全失败。固定父目录描述符，在同目录写临时文件、保留原文件 mode/UID/GID、同步文件，再原子 rename 替换；不保留扩展属性/ACL。审计仅记录操作和路径，不记录内容。
- 用户锁串行化面板保存；锁内读取当前配额和用量，要求现有用量 + 新内容大小不超配额（包括临时文件开销，否则 413）。因此接近配额时即使缩小文件也可能无法保存。外部进程不受该锁和配额检查约束。
- rename 前立即重新读取目标并核对最新内容/版本，但检查与 rename 不是文件系统 CAS：外部写入者仍可能在两者之间写入或删除目标，不能保证绝不覆盖外部修改。父目录被外部重命名时操作仍针对固定目录。需严格避免丢失修改时，先停止外部写入；不承诺断电后的目录项持久性。
- GET /api/ws/download?path=...：普通文件保持原始流与 Range 支持；目录返回 `application/zip` 附件（目录名.zip），空 path 或 `.` 下载整个工作区（workspace.zip）。ZIP 条目相对所选目录，保留空目录，不额外包裹顶层目录，使用 Store 模式（不压缩），不支持 ZIP Range。
- ZIP 先校验有界文件清单，再逐文件流式输出，不生成临时归档，不持 workspace/global 锁跨网络传输。每次最多 10,000 个文件/目录条目、64 层、10 GiB 文件总大小；进程最多同时 4 个目录下载，超出分别返回 413 / 429；扫描和传输合计限时 2 分钟。
- 路径必须位于工作区内；下载拒绝任意路径分量的符号链接、跨挂载点、特殊文件和多硬链接普通文件。目录含上述条目或含反斜杠/冒号的不安全 ZIP 文件名时整体拒绝（400），不静默漏文件。依赖 Linux openat2，内核不支持时安全失败。
- 下载不是原子快照：流式读取前复核文件身份、大小、类型及修改时间；并发修改、取消或传输错误可能导致连接中断/不完整 ZIP，请停止写入后重试。单文件下载不受 ZIP 数量、大小、并发和时限约束。
## profile / 备份
- GET /api/instances/{id}/profile → {content:string}（配置不存在时 content 为空）
- PUT /api/instances/{id}/profile {content,restart?}（停机原子写；restart=true启动并检查就绪）
- GET /api/instances/{id}/profile-preview（bundle清单和脱敏dump-config，字符串值隐藏；运行中才能dump）
- POST /api/instances/{id}/backups {password}（当前登录密码加密；自动恢复原运行状态）
- POST /api/instances/{id}/backups multipart file,password（导入外部包，password为原包密码；不立即覆盖home）
- GET /api/instances/{id}/backups
- GET /api/instances/{id}/backups?download={backup_id}
- DELETE /api/instances/{id}/backups?id={backup_id} → {ok:true}
- GET /api/instances/{id}/snapshots → [{id,created_at,size,protected,protection_reason,restorable,restore_reason,source_version?,source_image?,size_error?}]
- DELETE /api/instances/{id}/snapshots?id={snapshot_id} → {ok:true}
- GET /api/instances/{id}/snapshots?download={snapshot_id} → 完整 home 的 `.tar.gz` 附件（未加密，可能含凭据）
- POST /api/instances/{id}/snapshots {id,password,confirm:true,allow_unknown_version?} → {ok:true,snapshot_id}（当前登录密码；snapshot_id 为切换前 home 的保留快照）
- POST /api/instances/{id}/restore {backup_id,password}（原备份密码；旧平台key包可留空）
- POST /api/instances/{id}/upgrade {image_id,password}（当前登录密码，加密pre-upgrade备份）
Profile PUT 与 restore 成功返回 {ok:true}，upgrade 返回 Instance。profile-preview 返回 {bundles,config,redacted:true,preview_error?}；无法运行 dump 时仍可200并携带 preview_error，不保证 config 非空。备份列表为 [{id,size,created_at,kind,schema}]，创建/导入返回同形单个对象，download 返回二进制附件。备份数量上限20，达到上限409；恢复快照达到保留上限也返回409，需管理员清理。配置非GET请求体有界读取，超限413、读取超时408、并发槽耗尽503；Profile JSON最多2 MiB，普通备份/restore JSON最多4096字节。
Profile写入要求实例先停止。备份/还原/升级会受控停启；恢复保留原运行状态，升级成功运行新版，失败尝试恢复旧状态，回滚失败需人工处理。运行镜像的插件准备行为由发布分支维护，应针对精确镜像验证。删除实例先清理容器与网络，再永久删除该实例的 home、快照及备份，最后移除实例索引；数据清理失败保留 error 实例以便重试。升级仅支持实例home到 /workspace 的单一bind布局，不迁移旧共享挂载；home回滚不撤销外部系统副作用。
创建恢复：Docker 创建响应丢失时，首次 inspect 404 不代表原请求已结束。实例保留为可见的 error 状态并继续占用名额，用户可重试删除；后台不会仅凭名称删除身份不匹配资源。不确定创建记录不会自动过期：若资源始终不存在，需管理员确认 Docker 原创建请求已终止后再处理保留记录，不能仅凭等待时长判定安全。构建验证容器在创建前持久登记任务身份，启动时即使没有排队构建也会回收并重试失败清理；持续不存在的未知结果仍保留登记以捕获晚到容器。构建终态和镜像索引事务失败会独立重试并记录日志，不阻塞后续构建；数据库关闭停止重试，重启将未完成构建标记失败。
镜像验证、Docker 生命周期、token/cookie 桥接、加密备份恢复和同镜像替换升级已有实现与历史集成记录，具体版本和本次验证范围见 ACCEPTANCE.md，不能据此推断当前全部链路已重新验收。WS/SSE 代理有单元测试，真实完整会话需单独验证。硬配额与跨公网租户域名隔离未实现；已验证的 DSH rc.1/rc.2 启动流程不使用 --public-url。默认仅 loopback 监听，不应直接暴露公网。完整限制见 README.md。
DESIGN.md
-316
View File
@@ -1,316 +0,0 @@
# DeepSeek Harness 多租户容器管理平台 — 设计文档
> 当前发布方式为管理员检查Git发布并确认精确commit，本地构建runtime；release.json、image/与初始化测试仅在dockerfile发布分支维护。旧npm/实验构建和构建预约已移除，详见§5。历史验收不自动适用于当前架构，验证范围见ACCEPTANCE.md，部署操作见OPERATIONS.md。
> 状态：当前代码具备可信团队管理面板核心流程；历史真实 Docker 验收仅适用于各自记录的版本，不代表当前 Git 发布与全部新增功能已验收。本文“当前实现基线”优先于后面的历史设计；未实现规划不作为已交付能力。验收证据与缺口见 ACCEPTANCE.md。
> 目标项目：`deepseek-ai/deepseek-harness`（npm CLI `@deepseek-ai/dsh`，命令 `dsh web`）
> 控制面技术栈：Go + 前端 SPA
> 部署形态：单机 Docker daemon；面板自身亦运行于容器中
> 鉴权/配额：内置账号 + 目录配额
---
## 当前实现基线（本次源码核对）
本节描述仓库代码，而不是部署状态。本次本地测试与构建结果见 ACCEPTANCE.md 当前记录；代码已实现和自动化通过均不等于真实环境已验证。正文标为“当前实现”的章节是现行方案，历史调研与可选规划另行标注。
### 技术栈与历史 Flutter 的边界
- 当前前端是 `web/` 下的 React 19 + TypeScript + Vite，Tailwind CSS v4、Radix 原语的 shadcn 风格组件、lucide 图标及按需加载的本地 Monaco；入口为 `web/src/main.tsx`、`web/src/App.tsx`。登录默认进入内嵌 DSH，桌面侧栏与移动底栏共用响应式页面；主题跟随系统。
- Flutter/Android 属于历史本地工程痕迹：工作区可能仍有 `android/`、`.dart_tool/`、`build/`、IDE 文件，但当前 Git 跟踪文件没有 Dart 源码或 pubspec。它们不是现行应用、构建输入或受支持的 Android 发布产物，不能据目录存在宣称 Flutter 客户端仍在维护。
- 后端模块要求 Go 1.25（`go.mod`），Docker 构建阶段使用 Go 1.26；标准库 `net/http` + 自有 Docker Engine HTTP client（API v1.47），不是官方 Docker SDK。SQLite 使用 `modernc.org/sqlite`，密码为 bcrypt，配置备份采用 Argon2id + AES-GCM。
```text
cmd/server/ Go 启动、静态 web/dist 服务
internal/api/ HTTP、鉴权、发布构建、实例/文件/代理/配置/升级/恢复编排
internal/docker/ 标准库 Docker Engine HTTP client
internal/backup/ 配置打包与加密
internal/store/ SQLite、会话、实例名额事务
web/src/ React 页面、组件、样式与前端测试
scripts/ 本地检查及需显式运行的集成/镜像冒烟脚本
Dockerfile 构建面板及 React 静态资源，不构建 DSH runtime
compose.yaml 当前面板部署模板
```
`internal/store/intents.go` 当前承载实例名额事务，不表示仍有预约 API。runtime 的 `release.json`、`image/` 与初始化测试在 `dockerfile` 发布分支，main 不包含这些构建资产。
### 架构与功能边界索引
现行网络、路径、目录、构建、生命周期与认证详见§2–8；文件/配置/升级见§9–12。特别注意：每实例home独立，实例删除会清理自身数据及备份；共享文件入口与构建预约均已移除；旧不确定创建“宽限期后404释放”策略已被替代。
### 当前发布与准入流程（详见 §5）
1. 管理员通过 `GET /api/releases/latest` 检查配置的 Git 发布仓库/分支及 manifest；通过 `POST /api/releases/build` 确认精确 commit。分支已变化返回冲突，需重新检查。
2. 构建任务绑定仓库、commit、release ID、版本、部署协议与平台；执行时重新验证信任源和管理员权限，获取锁定 commit 的上下文，本地构建并验证，保留分页历史及详情日志（`releases.go`、`releases_git.go`、`images.go`、`builds_read.go`）。不自动升级已有实例。
3. 镜像准入仍区分推荐、兼容、仅存量与禁止；验证记录及实际镜像身份必须匹配。当前 `supportedLayout` 仅接受协议/模板 7（`admission.go`），不能套用历史“模板 3–6 可正向准入并使用双挂载”的记录。
4. 普通用户仅选择可用本地镜像。旧 latest/compatibility、写 builds、全部 intents 接口已经移除；实验构建、构建预约、旧任务续跑不属于当前能力。
### 当前部署、未实现规划与验证边界
- 当前 `compose.yaml` 为面板 UID/GID 1000、socket 附加组、只读根、tmpfs、资源限制，默认发布 `127.0.0.1:8088`；需配置数据根、DockerRootDir、公开 URL 与可信代理。部署说明见 §16、仓库实际 Compose 和 OPERATIONS.md。
- XFS/ZFS 硬配额、特权配额 sidecar、socket-proxy、子域名入口属于未交付的可选设计，不是现行默认能力；实际公网 TLS/代理、真机与长期稳定性需独立验收。软配额不能阻止容器直接写盘瞬时超限。
- `scripts/check-trial.sh` 当前检查 Go 测试/race/vet/build、前端测试/build 和集成脚本语法，不自动运行真实 Docker、浏览器或发布分支初始化测试。完整新 Git 发布链路及部署验证的通过状态以 ACCEPTANCE.md 的当前矩阵为准，不由历史 M1–M7 结案推导。
---
## 详细设计与历史调研
下文当前实现章节已同步；§0为历史上游调研，§9中的硬配额为可选规划。上游链接及版本调研未在本轮重新验证，不能据此推断未来发行版行为。
## 0. 历史上游调研结论与硬约束
> 实现阶段实测更正：本地 npm `0.2.0-rc.2` 支持 `--trusted-host`，但拒绝 `--public-url`；下文基于 upstream master 的 public-url 描述不能直接用于该发行版。实际启动仅传 web/no-open/port/trusted-host，代理负责外部路径。`scripts/probe-runtime.mjs` 已验证默认 token 交换为 303 + cookie、认证首页 200；完整代理以集成测试为准。
针对本体 `deepseek-ai/deepseek-harness`（npm CLI `@deepseek-ai/dsh`，命令 `dsh web`）确认：
| 事实 | 来源 | 对设计的影响 |
|---|---|---|
| **无官方 Docker 镜像/Dockerfile** | repo 根、`apps/`、`.gitlab-ci.yml` 均无 | 必须自建镜像包装 `dsh web` |
| **`dsh web` 默认监听 `127.0.0.1:3080`，且启动时拒绝 `--host 0.0.0.0`**（安全原因） | `packages/bundle/web-app/README.md` | 容器内必须跑反向代理绑 `0.0.0.0` 转发到 loopback |
| 端口/主机是 **CLI flag**（`--port/--host/--public-url/--trusted-host/--no-open`），**无 `DSH_PORT/DSH_HOST` 环境变量** | 同上 | 容器启动参数化靠 CMD args，不靠 env |
| 数据根 = `$DSH_HOME`（默认 `~/.dsh`） | `docs/user/guide/providers.md`、CLI README | 每实例独立 `DSH_HOME`，bind-mount 即可保留数据 |
| **多实例硬约束：每个容器必须独立 `DSH_HOME` + 独立端口**（task-board 单写者） | architecture.md | 一用户两容器 ⇒ 两个 `DSH_HOME`，不能共享 |
| 模型/API Key 存储：密钥在 `$DSH_HOME/.credentials.yaml`（YAML，上游配置接口的 write-only 语义不限制本面板所有者文件访问），模型/provider 配置在 `$DSH_HOME/profiles/<profile>/cordis.patch.yml`；`cordis.patch.yml` 只存 `apiKeyEnv` 引用 | `providers.md` | 备份"模型"= 备份这两个文件 |
| **"profile 文件" = `profiles/web/cordis.patch.yml`（YAML，用户可编辑的 patch 层）**；profile 目录还有 `package.json`（含插件依赖 + `dsh.profile` bundle 清单）+ `node_modules`（pnpm 装的插件，可重建） | `apps/cli/README.md` | 编辑 profile = 编辑 `cordis.patch.yml`；插件清单在 `package.json` |
| 升级：用户数据全在 `DSH_HOME`（npm 安装树之外），升级 `@deepseek-ai/dsh` 后数据保留；`node_modules` 由 pnpm 重建 | README、CLI README | "保留数据 + 重启新版容器"可行 |
| 公网部署：`--public-url <url/mount/>` + `--trusted-host`；代理必须 **剥前缀、保留 Host、把 `Path=/` 的 cookie 改写为 mount + 加 Secure** | `docs/user/guide/public-deployments.md` | 面板反代要处理 cookie/前缀，是最复杂点 |
| 启动 URL 带 **per-process token**，首次访问换签名 cookie | 同上 | 面板需从容器日志取 token，服务端代换 cookie |
> 结论：DSH 不是为容器多租户设计，但靠"自建镜像 + 容器内反代 + 面板反代桥接 token/cookie + 每实例独立 DSH_HOME"可稳态支撑。
---
## 1. 目标与范围
### 已确认的访问与镜像策略
- 面向可信团队，实例所有者可通过文件管理读取、下载、修改完整 DSH_HOME，包括自己的凭据。面板不承诺凭据 write-only；跨用户隔离、路径安全、配额与恢复状态保护保持不变。下载文件/打包目录会提示泄密风险，提示属于交互防误操作，不是阻止所有者访问的安全边界。
- 专用 Profile 编辑器提供 YAML 校验与受控停启流程，但通用文件操作、DSH 终端和 agent 不强制经过该流程；修改运行中配置存在不一致风险，建议先停机并备份。
- 镜像准入分为推荐、仍兼容、仅存量运行、禁止使用。兼容依据绑定精确版本、模板修订、平台与镜像身份；不再仅因历史模板不是当前模板就一律判为不可用，也不能仅因历史构建成功就认定兼容。
- 新版本仅通过管理员 Git 发布流程构建；实验构建已移除。管理员正向准入须填写验收依据并记录审计，不绕过布局、平台、实际镜像身份与验证记录检查。
- 通过 docker.sock 在单机上**创建/启停/删除**多个 `dsh web` 容器，**每用户最多 2 个**。
- 每实例独立完整 home，支持文件上传/下载/删除/重命名、文本编辑和图片预览，按账号统计软配额。
- 实例 home 挂入自身容器的 /workspace，不与同账号其他实例共享挂载。
- **备份/还原 dsh 配置**（含模型与插件清单）。
- **编辑 profile 文件**（`cordis.patch.yml`）。
- **升级**：保留数据，用新版本镜像重启容器。
- **本地镜像管理**：管理员检查 Git 发布并确认精确 commit 构建；用户选择经验证且准入的本地镜像创建，不提供构建预约。不推送生成的镜像。
- 控制面：Go 后端 + SPA 前端；内置账号 + 目录配额；**面板自身容器化运行**。
---
## 2. 总体架构（当前实现）
面板是通过 docker.sock 编排 sibling runtime 的 Go 容器，提供 React 静态资源、API、路径反代和 SQLite 持久化。每个实例独立 bridge，面板接入对应网络，runtime 不发布主机端口。容器内 DSH 监听 loopback:3080，由发布镜像内 Caddy 在8080转发；镜像内部实现须以精确发布 commit 为准。
浏览器 → 面板 SPA/API 或 /dsh/<id>/ → 实例专属 bridge → runtime:8080 → DSH:3080。
socket-proxy 仍为可选规划，当前 Compose 直接挂 socket，不代表具备宿主级隔离。
---
## 3. 宿主路径 ↔ 面板内路径映射（当前实现）
Docker bind.Source 必须使用 daemon 宿主路径。面板文件操作使用 CONTAINER_DATA_ROOT（Compose 为 /data），通过 internal/api/paths.go 的 toHostPath 映射到 HOST_DATA_ROOT。
| 用途 | 面板路径 | Docker bind.Source / Target |
|---|---|---|
| 实例完整 home | /data/workspaces/<uid>/instances/<id>/home | <HOST_DATA_ROOT>/workspaces/<uid>/instances/<id>/home → /workspace |
| 配置备份 | /data/backups/<uid>/<id>/ | 不挂入 runtime |
构建上下文是从锁定 Git commit 获取并提交的归档流，不应把历史内置 image/ 当作宿主 bind 源。
---
## 4. 数据与目录布局（当前实现）
```
<HOST_DATA_ROOT>/
db.sqlite
workspaces/<uid>/
instances/<instId>/home/ # 完整 DSH_HOME → /workspace
.credentials.yaml
cordis.patch.yml
profiles/web/ # 配置、插件清单、可重建依赖
storages/ # 实例工作区数据等
files/ # 可能遗留；当前无共享入口或新容器挂载
backups/<uid>/<instId>/ # 加密配置备份；不是完整 home 快照备份
```
账号软配额统计 workspaces/<uid>/，并非只统计当前实例。实例删除会清理该实例目录及对应备份；历史共享 files 不自动合并到 home。旧目录结构和旧删除保留策略不能作为现行承诺。
---
## 5. 自建 DSH 镜像（当前 Git 发布流程）
管理员检查 GET /api/releases/latest，再以 POST /api/releases/build 确认精确 commit；分支变化必须重新确认。后端从受信任仓库锁定 commit 获取 release.json 与 image/，校验部署协议7及当前面板平台，构建本地镜像并验证。构建记录绑定仓库/commit/release ID/版本/平台，任务去重、限频、容量检查与失败记录仍保留。GET /api/builds 及 GET /api/builds/{id} 提供历史与日志。
不再从npm查询latest，不接受普通用户实验构建，不打包main内置image/，不提供构建后创建预约。创建实例必须另行选择经验证且准入的本地image ID；管理员构建成功不等于已完成完整兼容验收或自动升级实例。
镜像准入绑定实际身份和成功验证记录；supportedLayout只接受模板7，平台须匹配当前面板平台，内置defaultAdmission规则另有限定linux/amd64。旧模板记录可以保留，不意味着可通过填写说明正向准入。镜像清理保护有效引用，不自动全局prune。构建仍可能需要基础镜像和依赖网络，不能宣称离线构建。
---
## 6. 容器生命周期（当前实现）
标准库 Docker HTTP client 使用 API v1.47。创建固定 image ID，事务限制每用户最多两个实例，无构建预约。每实例独立网络与 home，UID/GID 1000，工作目录 /workspace，DSH_HOME/HOME 同为 /workspace，XDG 路径位于其内。启动参数为 web --no-open --port 3080 --trusted-host <host>，不传 --public-url。
当前资源约束为只读 rootfs、drop ALL capabilities、no-new-privileges、2GiB 内存、2 CPU、256 PID 及有界可执行 /tmp。实例支持启停、重启、日志、删除；删除回收容器/网络并清理实例数据与备份。创建不确定时保留恢复依据，不能靠一次或超时后的404清除。孤儿资源仅管理员明确确认并核验后处理，不自动纳管。
## 7. 网络与访问方式（当前实现）
默认已经选择路径前缀代理，不是待拍板选项。面板将 /dsh/<id>/ 转发至对应实例网络中的 runtime:8080；归属检查在代理之前。Compose 默认仅向 loopback 发布面板8088，公网需管理员另配 HTTPS 入口与可信代理。子域名、直接发布 runtime 端口不是当前默认能力。
## 8. 访问与认证（当前实现）
面板校验账号和实例归属，从启动日志取得 token 并在服务端交换/缓存 DSH cookie；token 按容器身份/启动时间加密持久化。反代剥前缀、处理 Host 与转发头，注入服务端 cookie 并移除上游 Set-Cookie，不把 DSH cookie 改写下发给浏览器。客户端持有的是面板认证 cookie。
WS/SSE 由反代支持；会话续期由面板/runtime 注入脚本协调。上游拒绝 cookie 时失效缓存，下次请求重新交换，不重放写请求。已建立 WS/SSE 不承诺即时撤销；真实部署的长会话、插件与公开入口仍需分别验证。
## 9. 工作区文件管理 + 容量配额
**文件 API**（除 usage 外必须提供 instance 并校验归属；path 限于该实例 home，拒穿越、绝对路径和不安全符号链接）：
- `GET /api/ws/tree?path=` 列目录（带 size/mtime）
- `POST /api/ws/mkdir`、`POST /api/ws/rename`、`DELETE /api/ws/delete`
- `POST /api/ws/upload`（有界上传、写前查配额；不是分片续传协议）
- `GET/PUT /api/ws/text`（文本读取/带版本冲突检查的保存）、`GET /api/ws/image`（受控图片读取）
- `GET /api/ws/download?path=`（单文件流；目录自动 zip）
- `GET /api/ws/usage`（当前用量/配额）
**配额实现**（容器化面板的约束）：
- **默认：软配额**。面板写前遍历统计账号目录用量，写入串行化并限制并发，每五分钟巡检记录超限。容器/agent直接写盘不受面板写锁约束，仍可超过软配额；不通过删用户数据恢复容量。
- **可选硬配额**（二选一，需运维确认）：
- **XFS prjquota**：面板 `--privileged` + `cap_add:[SYS_ADMIN,CHOWN]` + 把宿主 XFS 挂载点 bind 进来，面板内对 `/data` 跑 `xfs_quota`/`chattr -p`。每用户唯一 project ID（uid+1000）。
- **ZFS refquota**（若宿主用 ZFS）：面板 `--privileged` + zfsutils，每用户 `zfs create pool/users/<uid> && zfs set refquota=10G`，再 bind 该 dataset 到 `/data/workspaces/<uid>`。
- 或在宿主跑**极小特权配额 sidecar**（gRPC/HTTP），面板远程调它做 quota 操作——面板自身保持非特权。
- 配额值存 DB，按用户可配（默认 10GiB）。面板展示用量。
> 当前仅软配额；上述 XFS/ZFS 和特权 sidecar 是未实现规划，不是可直接启用的配置。
---
## 10. 配置备份与还原（含模型与插件）
**备份内容**（tar.gz，**排除可重建/运行态**）：
| 包含 | 说明 |
|---|---|
| `.credentials.yaml` | 模型 API Key（机密）→ 备份包加密 |
| `cordis.patch.yml`（home 级） | home patch |
| `profiles/web/package.json` | **插件清单**（含 dsh.profile bundle 列表） |
| `profiles/web/cordis.patch.yml` | **模型/provider 配置** + apiKeyEnv 引用 |
| `profiles/web/pnpm-lock.yaml`（若有） | 插件版本锁 |
| `skins/ pets/ agent-presets/ .agent-presets/` | 用户资源 |
| `dsh-ssh.json` | SSH 凭据（0600，加密入包） |
| `manifest.json` | dsh 版本、备份 schema 版、时间戳 |
**范围**：仅上述六个固定配置文件、四类资源树和生成的manifest；不是完整home备份，通常位于白名单外的工作区、会话和插件node_modules不会入包。资源树按路径递归纳入普通文件，不根据文件内容识别运行态或凭据；不安全路径、符号链接及特殊文件会被拒绝。
**备份包加密**：用用户密码派生密钥（Argon2id → AES-GCM）加密整包；密钥不落库。下载/上传均加密包。
**备份流程**：按受控流程停止原本运行的容器，打包白名单并加密落盘，再恢复原运行状态；支持下载。不是任意运行中文件的一致性全量快照。
**还原流程**：校验并解密包，受控停机、保存home恢复快照，应用配置与协调插件，启动验证后恢复应有运行状态；失败尝试快照回滚，失败恢复可能需人工介入。实际快照命名和恢复记录以config.go及运维文档为准，不提供任意第三方插件兼容保证。
---
## 11. profile 文件编辑
- 编辑对象：`profiles/web/cordis.patch.yml`（YAML）。
- 前端用本地 Monaco；后端 `GET` 返回原文，`PUT` 校验 YAML 后原子写入，专用编辑器修改前需停止实例。未实现的 schema 比对不作为已交付能力。
- 页面“保存并启动”发送 `restart:true`，写入后启动并检查就绪；API省略该参数时仅保存，不自动启动。
- 只读展示 `dsh.profile`（bundle 清单）与 `--dump-config` 预览（面板 `docker exec` 跑 `dsh --profile web --dump-config` 取回渲染）。
- 专用配置预览保持脱敏；完整 home 文件管理允许实例所有者读取 `.credentials.yaml` 原文，不承诺 write-only。
---
## 12. 升级流程（保留数据 + 重启新版容器）
```
1. 用户选择本地可用目标版本；若无目标镜像，先由管理员完成 §5 Git 发布构建。锁定已验证且准入的 image ID 后才进入备份/停机流程
2. 停止原本运行的旧容器并检查确实已停机（不 remove，保留作回滚）
3. 创建有界home快照，再用当前登录密码创建 pre-upgrade 配置备份
4. 保留旧容器回滚依据，创建替换容器：同逻辑实例、同home挂载、新image，不挂共享files
Cmd 使用 web/no-open/port/trusted-host，不传 --public-url
5. 启动新容器；发布镜像入口负责插件协调，其离线能力须针对精确镜像验证
6. 面板抓 token 并验证就绪
7. 成功删除旧容器；失败尝试清理新容器、恢复home快照及旧容器原运行状态，回滚失败保留恢复依据并提示人工处理
```
历史设计意图：保持 bind 并重建插件依赖以尽量保留数据；**这不保证零丢失或任意插件跨版本兼容**。当前升级还依赖备份、快照、健康检查和回滚，故障及兼容性必须实测。
---
## 13. 安全与资源
- **docker.sock**：面板容器能访问 sock 即等效宿主 root。当前非root进程和只读rootfs不能消除该权限；socket-proxy白名单只是尚未实施的可选增强。
- dsh 容器：**不 `--privileged`**、去 CAP_NET_RAW 等、只读 rootfs、tmpfs /tmp、mem/cpu 限制、不挂 sock。
- 面板账号：bcrypt + JWT（短期 access + refresh），失败锁定。
- 配置备份包加密；实例所有者通过文件 API 下载的凭据或目录 ZIP 不属于加密配置备份，应按机密数据保管。
- 路径穿越校验、上传大小/类型白名单（可配）。
- 审计日志：容器操作/备份/还原/升级/profile 改动入库。
---
## 14. 技术栈与项目结构（当前实现）
当前 Go/React 栈和实际目录见文首基线：Go 模块版本1.25、net/http、自有 Docker HTTP client、modernc SQLite；React 19、TypeScript、Vite、Tailwind v4、Radix 原语、Monaco。不是 Flutter 客户端，也不是独立 auth/workspace/proxy/upgrade 包的原设计拆分；多数编排位于 internal/api。
旧设想中的 main/image/、deploy/ 不能用于当前构建或部署：runtime 资产在 dockerfile 发布分支，面板使用根目录 Dockerfile 和 compose.yaml。
---
## 15. 数据模型（当前实现概要）
SQLite基础表users、sessions、instances、audit由internal/store初始化；认证轮换、构建/镜像准入、发布身份、runtime创建/网络恢复和升级恢复等表由相应API模块维护。实际字段与迁移以源码为准，不将早期设计的示意DDL当作数据库接口。
image_builds包含任务、验证及发布来源信息；image_admissions绑定精确image ID/version/recipe/platform及验收依据；runtime_creates/runtime_networks保留资源恢复依据。旧表或文件名存在不代表旧预约服务仍运行。
---
## 16. 面板自身部署（当前实现）
以根目录 compose.yaml、.env.example 和 OPERATIONS.md 为操作入口，不复制历史80/443或共享bridge样例。默认面板绑定127.0.0.1:8088，运行用户1000:1000，附加宿主socket组；只读rootfs、/tmp tmpfs、1GiB内存、2 CPU、512 PID及健康检查。
配置 HOST_DATA_ROOT / CONTAINER_DATA_ROOT、DockerRootDir只读容量探测挂载、PANEL_CONTAINER_ID，以及实际公开URL和可信代理CIDR。裸docker.sock挂载并未被只读rootfs降权，仍需信任面板与账号。当前模板不部署socket-proxy或硬配额sidecar。
---
## 17. 关键风险与待验证
1. **token/cookie 桥接**（§8）：需在实际入口验证服务端cookie交换与注入、前缀代理及WS/SSE，不以本地模拟替代部署验收。
2. **插件首启重建**：跨 dsh 版本 `pnpm install` 在 entrypoint 无网时失败——可能需升级时由面板 `docker exec` 联网重建后再切流量。
3. **XFS/ZFS 硬配额**依赖宿主 FS 与特权/agent；否则退化为软配额。
4. **dsh agent 工作目录**：`dsh-fs-local`/`dsh-fs-sandbox` 的根是否等于 cwd——需确认 `/workspace` 作为 cwd 即可让 agent 在工作区操作，或需额外配置 fs root。
5. dsh 版本仍为 `0.2.0-rc.*`（RC），API/flag 可能变动——需维护版本兼容矩阵。
6. **sibling bind 路径翻译**：`toHostPath` 出错会导致 bind 挂错目录或挂空目录，需单测覆盖。
7. **Git 发布构建**：须验证精确 commit 确认与分支变化、信任源/权限变化、构建去重及恢复、外部删除镜像、容量及网络故障。构建成功不等于完整兼容验收，不存在旧构建预约释放名额流程。
---
## 18. 已决策与后续规划
已选择每实例独立bridge、路径前缀反代、服务端cookie桥接、账号软配额、React响应式Web及管理员Git发布本地镜像。旧npm最新构建/预约、共享工作区和Flutter发布不是当前路线。
可选后续增强：XFS/ZFS硬配额或宿主配额sidecar、socket-proxy、独立域名隔离。上述未纳入当前默认交付；是否实施需另行设计、授权和验收，不能以文档列出视为支持。
## 19. 里程碑与交付判定
M1镜像、M2账号/生命周期、M3代理、M4文件/软配额、M5配置备份、M6升级回滚、M7界面/管理安全保留为能力分组，不再以旧npm构建及预约作为当前M2标准。当前Git发布、实例home独占、删除清理、文本编辑/图片预览与主题等变更须按ACCEPTANCE.md当前矩阵分别补证据。
历史M1–M7结案不等于当前commit或真实部署整体通过；本地测试通过、真实Docker通过、浏览器通过与公网验收必须分开记录。
---
## 附：调研依据（关键 URL）
- 仓库：https://github.com/deepseek-ai/deepseek-harness
- README：https://raw.githubusercontent.com/deepseek-ai/deepseek-harness/master/README.md
- CLI README：https://raw.githubusercontent.com/deepseek-ai/deepseek-harness/master/apps/cli/README.md
- web-app README：https://raw.githubusercontent.com/deepseek-ai/deepseek-harness/master/packages/bundle/web-app/README.md
- providers 指南：https://raw.githubusercontent.com/deepseek-ai/deepseek-harness/master/docs/user/guide/providers.md
- 公网部署指南：https://raw.githubusercontent.com/deepseek-ai/deepseek-harness/master/docs/user/guide/public-deployments.md
- npm 包：https://registry.npmjs.org/@deepseek-ai/dsh/latest
Dockerfile
-24
View File
@@ -1,24 +0,0 @@
FROM node:24-bookworm-slim AS web
WORKDIR /src/web
COPY web/package*.json ./
RUN npm ci
COPY web/ ./
RUN npm run build
FROM golang:1.26-bookworm AS backend
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY cmd/ ./cmd/
COPY internal/ ./internal/
RUN CGO_ENABLED=0 go build -trimpath -o /out/dsh-mgr ./cmd/server
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates git && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=backend /out/dsh-mgr /usr/local/bin/dsh-mgr
COPY --from=web /src/web/dist ./web/dist
USER 1000:1000
EXPOSE 8088
HEALTHCHECK --interval=30s --timeout=5s --start-period=15s --retries=3 CMD ["/usr/local/bin/dsh-mgr", "-healthcheck"]
ENTRYPOINT ["/usr/local/bin/dsh-mgr"]
OPERATIONS.md
-155
View File
@@ -1,155 +0,0 @@
# 本机或可信团队试运行运维
当前仅支持模板 7 的独立实例布局：实例 home 挂载到 /workspace，不再创建或使用 workspaces/<uid>/files，也不再迁移旧共享挂载布局。历史 files 目录可在确认旧容器已停止并移除挂载后，由管理员在维护窗口删除；面板不会在启动或查询用量时自动递归删除历史数据。旧版本已经删除实例所留下的目录也需核对数据库及容器引用后单独清理。
本版本面向本机或受信任团队，不应直接用于公网不可信租户。DSH 与面板使用同源路径代理；Docker socket 权限等价于宿主 root。
## 启动
1. 复制 `.env.example` 为 `.env`，填写专用绝对目录 `DSH_HOST_DATA_ROOT` 和 12–72 字节的随机管理员密码。限制 `.env` 的读取权限，不要提交。管理员账号创建后该密码不再被读取，应从 `.env` 中移除，再运行 `docker compose up -d --force-recreate panel` 重建面板容器，清除容器环境变量中保留的明文密码（`docker inspect` 可见）。仅修改 `.env` 或重启原容器不会更新其环境变量；重建面板会短暂中断面板访问。
2. 面板以 UID/GID 1000 运行。首次部署前由管理员创建专用数据目录并赋予 1000:1000 所有权、0700 权限；不要对已有未知目录递归改权限。将 `stat -c %g /var/run/docker.sock` 的结果填入 `.env` 的 `DOCKER_GID`，以附加组访问 socket。然后运行 `docker compose up --build -d`。默认访问 `http://127.0.0.1:8088`；远程管理优先使用 SSH 本地端口转发，保留此访问地址。Compose 定义了 `dsh-mgr -healthcheck` 健康检查（探测本机 `/healthz`）；连续探测失败时容器会被标记为 unhealthy，但该状态本身不会触发重启。`restart: unless-stopped` 用于进程退出后的自动拉起；`/healthz` 仅检查HTTP存活，不检查数据库、Docker或实例健康。
3. 管理员登录后在镜像页检查 Git 发布并确认精确 commit 后构建；普通用户选择已验证且准入的本地镜像创建实例。没有可选镜像时等待管理员构建，不再按 npm 版本或预约构建后创建。镜像不推送至仓库。
4. 创建实例后从面板进入 DSH。不要给 runtime 发布宿主端口或挂载 Docker socket。
首次启动创建管理员。修改环境变量不会重设已有账号密码。容器内 `/data` 必须对应 `HOST_DATA_ROOT` 指向的同一宿主目录，否则 sibling 容器会挂载错误数据。
新镜像构建前必须检查 Docker 存储空间：将 `docker info --format '{{.DockerRootDir}}'` 返回的绝对目录填入 `DSH_DOCKER_ROOT_DIR`。Compose 将该目录以相同路径只读挂载，面板只用 statfs 检查至少 2 GiB 可用空间；这不是授权浏览或修改 Docker 内部文件。映射无法确认、不可访问或空间不足时，新构建拒绝，已有本地镜像复用仍可进行。不要用面板数据盘冒充 Docker 存储盘。
## 公网 HTTPS 入口（仅可信账号）
反向代理与证书由部署管理员管理，面板仍建议只监听代理可达的私有入口。设置 `PANEL_PUBLIC_URL=https://你的域名`（仅 origin，不含业务子路径）；Compose 默认值仍为本机 HTTP。此值用于 Origin/CSRF 校验，HTTPS值会启用 Cookie 的 Secure 属性（直接TLS请求也会启用），不从客户端的 X-Forwarded-Proto/Host 推断。
默认只按直连来源 IP 限流，不信任 X-Forwarded-For。经反代部署时，设置 `TRUSTED_PROXY_CIDRS` 为面板实际看到的、逗号分隔的代理来源CIDR（单个IPv4/IPv6地址须写成 `/32` 或 `/128`），且确保只有这些代理能连接后端、代理正确维护转发链；不要填写全网 CIDR。未设置时，代理后所有用户将共享该代理来源的每分钟登录额度。公网开放前应实际验证不同客户端来源识别、登录/刷新/退出、DSH iframe、WS/SSE及插件设置接口。
账号由管理员创建，不支持不可信租户。软配额、同源插件与 Docker socket 权限边界不因 HTTPS 而改变；本版本不承诺抵御分布式拒绝服务。
## 配置与全量备份
实例所有者有完整 home 读写权限，包括自己的凭据。通用文件管理、DSH 终端及 agent 不受专用 Profile 编辑器的 YAML 校验或停机流程限制；这不是 write-only 凭据系统。文件下载和目录 ZIP 可包含明文密钥，不等同于下述加密配置备份。提示可帮助避免误操作，但无法阻止所有者导出凭据。共享下载文件前应自行检查，修改运行中配置前建议先停机和备份。
面板提供加密配置与资源备份：固定配置、插件清单/锁文件、skins、pets、agent-presets、.agent-presets及manifest；不包含会话、node_modules或工作区。新包用创建时的登录密码经Argon2id派生密钥、AES-GCM加密，不依赖或创建 `backup.key`。导入/恢复使用原备份密码；修改登录密码不会重加密旧包。旧版密钥包仍兼容，只有这类包依赖原 `backup.key`，应单独安全保管，不能因升级删除。所有包均可能含凭据。
配置备份自动停止原运行实例并恢复运行；恢复先校验包，再快照、写入、启动验证，失败尝试回滚，成功保留原运行/停止状态。当前模板7启动前按清单协调插件；历史跨版本及离线失败验证范围见 ACCEPTANCE.md，不保证任意第三方插件兼容。配置内容上限32 MiB/4096文件，备份入口最多保留20个包；快照和保留上限需管理员监控清理。此流程不能代替全量备份。
要获得一致的全量备份，先通过面板停止所有实例，再停止面板，使用宿主备份工具复制专用数据目录（包含数据库、可能存在的 WAL 文件、工作区、实例 home、快照和密钥）。只停止面板不会自动停止 sibling 实例。恢复时保留权限和所有权，将整个目录恢复到同一部署的配置路径；不要将运行中的数据库文件单独覆盖。备份应在隔离环境定期验证。
## 升级与故障
### 镜像兼容验收与准入
管理员 Git 发布构建会执行基础验证，但不能代替完整上线验收。准入前应在独立测试环境核验精确镜像的启动、认证/代理、终端与流式连接、文件读写、插件协调、配置备份恢复，以及所需升级/回滚路径；记录版本、模板、平台、镜像ID、测试范围和证据位置。不要在验收依据中粘贴凭据，也不要仅填写“构建成功”。本次实现不自动启动待验收镜像供普通用户使用，不自动运行真实部署测试。
在镜像页“准入管理”选择推荐或兼容并填写依据，后端还会检查精确镜像身份、成功验证记录和可支持的挂载布局。审批只作用于该镜像；新镜像需要自己的验证与审批。退为仅存量后不允许新建/升级目标；标为禁止使用还阻止新的启动/重启及受控配置修改、备份/恢复写操作，避免其自动重启绕过禁用。已有配置读取和通用home文件管理仍保留，不会自动停止正在运行的实例。若需要立即隔离已运行实例，管理员必须另行确认并执行停止操作。
当前准入仅支持模板7，使用完整home作为 `/workspace`；旧模板的独立 `/data/dsh-home` 与用户共享 `/workspace` 布局不再支持准入。升级也要求原容器只有实例home到 `/workspace` 的单一bind挂载，不会迁移旧共享布局；不符合时返回409。处理历史实例前先做好完整备份。
运行时认证材料由面板首次成功连接实例时提取，并使用 `auth-signing.key` 派生的密钥加密保存在数据根的 `runtime-auth/`（不在用户home内）。按容器 ID 和启动时间绑定，不会将旧启动周期凭据用于新容器；不同 Host 分别缓存 cookie。全量备份应连同签名密钥和该目录保存，不要向用户公开。
运行时返回401/403时会使该Host的cookie缓存失效，下次请求重新交换。不会自动重放写请求，避免插件操作重复执行；遇到认证失败请重新加载页面后确认操作结果。若runtime自行轮转了启动token且旧token也不可交换，需要管理员处理，不能保证任意上游认证机制变更自动恢复。
升级到本版本后应在维护窗口逐个访问现有实例，完成首次认证持久化。若在首次成功连接前，启动token已被Docker日志轮转删除，本版本无法凭空恢复；先保存工作，再由管理员安排重启相关实例并访问验证。不要为此删除或重建实例。初次日志扫描上限64 MiB，常规每文件10 MiB/3份保留在此范围内。
- 实例升级选择已经启动验证且准入的本地模板7镜像并提供当前登录密码，面板自动停机、创建密码配置备份与home快照、启动验证及失败回滚，无需先手动停止。home快照最大1 GiB；软链接按链接本身保存，保留相对、绝对、失效及循环链接的目标文本，不读取或备份链接指向的外部内容；仍拒绝 socket、FIFO 等特殊文件。模板7的 `/workspace` 是实例home，包含在home回滚内；旧共享挂载布局会在升级前被拒绝。跨DSH版本历史验证范围见 ACCEPTANCE.md。
- 升级快照先复制到 `.upgrade-<ID>-snapshot-partial`，完成后才发布正式目录并记录完成状态。复制失败会清理临时目录；清理失败会报告并记录 `upgrade.snapshot.cleanup_failed`，管理员核实后处理。进程中断也可能留下临时目录，不会作为可恢复快照列出。已发布但未记录完成的快照会显示不可恢复。旧版残留没有可靠的完成记录，本版本不会自动判定其完整或自动删除，建议保留证据并重新生成快照。
- 完整快照恢复和下载同样保留软链接原文；失效链接不会自动修复，外部目标不属于恢复范围。下载的 tar.gz 可能包含绝对或目录外链接，应在隔离目录使用安全解压工具，不要跟随链接写入宿主机。独立的加密配置备份仍只接受白名单普通文件，配置路径和资源中的软链接仍会被拒绝。
- Profile是例外：当前需先停止实例，再使用“保存并启动”；页面包含Monaco和脱敏配置预览。不要把备份/升级自动停启理解为Profile可在线改写。
- 若实例进入 `error`，不要直接修改数据库状态强制启动。保留原容器、实例目录与 `upgrade_journal` 记录，检查实际容器与 journal 的旧/新容器 ID、阶段、快照路径是否一致。
- 人工恢复前停止相关容器并另做完整备份。中断点不同，不能统一用一次目录覆盖或状态修改处理；当前无自动灾难恢复命令。
- 删除实例会删除其容器、网络、实例目录（含 home 和快照）及专属备份，无法撤销；请事先下载需要保留的数据。清理失败保留 error 实例，修复存储问题后重试删除。未完成升级恢复的实例仍禁止删除。
## Git 分支发布运行镜像构建配置
发布源默认为本仓库 `https://git.misaka.ren/m1saka/dsh-web.git` 的 `dockerfile` 分支，通过 `DSH_RELEASE_REPOSITORY` / `DSH_RELEASE_BRANCH` 配置。它发布的是构建文件，不是预构建镜像，也不是面板程序自身的更新。修改配置后需重新创建面板容器。
直接运行面板二进制时需要安装 `git`；面板 Dockerfile 已包含该依赖。当前通道仅支持无需认证的 HTTPS 仓库，不读取宿主 Git 凭据、不跟随重定向，也不支持 SSH 或 URL 内嵌令牌。私有仓库认证需另行扩展，不能把访问令牌写入仓库地址。
`release.json` 和完整的 `image/` 目录仅在 `dockerfile` 发布分支维护，main 不再保留或打包这些文件。发布分支根目录放 `release.json`；发布时一起提交 Dockerfile、Caddyfile、entrypoint 和全部安装/适配脚本。只把 Dockerfile 单独放到分支上是不完整的发布。不要把 `.env`、私钥、凭据、用户数据或 `.git` 放进构建目录；`.dockerignore` 只是辅助措施，发布者仍须审查内容。
Git 通道直接向 Docker API 发送构建上下文，不执行 Docker CLI 的 `.dockerignore` 过滤。面板会排除 `.git`、`.env` 系列和常见凭据文件，并拒绝链接、特殊文件及越界路径，但不能识别任意文件名下的秘密。构建归档最大 32 MiB；Git 获取有 45 秒超时及文件大小限制，建议发布分支仅保留发布所需文件。
`release.json` 示例：
```json
{
"schema_version": 1,
"release_id": "runtime-7.1",
"version": "0.2.0-rc.2",
"deployment_schema": 7,
"platforms": ["linux/amd64"]
}
```
`version` 是精确 DSH 版本；`release_id` 是构建配置发布编号，修改脚本或 Dockerfile 后即使 DSH 版本不变，也应使用新的编号。`deployment_schema` 声明运行布局兼容协议，当前 Git 发布通道支持协议 7：实例 home 是唯一bind挂载，目标为 `/workspace`，不挂载共享目录 `/shared`，代理端口为8080，并保留现有认证与启动约定。它不应随普通构建修订递增；不兼容的运行约定需要面板新增支持。`platforms` 只能声明发布者实际验证的平台。
首次发布时，在独立 checkout 或 worktree 准备 `dockerfile` 分支，将 `release.json` 和完整 `image/` 目录提交并推送到同一远程仓库。无需切换正在开发的主工作目录，也无需 CI。此功能的代码安装本身不会替你创建或推送远程分支；分支未发布前，检查失败是预期状态，旧实例不受影响，但不会回退到内置构建。这是 breaking change：移除 `/api/latest`、`/api/compatibility`、`POST /api/builds` 及 `/api/intents` 的全部查询、创建和取消接口，不提供 410 兼容入口。仅保留 `GET /api/builds` 和 `GET /api/builds/{id}` 构建历史读取。不迁移或恢复旧构建及旧创建意图，不在后台续跑或完成旧预约；这不意味着删除已有实例或数据。
管理员打开镜像页会检查 Git 发布，页面保持打开时每五分钟复查，也可手动刷新。确认构建后面板以固定 commit 获取文件；确认期间分支发生变化会拒绝并要求重新检查。构建任务记录发布编号、仓库和 commit，同一 DSH 版本的新构建发布不会误用旧版本构建任务。构建与验证成功后才可选择镜像；不会自动升级任何已有实例，升级仍走密码确认、备份、快照和失败回滚流程。
该分支是可执行代码信任边界：限制写权限，发布前审查 Dockerfile 与脚本，不向不可信仓库开放构建。可信发布只替代内置版本名单，不替代完整的上线验收；应在发布前验证启动、代理/认证、文件访问、插件以及升级恢复。管理员对精确镜像设置的禁用策略仍然有效。上游依赖、基础镜像和网络变化可能影响重建结果，commit 固定并不意味着所有外部依赖都已固定。Git 故障不会停止已有实例。
## 创建与构建恢复记录
创建失败不代表 Docker 没有创建资源。面板在发请求前记录 `runtime_creates`（实例 ID、容器/网络结果待确认标志及其置位时间）及 `image_build_checks`（构建 ID、镜像 ID、已确认的验证容器 ID、记录创建时间），已确认网络身份保存在 `runtime_networks`。资源可见后按身份校验再清理；已记录 ID 时按 ID 查询，不能把名称消失当成资源删除。身份不一致时拒绝删除。构建验证残留在启动时及后台重试回收，无需新建构建任务触发。
创建请求失联后，确定性名称的 404 仍视为结果不确定，不会仅因经过一分钟而清除恢复依据：实例保持 `error` 并占用名额，资源迟到出现后可重试删除并按身份清理；验证容器的不确定记录保留并后台降频重试、审计记录 `build.cleanup_error`，不会每秒新增审计。即使恢复记录的时间戳缺失或损坏，也不能据此推断创建已结束。一直未出现的资源需要按下述人工确认流程处理，不能保证所有失联创建自动释放预约。删除实例成功后总是清理其 `runtime_creates`/`runtime_networks` 行。数据库暂时不可写时，构建终态事务独立重试，不阻挡后续构建；面板退出后未提交的任务按中断失败处理，不承诺恢复为成功。
人工确认流程（仅部署管理员，不能因单次404直接改库）：
1. 停止面板接收新操作并停止面板进程，备份数据库（含 WAL）及相关恢复记录。确认 Docker daemon 健康，并结合 daemon 日志确认原创建请求已结束；必要的 daemon 维护会影响其他容器，应单独安排。
2. 根据精确实例/构建 ID 核查 `dsh-inst-<instance-id>` 网络与容器，或 `dsh-build-check-<build-id>` 验证容器；已记录容器 ID 时还必须按 ID 查询，不能仅按名称判断不存在。核对归属标签、镜像及名称，不运行全局 prune。
3. 资源存在且身份一致时，优先恢复面板并走正常删除/自动回收；身份冲突则保留记录，先调查冲突来源。
4. 只有确认原请求终结、对应资源确实不存在后，才可在维护备份基础上用 SQLite 事务清除该精确 ID 的待确认标志，或删除该精确 build ID 的 `image_build_checks` 行。不得批量清空恢复表，不修改实例为 running。重新启动面板后重试正常删除，并核对实例名额、构建状态与 Docker 资源。
## 容量与网络
实例文件请求只在检查归属和登记使用状态时取得生命周期锁，上传/下载期间不持全局生命周期锁。同一实例存在文件操作时，启停、删除、配置修改、恢复及升级返回 409，等待传输结束后重试。文件树单目录最多列出 10000 项，超限返回 413；用量与快照遍历仍有总条目/深度上限。此限制不是文件系统硬配额。
用户软配额覆盖工作区与实例目录，已接入上传、Profile增长、恢复/升级快照的写前检查及写入串行化，并每5分钟巡检记录超额/恢复审计；覆盖范围见 ACCEPTANCE.md。它不能阻止DSH直接写盘或插件安装增长。监控数据目录和Docker存储空间，快照也消耗容量。硬quota需宿主文件系统和管理员配置，是可选增强，不是默认路线必做项。
每个实例配置独立bridge、非root、只读根文件系统和资源上限，但并非完整网络安全边界：未限制访问宿主、局域网或互联网。公网不可信租户需另外设计独立origin、可信TLS入口与出站策略，并按威胁模型评估硬配额。不要给runtime发布端口；server-held DSH cookie不下发浏览器，不能代替面板归属鉴权。
## 验收
按版本分层记录的验收结果见 [ACCEPTANCE.md](ACCEPTANCE.md)。以下命令是检查入口；历史通过记录不能替代当前版本回归，自动化测试不能替代实际公网域名与浏览器验收。
```sh
bash scripts/check-trial.sh
```
该脚本执行本地自动化测试、race、静态检查及前后端构建，不启动容器、不安装依赖、不部署。需提前准备 Go/Node/npm 和前端依赖。可通过 `TEST_TMPDIR` 指定已存在的临时目录。面板构建单测已隔离真实磁盘空闲量；生产磁盘检查仍保留。`image/init-profile.test.mjs` 属于发布分支的 runtime 测试，不在 main 的检查脚本中运行。
`scripts/test-panel.mjs` 必须指向独立测试部署，通过 `TEST_PANEL_URL`、`TEST_USERNAME`、`TEST_PASSWORD` 指定访问参数；`TEST_UPGRADE=1` 包含同镜像替换升级。运行前需由管理员通过 Git 发布构建准备已验证且准入的本地 runtime 镜像；可用 `TEST_FROM_VERSION` / `TEST_TO_VERSION` 选择源和目标版本。脚本不提交构建，只复用这些镜像，会创建及删除测试实例，删除时一并清理实例数据。不要对生产账号运行。`scripts/smoke-image.sh <image-id>` 和 `scripts/test-mobile-image.mjs <image-id>` 仅测试已有镜像的运行行为，不读取 main 的构建源；仅在隔离测试环境使用。旧构建脚本和构建后创建意图的 live 测试已删除。
## Office 文档预览超时
Office 文档预览的运行时接口 `/api/officeToPdf/render`（面板路径为 `/dsh/:id/api/officeToPdf/render`）使用独立代理连接池，响应头等待上限为120秒；其它运行时请求及认证仍保持原有20秒上限。此设置避免转换尚未完成就被面板提前截断为502，不修改DSH转换器自身的超时（当前镜像默认60秒），也不保证转换一定成功。浏览器取消请求仍会传递至上游。若面板前还有Nginx等代理，其等待上限也应允许转换完成。此修复只需更新面板，无需为此重建运行镜像。
## 手动备份空间管理
实例备份包和历史完整工作区快照必须由用户显式删除，不执行自动清理。快照列表与删除接口见 API.md。删除快照会降低工作区逻辑用量；加密包在独立 backups 目录，不计入当前工作区软配额。快照大小不等于物理磁盘占用，硬链接按路径重复计数；`size_error` 表示无法完整扫描，应显示未知而非零。
恢复失败、未完成升级及恢复日志不可读时，列表仍报告保护状态，但禁止删除包和快照。先完成恢复并核实运行状态和升级日志，不要为了释放空间直接修改日志状态。`home` 永远不属于历史快照管理对象。删除不需要停机或 Docker 可用，也不会触发重启。
文件删除与 SQLite 提交不具备跨系统原子性。若加密包已删除而索引提交失败，接口明确返回500；重试相同删除会清理残留索引。快照删除先持久化 `snapshot.delete.started`，失败则不触碰目录；递归删除期间不占用 SQL 连接或事务，仍保留实例锁和用户锁。随后分别记录 `snapshot.delete.completed` 或 `snapshot.delete.failed`。终态审计写入失败返回500；只有 started 的记录也可能表示进程中断或操作仍在进行，不代表删除成功，应由管理员核实目录和审计记录。目录删除中途遇到存储错误可能部分完成，排除故障后再重试。明确不安全路径返回400，权限或 I/O 故障返回500，缺失快照删除返回404（列表为空）。
## 完整快照下载与手动恢复
实例的备份面板提供完整快照下载和恢复入口。下载的是未加密 `.tar.gz`，不同于仅包含配置资源的加密 `.dshbak`；可能包含密钥、会话和依赖，请妥善保管。归档保留权限及安全的相对符号链接，不跟随链接读取外部文件，遇到特殊文件或逃逸链接整体拒绝。下载先在系统临时目录完成归档，再释放实例锁传输；临时文件在请求结束时清理，需预留临时磁盘空间。文件内容总量上限1 GiB、100000条目、128层，另有限额容纳归档头及压缩开销；归档和传输分别有两分钟时限，共用最多4个目录归档下载槽。
恢复针对服务器保留的快照，不支持上传任意 tar 包恢复。需当前登录密码和明确确认；源快照保留不动，先复制到暂存目录，再停机并保留当前 home，切换后启动检查，失败尝试回退。原先停止的实例检查后仍停止。恢复需要额外容纳一份所选快照的配额和磁盘空间；已有恢复快照达到3个时先清理。已知源镜像或版本与当前实例不一致时禁止恢复；历史快照缺少来源信息时须额外确认风险。新升级、配置恢复和完整快照恢复会记录来源信息。
error 状态、未完成升级、活跃文件操作、禁用镜像或不兼容挂载布局会阻止恢复。恢复中断或回退失败保留 error 状态及数据，需管理员结合 `snapshot.restore.*` 审计核查 home、pre-restore、failed-restore 和暂存目录；不要直接把状态改成 running。快照只恢复 home，不回滚镜像、外部服务或启动产生的外部副作用。
### 完整快照恢复安全说明
快照恢复会替换整个实例 home，必须由所有者重新输入当前登录密码并明确确认。仅恢复到相同的已知镜像与版本；历史快照没有可靠来源记录时必须显示未知版本风险并额外确认，不得把当前实例版本当作历史来源。旧升级日志的 image_id 是目标镜像，不是旧 home 的可靠来源。新快照的来源写入 SQLite snapshot_metadata 表，升级回滚留下的 failed-home 使用尝试升级的目标镜像来源。
源快照不会被消耗。操作将源复制到实例目录下的 .snapshot-restore-<32hex>，确认 Docker 已停止后，把原 home 重命名为 pre-restore-<32hex>，再以同文件系统重命名安装副本并 fsync 父目录。运行时启动并通过就绪验证后恢复原运行/停止状态。原来停止的实例也会临时启动验证，因此应把恢复视为可执行该快照内容的操作，仅恢复可信数据。
开始停止/切换前实例状态持久化为 error；中途进程退出不会自动清除此状态。错误恢复时先停止容器，核对 snapshot.restore.started 审计中的源、pre-restore、failed-restore ID，以及 home 和暂存目录。不要盲目删除或覆盖这些目录；先确认哪一个保存原 home，保留全部恢复证据与源快照。只有核对目录、挂载和运行时就绪后才解除恢复状态。无法停止、重命名、同步或回滚就绪时接口保持 error 并要求人工处理。
恢复前按源副本大小检查用户配额，停止运行时后再次核验总用量；原 home 通过重命名保留，不重复复制。已有 pre-restore / failed-restore 数量达到3时拒绝，需显式清理不再需要的快照。没有自动保留清理；进程异常退出的隐藏暂存副本须在维护窗口核验后清理。文件复制期间不持有数据库事务。此流程仍需部署环境提供同文件系统原子重命名及 fsync 持久性，不能替代异机备份。
## 会话有效期
在 `.env` 设置 `SESSION_TTL=24h`（默认），七天可设置为 `168h`。使用 Go duration 单位 `h`、`m`、`s`，不支持 `d`；必须至少一秒且为整秒，非法值会阻止启动。修改后运行 `docker compose up -d --force-recreate panel`。新登录使用新期限；已有会话保留原绝对到期时间，刷新不会延长。access token签发有效期为五分钟，但数据库会话过期或被撤销后，即使token尚未到期也不能认证。修改自己的密码或管理员重设密码会撤销该用户所有会话，需要重新登录。`.env` 由 Compose 读取，直接运行二进制时应在进程环境中导出 `SESSION_TTL`。
README.md
-225
View File
@@ -1,225 +0,0 @@
# DSH Web Manager
面向本机或受信任团队的DSH容器管理面板，使用Go、React和SQLite。目标见 [DESIGN.md](DESIGN.md)，接口见 [API.md](API.md)，部署和恢复见 [OPERATIONS.md](OPERATIONS.md)。不适合直接作为公网不可信多租户平台。
本项目定位为本机或可信团队试运行，可由管理员通过 HTTPS 反代提供公网入口；不支持不可信租户。选型差异及真实验证边界见 [ACCEPTANCE.md](ACCEPTANCE.md)。
## DSH 工作台与移动端
登录后默认进入内嵌 DSH 工作台。在“设置”中选择当前账号的实例；选择仅保存在当前浏览器并按账号区分，不保存登录凭据。没有实例时先到“实例”创建，停止或异常的实例需先启动或处理。切换实例会重新加载内嵌页面，但不会停止原容器。
移动端使用底部导航：DSH、实例、文件、设置；桌面保留侧栏，管理员可直接从侧栏进入镜像与用户管理，移动端这两个入口仍在“设置”中。切换面板栏目会保留内嵌 DSH 页面，重新加载与独立窗口入口收在工具栏的溢出菜单中。此为同源 iframe，不是安全隔离边界，仍仅适合本机或受信任团队。
## 当前实现
文件管理中的 PNG、JPEG、GIF、WebP、SVG 可通过“预览”按钮查看（最大 16 MiB），支持适应窗口、原始尺寸、缩放和下载。预览沿用实例归属校验，SVG 只作为图片加载，响应限制脚本与外部资源；直接打开预览地址会下载文件，不作为网页执行。
文件管理支持通过“编辑”按钮在线查看和保存不超过 1 MiB 的 UTF-8 文本，使用本地 Monaco 编辑器并跟随系统深浅色。二进制或超限文件请下载处理。关闭、重新加载及离开浏览器前会提醒未保存修改；保存失败保留草稿。保存按读取版本检查冲突，不提供强制覆盖；原子替换需要额外的临时文件配额。DSH 等外部写入不受面板锁控制，最终检查与替换之间仍有竞态，修改运行配置前建议停机并备份。通用编辑不执行专用 Profile 流程的 YAML 校验或强制停机。
DSH 新运行镜像仅由管理员从本仓库 `dockerfile` 分支获取独立发布的构建配置，无需 CI 预构建镜像。`release.json` 和完整的 `image/` 目录仅维护在发布分支，main 不再内置运行镜像构建资产；面板锁定 Git commit，在本地构建并验证。管理员在镜像页检查并确认构建，实例升级仍需另行确认，不会因分支更新自动停机。发布约定和首次启用步骤见 [OPERATIONS.md](OPERATIONS.md)。
实例所有者可管理完整 home，包括读取自己的配置和凭据；不承诺密钥 write-only，跨用户隔离保持不变。通用文件操作不受专用 Profile 编辑器的 YAML 校验和停机要求约束。下载、打包与敏感修改提示用于防误操作，不是权限限制；文件下载和目录 ZIP 不会自动加密，修改配置前建议先停止实例并备份。
- 内置账号、管理员配额、短JWT与refresh轮换、失败锁定及审计。
- 管理员 Git 发布检查、本地镜像异步构建/验证/清理、使用已验证且准入的本地镜像创建实例与事务名额（每用户最多两个实例）、启动对账。
- 实例启停删除、独立home（模板7挂载为 `/workspace`）、每实例bridge、归属鉴权及路径反代；DSH认证cookie由服务端保管。
- 文件上传/下载/重命名/删除及目录ZIP；软配额写前检查、写入串行化与周期巡检。上传最大32 MiB且不覆盖已有文件，删除目录须先清空；删除实例会永久删除其 home、快照及备份。
- 密码加密配置备份、资源目录与manifest、外部加密包导入、恢复健康检查及回滚。备份只包含六个配置文件及 `skins/`、`pets/`、`agent-presets/`、`.agent-presets/` 资源目录，不是整个 home 的备份；会话、普通工作文件及node_modules不在备份范围内。
- 备份/恢复/升级自动编排停启；升级保存密码配置备份和home快照，失败尝试回滚。Profile使用本地Monaco及脱敏预览；目前仍需先停止实例，页面“保存并启动”随后验证就绪。
新备份使用创建时的登录密码经Argon2id派生密钥、AES-GCM加密，不依赖或创建 `backup.key`。导入/恢复须使用原备份密码，修改登录密码不会重加密旧包。旧版密钥包仍兼容，保留这些包时必须保管原 `backup.key`。
## 启动面板
需要Linux 5.6+（openat2/renameat2）、Go 1.25+、Node/npm及本机Docker daemon。Docker socket权限等价宿主root。
```sh
npm --prefix web ci
npm --prefix web run build
go build -o bin/dsh-mgr ./cmd/server
# 首次启动设置ADMIN_USERNAME及12–72字节强密码ADMIN_PASSWORD
./bin/dsh-mgr
```
默认访问 http://127.0.0.1:8088，数据在 `./data`。修改环境密码不会重置已有账号。容器部署使用 `.env.example` 和 `compose.yaml`，按运维文档准备UID/GID 1000数据目录、Docker socket附加组及Docker存储目录映射，再执行 `docker compose up --build -d`。Compose仅发布loopback。
## 公网 HTTPS 反向代理（Nginx / Caddy）
以下两种方案任选一种，假设反代运行在面板所在宿主机，域名为 `dsh.example.com`，后端为 `http://127.0.0.1:8088`。将示例域名、证书路径替换为实际值。仅支持独立域名根路径部署，不要挂在 `/panel/` 等子路径下；所有路径（包括 `/api/` 和 `/dsh/`）统一转发到面板，不直接代理或公开 runtime 的端口。公网入口仍仅供可信账号使用，不是多租户安全隔离方案。
### 面板环境变量
使用 Compose 时在项目 `.env` 中设置；直接运行二进制时需通过进程环境传入（二进制不会自动读取 `.env`）：
```dotenv
PANEL_PUBLIC_URL=https://dsh.example.com
# 仅适用于反代和面板二进制均在宿主机、通过127.0.0.1连接的情况
TRUSTED_PROXY_CIDRS=127.0.0.1/32
```
`PANEL_PUBLIC_URL` 必须与浏览器访问的 origin 一致，不带路径；非标准 HTTPS 端口需包含端口号。它控制 Origin/CSRF 校验和 Secure Cookie，不依赖 `X-Forwarded-Proto` 推断。修改环境变量后需重启面板进程；Compose 部署应执行 `docker compose up -d panel` 使容器读取新环境，单纯 `docker compose restart` 不会更新环境。
注意 Docker 网络差异：宿主机反代访问 Compose 发布端口时，面板看到的来源可能是 Docker 网桥网关，而不是 `127.0.0.1`。此时应将 `TRUSTED_PROXY_CIDRS` 改为确认过的实际代理来源 IP `/32`（IPv6 用 `/128`），不要照抄上述值，也不要信任 `0.0.0.0/0`、`::/0` 或整个不受控内网。未配置信任时，转发头会被忽略，所有用户可能共享代理 IP 的登录额度。可通过后端连接观测确认来源，Docker 网络 inspect 只能提供候选网关信息，不能替代实际确认。
如果反代也运行在容器内，`127.0.0.1` 指向反代容器自身，不能使用下面的后端地址。应让反代与面板加入专用 Docker 网络，将后端改为 `panel:8088`（实际服务名），并按面板看到的代理容器地址设置可信来源；不要在同一受信网络放入不可信容器。后端8088和runtime端口不应直接暴露公网。
### Nginx
示例可放入由 `http {}` 引入的 `/etc/nginx/conf.d/dsh.conf`；`map` 必须位于 `http` 上下文，不能放在 `server` 或 `location` 内。证书需提前申请并配置续期；示例不负责签发证书。
```nginx
map $http_upgrade $dsh_connection_upgrade {
default upgrade;
'' close;
}
server {
listen 80;
server_name dsh.example.com;
return 308 https://dsh.example.com$request_uri;
}
server {
listen 443 ssl;
server_name dsh.example.com;
ssl_certificate /etc/letsencrypt/live/dsh.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/dsh.example.com/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
# 为multipart封装留余量；应用仍执行自己的上传/备份上限
client_max_body_size 64m;
client_body_timeout 60s;
location / {
proxy_pass http://127.0.0.1:8088;
proxy_http_version 1.1;
proxy_set_header Host $http_host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $dsh_connection_upgrade;
# 单层公网边缘代理：覆盖客户端提供的转发来源，不能原样信任
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $http_host;
proxy_buffering off;
proxy_request_buffering off;
proxy_cache off;
proxy_connect_timeout 10s;
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
send_timeout 3600s;
}
}
```
`Upgrade/Connection` 支持 WebSocket，关闭响应缓冲可及时传送 SSE；读写超时指相邻读写的空闲时间，不是保证连接永久存活。不要改写或删除 `Origin`、Cookie、`X-CSRF-Token`，不要额外添加阻止同源 iframe 的 `X-Frame-Options: DENY`。
保存后先检查，再由管理员重载：
```sh
sudo nginx -t
# 仅在检查成功后执行
sudo systemctl reload nginx
```
### Caddy
将以下内容加入 `/etc/caddy/Caddyfile`。域名 DNS 应指向该主机，80/443 需按证书签发方式可达；Caddy 默认自动申请/续期 HTTPS 证书并将 HTTP 重定向到 HTTPS。
```caddyfile
dsh.example.com {
request_body {
max_size 64MB
}
reverse_proxy 127.0.0.1:8088 {
# 单层公网边缘代理，覆盖外部传入的XFF
header_up X-Forwarded-For {http.request.remote.host}
flush_interval -1
transport http {
dial_timeout 10s
}
}
}
```
Caddy 自动支持 WebSocket，默认保留请求 Host，并设置转发协议信息；`flush_interval -1` 用于及时刷新流式响应。此处不设置较短的响应读取/流生命周期超时，避免中断长时间 SSE/终端连接。Caddy 重载可能关闭现有 WebSocket，客户端需重连。
```sh
sudo caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
# 仅在检查成功后执行（使用systemd安装时）
sudo systemctl reload caddy
```
以上示例均假设客户端直接连接这层反代。若前方还有 CDN、负载均衡或第二层代理，不能直接依赖这里的 `$remote_addr` / `remote.host` 得到真实客户端 IP；需要在边缘代理单独配置严格的可信上游及来源解析，并相应调整转发策略，否则用户会共享上一层代理的限流额度。
### 部署后检查
- 确认 HTTP 跳转 HTTPS，证书有效，公网只能到达反代而不能直接访问面板8088或runtime端口。
- 通过实际域名完成登录、刷新、退出，确认无 Origin/CSRF 403，认证 Cookie 带 Secure；直接使用后端HTTP测试不能替代HTTPS验收。
- 打开内嵌 DSH、终端/WebSocket、SSE及插件设置；空闲后重连、登录续期均需验证。
- 验证文件上传/下载及备份导入；应用限制仍有效，反代64MB上限不代表单文件可上传64MB。
- 使用不同真实客户端确认登录限流来源隔离；测试应使用专用账号，避免锁定实际账号。
这些是配置模板，不代表已在你的公网域名部署或验收；应用侧测试和运维边界见 [ACCEPTANCE.md](ACCEPTANCE.md) 与 [OPERATIONS.md](OPERATIONS.md)。
## 用户密码重设
所有已登录用户（包括管理员）可在「设置 → 账号 → 修改密码」输入当前密码、新密码及确认新密码。当前密码必须验证通过；成功后撤销该账号全部旧会话并返回登录页。忘记当前密码时仍需联系管理员，不提供匿名找回入口。
管理员可在「设置 → 用户与审计 → 用户列表 → 重设密码」为用户设置新密码（12–72 UTF-8字节，二次输入及确认）。重设会撤销目标账号全部登录会话和refresh令牌、清除登录失败锁定；旧密码不能再登录，审计仅记录操作者和目标，不记录密码。重设管理员自身会返回登录页。已建立的DSH WebSocket/SSE连接不保证在重设瞬间断开，后续HTTP请求、连接重建与会话续期会被拒绝。
## 本地runtime构建
### Git 发布与镜像准入
镜像的构建验证与使用准入分开管理：推荐/兼容镜像还必须通过精确身份与实际验证检查才可选择；仅存量镜像允许原实例继续运行和启停，不可新建或作为升级目标；禁止使用镜像还拒绝新的启动/重启，但不会自动停止现有容器或断开工作台。
管理员在镜像页检查 Git 发布并确认精确 commit 后构建。普通用户只能选择已验证且准入的本地镜像创建实例；没有可选镜像时需等待管理员发布与构建，不再提供 npm 版本、实验构建或构建后自动创建入口。这是 breaking change：移除 `/api/latest`、`/api/compatibility`、`POST /api/builds` 及 `/api/intents` 的全部查询、创建和取消接口，不提供 410 兼容入口。仅保留 `GET /api/builds` 和 `GET /api/builds/{id}` 构建历史读取。不迁移或恢复旧构建及旧创建意图，不在后台续跑或完成旧预约；这不意味着删除已有实例或数据。准入绑定精确镜像ID、版本、模板和平台，不自动信任同tag的新镜像。
历史模板保留其精确身份验证记录，不因模板号不是当前版本就标为缺失。但当前新建/升级及正向准入只支持模板7的挂载布局，且镜像平台必须匹配当前面板平台；旧模板即使有成功验证记录也不能靠管理员填写说明放行。内置默认推荐/兼容策略针对 `linux/amd64`，其他平台不能据此推断已经验收。存量实例的启停仍受实际资源状态与禁止使用策略约束；界面展示具体准入原因，以后端校验为准。
当前模板为修订7，同时内置 `[email protected]` 与 `[email protected]`（MIT，https://github.com/DDDMUC/dsh-free-search）。Free Search及其普通依赖固定版本、校验SHA-512并保留镜像内原始tarball；新profile默认正式安装并启用，已有profile不自动注入。模板4的mobile恢复路径继续保留。仅包含内置固定依赖的profile在启动前离线安装，不限制运行期插件管理器联网。
Free Search默认引擎为Bing，注册的统一provider ID为ddg；未设置或等于deepseek-official的搜索provider会被它接管（包括显式选择同名官方provider），默认http抓取也会被接管，其他自定义provider保留。名称含free不等于保证零费用：自动回退可能读取已有API凭据并调用计费服务。请在插件设置中按需禁用引擎；fallback顺序不是引擎白名单。内置集成不自动测试搜索、配置密钥或执行在线更新。
插件设置页的八个固定API路径由面板在实例归属、登录和同源检查后适配前缀及loopback Host，不开放匿名根路径接口。用户在插件内更新到其他版本可能改变接口或离线依赖，需重新验证兼容性。旧实例需显式安装/启用；升级镜像不会覆盖其已有插件选择。
模板修订4默认内置 `[email protected]`（MIT，来源 https://github.com/mexiaosqwq/dsh-web-mobile）。构建校验固定SHA-512、不执行插件安装脚本；新建web profile使用镜像内tarball作为正式依赖并默认启用bundle，可在Plugins的Installed列表查看及启停。为兼容旧profile仍保留宿主副本；UI可能显示Uninstall，但本轮未验证卸载行为，不承诺移除镜像内置副本。已有profile不会被覆盖或强制启用；模板3旧实例须在升级后补正式依赖登记，仅升级镜像不会自动出现在Installed列表。该插件按触屏能力启用移动布局，仅缩窄桌面窗口不一定触发。
依赖固定引用 `file:/usr/local/lib/dsh-mgr/dsh-web-mobile-3.0.5.tgz`，首次安装与备份恢复无需下载该包。备份包含依赖和bundle清单，恢复目标必须提供相同tarball路径（模板4及兼容后续镜像）；初始化会在此依赖存在且workspace缺失时补建官方no-auto-peers配置，保留已有workspace及禁用状态。
运行镜像的构建源与初始化单测在 `dockerfile` 分支维护；main 只构建面板，不再提供 `scripts/build-image.sh`。镜像构建仍需要基础镜像与依赖网络访问，不推送镜像。
注意：dsh-web-mobile还包含响应压缩、移动会话删除入口和自定义模型推理档位补全，并非纯CSS。内置版本随镜像固定，不跟踪GitHub main或npm latest。
模板4目前限定DSH 0.2.0-rc.1/rc.2，并校验已安装宿主的Cordis/DSH peer版本；其他版本需先更新兼容验证，不会自动安装第二套宿主依赖。插件没有独立参数配置页；它修改DSH原有移动界面与设置页。
```sh
bash scripts/smoke-image.sh "$RUNTIME_IMAGE_ID" # 已由管理员构建的本地镜像 ID
```
`scripts/smoke-image.sh` 只检查已有镜像运行，不读取源码、不构建镜像；需 Docker 和 Bash，会创建并清理临时容器。请从镜像页取得精确镜像 ID，不依赖旧版本 tag 推断 Git 发布身份。当前部署协议为7。runtime以UID/GID 1000运行，home和workspace需允许该身份读写。DSH监听loopback:3080，Caddy监听容器内8080；不要发布runtime端口或挂载socket。原生模块需要可执行的 `/tmp`，保留nosuid/nodev和容量限制。存在插件清单时先协调插件，失败不启动应用；真实验证范围见 ACCEPTANCE.md。
## 设计替代与边界
构建历史接口现默认分页（20条，最大100条），列表不再携带日志正文；日志通过 `/api/builds/{id}` 按需读取。升级面板后请刷新已打开的旧页面；外部脚本需按 API.md 更新，不能再依赖 `/api/builds` 返回全部历史或日志。
合理替代方案为：stdlib HTTP Docker client而非官方SDK；每实例独立bridge而非共享bridge；服务端保管DSH cookie而非下发浏览器。前端已改用 Tailwind CSS + Radix 原语的 shadcn 风格组件（保留本地Monaco），桌面与移动端共用一套响应式界面。界面跟随系统深浅色设置自动切换夜间模式（含 Monaco 编辑器），与内嵌 DSH 行为一致，无独立开关。这些是选型差异，不降低功能、安全或验收标准，详见验收表。
同源路径代理仅适用于可信用户/插件，尚无独立origin及宿主/局域网出站防护。默认软配额不能阻止agent直接写满磁盘；硬quota是依赖宿主的可选增强，不是默认路线必做项。升级/恢复快照需额外容量，home快照上限1 GiB，模板7的 `/workspace` 就是实例home，会随home快照回滚。旧的用户级共享工作区接口和新容器共享挂载已经移除；磁盘上可能遗留的旧共享目录不属于实例home快照，也不会自动迁移。中断及回滚失败仍可能需要管理员恢复。
## 开发验证与文档维护
安装前端依赖后，可运行本地检查：
```sh
bash scripts/check-trial.sh
node --test scripts/check-trial.test.mjs
```
前者依次执行 Go 单测、race 检查、vet、构建、前端单测与生产构建，以及浏览器验收脚本的语法检查；后者验证检查脚本自身的失败传播行为。它们不等于真实 Docker、浏览器或公网 HTTPS 验收，需要外部环境的测试及历史记录见 [ACCEPTANCE.md](ACCEPTANCE.md)。
修改接口时同步 [API.md](API.md) 和 [前端对接说明](web/API_REQUEST.md)；修改用户流程时同步本页和 [前端说明](web/README.md)；修改部署、认证或数据布局时同步 [OPERATIONS.md](OPERATIONS.md)、[认证说明](internal/api/AUTH_DESIGN.md) 与 [DESIGN.md](DESIGN.md)。验收记录应注明验证范围，不把历史通过记录视为当前版本的全量验证。
cmd/server/main.go
-164
View File
@@ -1,164 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/api"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
"io"
"log"
"net"
"net/http"
"os"
"os/signal"
"path/filepath"
"strings"
"syscall"
"time"
)
func main() {
if len(os.Args) > 1 && os.Args[1] == "-healthcheck" {
if err := healthcheck(); err != nil {
log.Printf("healthcheck failed: %v", err)
os.Exit(1)
}
return
}
if err := run(); err != nil {
log.Fatal(err)
}
}
// healthcheck probes the local /healthz endpoint for container health
// reporting; it avoids shipping curl into the runtime image.
func healthcheck() error {
addr := getenv("LISTEN_ADDR", "127.0.0.1:8088")
if host, port, err := net.SplitHostPort(addr); err == nil && (host == "" || host == "0.0.0.0" || host == "::") {
addr = net.JoinHostPort("127.0.0.1", port)
}
client := &http.Client{Timeout: 5 * time.Second}
resp, err := client.Get("http://" + addr + "/healthz")
if err != nil {
return err
}
defer resp.Body.Close()
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, 4096))
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("healthz returned %d", resp.StatusCode)
}
return nil
}
func run() error {
if err := api.ValidateAuthConfig(); err != nil {
return err
}
root, err := filepath.Abs(getenv("CONTAINER_DATA_ROOT", getenv("DATA_ROOT", "./data")))
if err != nil {
return err
}
if err = os.MkdirAll(root, 0700); err != nil {
return err
}
if os.Getenv("HOST_DATA_ROOT") == "" {
os.Setenv("HOST_DATA_ROOT", root)
}
s, err := store.Open(filepath.Join(root, "db.sqlite"))
if err != nil {
return err
}
defer s.Close()
var count int
if err = s.DB.QueryRow("SELECT COUNT(*) FROM users").Scan(&count); err != nil {
return err
}
if count == 0 {
password := os.Getenv("ADMIN_PASSWORD")
if len(password) < 12 || len(password) > 72 {
return fmt.Errorf("first startup requires ADMIN_PASSWORD of 12–72 bytes")
}
hash, e := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
if e != nil {
return e
}
if err = s.CreateUser(store.User{ID: store.ID(), Username: getenv("ADMIN_USERNAME", "admin"), Admin: true, Quota: api.DefaultQuota()}, string(hash)); err != nil {
return err
}
}
panel := api.New(s, root)
if err := panel.ConfigError(); err != nil {
return err
}
checkCtx, checkCancel := context.WithTimeout(context.Background(), 20*time.Second)
if err := panel.Reconcile(checkCtx); err != nil {
log.Printf("startup reconciliation unavailable; Docker connectivity must be checked: %v", err)
}
checkCancel()
if err := panel.StartImageBuilds(); err != nil {
return fmt.Errorf("image build recovery: %w", err)
}
backend := panel.Handler()
static := http.FileServer(http.Dir(getenv("WEB_DIR", "web/dist")))
handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Referrer-Policy", "no-referrer")
if strings.HasPrefix(r.URL.Path, "/api/") || strings.HasPrefix(r.URL.Path, "/dsh/") {
backend.ServeHTTP(w, r)
return
}
if r.URL.Path == "/healthz" {
w.Write([]byte("ok"))
return
}
if r.Method != http.MethodGet && r.Method != http.MethodHead {
http.Error(w, "method not allowed", 405)
return
}
// The embedded runtime visits the same-origin SPA renewal bridge.
// Keep cross-origin framing blocked without blocking that bridge.
w.Header().Set("X-Frame-Options", "SAMEORIGIN")
w.Header().Set("Content-Security-Policy", "frame-ancestors 'self'")
static.ServeHTTP(w, r)
})
srv := &http.Server{Addr: getenv("LISTEN_ADDR", "127.0.0.1:8088"), Handler: handler, ReadHeaderTimeout: 10 * time.Second, IdleTimeout: 60 * time.Second, MaxHeaderBytes: 1 << 20}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
go panel.RunQuotaMonitor(ctx)
go panel.RunSessionCleanup(ctx)
return serve(ctx, srv)
}
func serve(ctx context.Context, srv *http.Server) error {
drained := make(chan struct{})
finished := make(chan struct{})
defer close(finished)
go func() {
defer close(drained)
select {
case <-ctx.Done():
case <-finished:
return
}
c, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
if err := srv.Shutdown(c); err != nil {
log.Printf("graceful shutdown deadline reached: %v", err)
_ = srv.Close()
}
}()
log.Printf("dsh panel listening on %s", srv.Addr)
err := srv.ListenAndServe()
if errors.Is(err, http.ErrServerClosed) {
<-drained
return nil
}
return err
}
func getenv(k, d string) string {
if v := os.Getenv(k); v != "" {
return v
}
return d
}
cmd/server/main_test.go
-117
View File
@@ -1,117 +0,0 @@
package main
import (
"context"
"io"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestInvalidAuthConfigFailsBeforeDataCreation(t *testing.T) {
for _, key := range []string{"PANEL_PUBLIC_URL", "TRUSTED_PROXY_CIDRS"} {
t.Run(key, func(t *testing.T) {
t.Setenv("PANEL_PUBLIC_URL", "")
t.Setenv("TRUSTED_PROXY_CIDRS", "")
t.Setenv(key, "invalid")
root := filepath.Join(t.TempDir(), "not-created")
t.Setenv("CONTAINER_DATA_ROOT", root)
if err := run(); err == nil || !strings.Contains(err.Error(), key) {
t.Fatalf("unexpected error: %v", err)
}
if _, err := os.Stat(root); !os.IsNotExist(err) {
t.Fatalf("data directory created: %v", err)
}
})
}
}
func TestServeWaitsForActiveHandler(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
address := make(chan string, 1)
entered := make(chan struct{})
release := make(chan struct{})
srv := &http.Server{Addr: "127.0.0.1:0", BaseContext: func(l net.Listener) context.Context {
address <- l.Addr().String()
return context.Background()
}, Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
close(entered)
<-release
_, _ = w.Write([]byte("finished"))
})}
done := make(chan error, 1)
go func() { done <- serve(ctx, srv) }()
var addr string
select {
case addr = <-address:
case <-time.After(5 * time.Second):
t.Fatal("server did not listen")
}
requestDone := make(chan error, 1)
go func() {
resp, err := http.Get("http://" + addr)
if err == nil {
_, err = io.Copy(io.Discard, resp.Body)
resp.Body.Close()
}
requestDone <- err
}()
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("handler did not start")
}
cancel()
select {
case err := <-done:
close(release)
t.Fatalf("returned before handler completed: %v", err)
case <-time.After(100 * time.Millisecond):
}
close(release)
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(5 * time.Second):
t.Fatal("shutdown stalled")
}
if err := <-requestDone; err != nil {
t.Fatal(err)
}
}
func TestHealthcheckProbesLocalEndpoint(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/healthz" {
http.NotFound(w, r)
return
}
w.Write([]byte("ok"))
}))
defer srv.Close()
_, port, err := net.SplitHostPort(srv.Listener.Addr().String())
if err != nil {
t.Fatal(err)
}
t.Setenv("LISTEN_ADDR", "127.0.0.1:"+port)
if err := healthcheck(); err != nil {
t.Fatalf("healthy server rejected: %v", err)
}
// The container listens on 0.0.0.0; the probe must target loopback.
t.Setenv("LISTEN_ADDR", "0.0.0.0:"+port)
if err := healthcheck(); err != nil {
t.Fatalf("wildcard address rejected: %v", err)
}
t.Setenv("LISTEN_ADDR", "127.0.0.1:1")
if err := healthcheck(); err == nil {
t.Fatal("dead endpoint accepted")
}
}
compose.yaml
-56
View File
@@ -1,56 +0,0 @@
services:
panel:
container_name: dsh-mgr-panel
build: .
image: dsh-mgr:local
restart: unless-stopped
stop_grace_period: 20s
user: "1000:1000"
group_add:
- ${DOCKER_GID:?Set the numeric group ID of the Docker socket}
ports:
- '127.0.0.1:8088:8088'
networks:
default:
# Keep the default gateway stable when joining runtime instance networks.
gw_priority: 100
environment:
LISTEN_ADDR: 0.0.0.0:8088
DATA_ROOT: /data
CONTAINER_DATA_ROOT: /data
HOST_DATA_ROOT: ${DSH_HOST_DATA_ROOT:?Set an absolute host data directory}
PANEL_PUBLIC_URL: ${PANEL_PUBLIC_URL:-http://127.0.0.1:8088}
TRUSTED_PROXY_CIDRS: ${TRUSTED_PROXY_CIDRS:-}
SESSION_TTL: ${SESSION_TTL:-24h}
DSH_RELEASE_REPOSITORY: ${DSH_RELEASE_REPOSITORY:-https://git.misaka.ren/m1saka/dsh-web.git}
DSH_RELEASE_BRANCH: ${DSH_RELEASE_BRANCH:-dockerfile}
WEB_DIR: /app/web/dist
UPLOAD_ALLOWED_EXTENSIONS: ${UPLOAD_ALLOWED_EXTENSIONS:-}
ADMIN_USERNAME: ${DSH_ADMIN_USERNAME:-admin}
# Read only on first startup while the user table is empty; remove the
# value from .env afterwards so the plaintext password is not retained.
ADMIN_PASSWORD: ${DSH_ADMIN_PASSWORD:-}
DOCKER_HOST: unix:///var/run/docker.sock
DSH_DOCKER_ROOT_VISIBLE: ${DSH_DOCKER_ROOT_DIR:?Set the exact DockerRootDir reported by docker info}
PANEL_CONTAINER_ID: dsh-mgr-panel
volumes:
- ${DSH_HOST_DATA_ROOT}:/data
- /var/run/docker.sock:/var/run/docker.sock
- ${DSH_DOCKER_ROOT_DIR}:${DSH_DOCKER_ROOT_DIR}:ro
read_only: true
tmpfs:
- /tmp:size=256m
mem_limit: 1g
cpus: 2
pids_limit: 512
healthcheck:
test: ["CMD", "/usr/local/bin/dsh-mgr", "-healthcheck"]
interval: 30s
timeout: 5s
retries: 3
start_period: 15s
logging:
driver: json-file
options:
max-size: 10m
max-file: '3'
go.mod
-20
View File
@@ -1,20 +0,0 @@
module git.misaka.ren/m1saka/dsh-web
go 1.25
require modernc.org/sqlite v1.34.5
require golang.org/x/crypto v0.28.0
require (
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/ncruces/go-strftime v0.1.9 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
golang.org/x/sys v0.26.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
modernc.org/libc v1.55.3 // indirect
modernc.org/mathutil v1.6.0 // indirect
modernc.org/memory v1.8.0 // indirect
)
go.sum
-48
View File
@@ -1,48 +0,0 @@
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
golang.org/x/crypto v0.28.0 h1:GBDwsMXVQi34v5CCYUm2jkJvu4cbtru2U4TN2PSyQnw=
golang.org/x/crypto v0.28.0/go.mod h1:rmgy+3RHxRZMyY0jjAJShp2zgEdOqj2AO7U0pYmeQ7U=
golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.26.0 h1:KHjCJyddX0LoSTb3J+vWpupP9p0oznkqVk/IfjymZbo=
golang.org/x/sys v0.26.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
modernc.org/sqlite v1.34.5 h1:Bb6SR13/fjp15jt70CL4f18JIN7p7dnMExd+UFnF15g=
modernc.org/sqlite v1.34.5/go.mod h1:YLuNmX9NKs8wRNK2ko1LW1NGYcc9FkBO69JOt1AR9JE=
modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
internal/api/AUTH_DESIGN.md
-52
View File
@@ -1,52 +0,0 @@
# Authentication boundary (§13)
- Real logins retain `dsh_session` as a database session binding and require a
five-minute HS256 access JWT in an additional HttpOnly cookie. JWT verification
fixes the algorithm and binds subject/session/issuer/audience/expiry. Bare
session IDs never authenticate. Internal callers and test fixtures must also
attach a signed access cookie. Database session expiry or revocation rejects
access even if the JWT's five-minute expiry has not yet elapsed.
- The initial refresh token is random 256-bit data in an HttpOnly cookie, scoped to `/api`, with only its
SHA-256 hash stored in SQLite. The session has an absolute lifetime configured
by SESSION_TTL (default 24h), applied only to new logins. Values use Go duration
syntax (e.g. 168h, not 7d), must be at least one second and a whole number of
seconds; invalid values prevent startup when the store is opened.
Session, refresh and CSRF cookies persist across browser restarts with explicit
Max-Age/Expires bounded by that database expiry; rotation does not extend it.
Transactional rotation stores current and previous token hashes. A keyed,
session-bound successor derivation makes retries of the previous token return
the same current token, including after response loss or server restart. Only
one previous generation is accepted (until the next rotation or session expiry);
this intentionally permits that generation to be replayed with the required
session/Origin/CSRF checks. Older and unknown tokens are rejected. Plaintext
refresh credentials are never stored.
Logout deletes the database session and its refresh binding. Both refresh and
logout retain mandatory exact Origin and session-bound CSRF checks, even when
access has expired, but require an unexpired database session. Logout does not
require the refresh cookie. Other protected operations require valid access.
Self-service password changes and administrator password resets revoke all of
that user's sessions and refresh bindings, requiring a new login.
- Signing key is generated once under the configured data root, private mode
0600, and reused across restarts. Initialization errors fail authentication closed.
Session/access/refresh credentials are issued as cookies, not JSON or URLs,
and are not included in authentication audit detail or application logs. The
CSRF token is intentionally returned as `csrf_token` by login, refresh and
`/api/me`, and is also in the JavaScript-readable `dsh_csrf` cookie. All four
cookies use SameSite=Lax; the other three are HttpOnly. Secure is enabled by
direct TLS or an HTTPS PANEL_PUBLIC_URL, not forwarded protocol headers.
Origin checks compare against the configured public origin, or direct request
scheme/Host if it is unset; forwarded Host/Proto headers are not used.
- Known accounts lock for 15 minutes after five failures; the counter/lock survive
restart. Successful login resets the counter; expired locks reset the counter.
Current-password reauthentication failures share this account lock; successful
reauthentication alone does not reset the counter. Additional login protection
allows 30 attempts per source per minute, at most 1024 source buckets, and eight
admitted concurrent login requests (credential checks are serialized). Source
identity uses the direct peer unless it matches TRUSTED_PROXY_CIDRS, a
comma-separated CIDR list; then X-Forwarded-For is scanned right-to-left for
the first untrusted address. These in-memory limits reset on process restart.
The server runs expired-session/refresh and elapsed-lock cleanup immediately
and every ten minutes; expiry checks do not wait for cleanup.
- Tests cover missing/tampered/expired access, database binding, refresh rotation
and replay, CSRF/Origin rejection, logout revocation, key reuse/permissions,
account isolation, restart persistence, lock expiry, and runtime compatibility.
internal/api/admin.go
-222
View File
@@ -1,222 +0,0 @@
package api
import (
"database/sql"
"encoding/json"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
"io"
"net/http"
"os"
"regexp"
"strconv"
"time"
)
var usernamePattern = regexp.MustCompile("^[a-zA-Z0-9_-]{3,40}$")
func DefaultQuota() int64 {
if n, err := strconv.ParseInt(os.Getenv("DEFAULT_QUOTA_BYTES"), 10, 64); err == nil && n > 0 {
return n
}
return 10 << 30
}
func (a *API) users(w http.ResponseWriter, r *http.Request) {
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
switch r.Method {
case "GET":
rows, err := a.s.DB.Query("SELECT id,username,admin,quota,created FROM users ORDER BY created")
if err != nil {
fail(w, 500, "database error")
return
}
defer rows.Close()
out := []store.User{}
for rows.Next() {
var v store.User
var created int64
if err := rows.Scan(&v.ID, &v.Username, &v.Admin, &v.Quota, &created); err != nil {
fail(w, 500, "database error")
return
}
v.Created = time.Unix(created, 0)
out = append(out, v)
}
if err := rows.Err(); err != nil {
fail(w, 500, "database error")
return
}
writeJSON(w, out)
case "POST":
var in struct {
Username string
Password string
Admin bool
Quota int64 `json:"quota_bytes"`
}
if json.NewDecoder(r.Body).Decode(&in) != nil || !usernamePattern.MatchString(in.Username) || len(in.Password) < 12 || len(in.Password) > 72 || in.Quota < 0 {
fail(w, 400, "username must be 3–40 letters/digits/_/-, password 12–72 bytes, quota nonnegative")
return
}
if in.Quota == 0 {
in.Quota = DefaultQuota()
}
h, e := bcrypt.GenerateFromPassword([]byte(in.Password), bcrypt.DefaultCost)
if e != nil {
fail(w, 500, "password error")
return
}
v := store.User{ID: store.ID(), Username: in.Username, Admin: in.Admin, Quota: in.Quota, Created: time.Now()}
if e = a.s.CreateUser(v, string(h)); e != nil {
fail(w, 409, "username already exists or database unavailable")
return
}
a.s.AddAudit(u.ID, "user.create", v.ID, "")
writeJSON(w, v)
case "PATCH":
var in struct {
ID string
Quota int64 `json:"quota_bytes"`
}
if json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&in) != nil || in.ID == "" || in.Quota <= 0 {
fail(w, 400, "positive quota and user ID required")
return
}
user := a.userLocks.Acquire(in.ID)
defer user.Release()
result, err := a.s.DB.Exec("UPDATE users SET quota=? WHERE id=?", in.Quota, in.ID)
if err != nil {
fail(w, 500, "quota update failed")
return
}
count, _ := result.RowsAffected()
if count != 1 {
fail(w, 404, "user not found")
return
}
a.s.AddAudit(u.ID, "user.quota", in.ID, strconv.FormatInt(in.Quota, 10))
writeJSON(w, map[string]bool{"ok": true})
default:
fail(w, 405, "method not allowed")
}
}
func (a *API) resetPassword(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-store")
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
var in struct{ ID, Password string }
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096))
if decoder.Decode(&in) != nil || decoder.Decode(new(any)) != io.EOF || in.ID == "" || len(in.Password) < 12 || len(in.Password) > 72 {
fail(w, 400, "user ID and password 12–72 bytes required")
return
}
hash, err := bcrypt.GenerateFromPassword([]byte(in.Password), bcrypt.DefaultCost)
if err != nil {
fail(w, 500, "password error")
return
}
// Login must not issue a session using a hash read before this reset.
a.authLoginMu.Lock()
defer a.authLoginMu.Unlock()
u, _, err = a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
tx, err := a.s.DB.BeginTx(r.Context(), nil)
if err != nil {
fail(w, 500, "password reset failed")
return
}
defer tx.Rollback()
var username string
err = tx.QueryRowContext(r.Context(), "SELECT username FROM users WHERE id=?", in.ID).Scan(&username)
if err == sql.ErrNoRows {
fail(w, 404, "user not found")
return
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "UPDATE users SET pass_hash=? WHERE id=?", string(hash), in.ID)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "DELETE FROM auth_refresh WHERE sid IN (SELECT id FROM sessions WHERE uid=?)", in.ID)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "DELETE FROM sessions WHERE uid=?", in.ID)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "DELETE FROM auth_failures WHERE username=?", username)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "INSERT INTO audit(uid,action,target,detail,at) VALUES(?,?,?,?,?)", u.ID, "user.reset_password", in.ID, "", time.Now().Unix())
}
if err == nil {
err = tx.Commit()
}
if err != nil {
fail(w, 500, "password reset failed")
return
}
writeJSON(w, map[string]bool{"ok": true, "reauthenticate": u.ID == in.ID})
}
func (a *API) audit(w http.ResponseWriter, r *http.Request) {
u, _, e := a.user(r)
if e != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
if r.Method != "GET" {
fail(w, 405, "method not allowed")
return
}
rows, e := a.s.DB.Query("SELECT id,uid,action,target,detail,at FROM audit ORDER BY id DESC LIMIT 500")
if e != nil {
fail(w, 500, "database error")
return
}
defer rows.Close()
out := []map[string]any{}
for rows.Next() {
var id, at int64
var uid, action, target, detail string
if rows.Scan(&id, &uid, &action, &target, &detail, &at) != nil {
fail(w, 500, "database error")
return
}
out = append(out, map[string]any{"id": id, "user_id": uid, "action": action, "target": target, "detail": detail, "created_at": time.Unix(at, 0)})
}
if err := rows.Err(); err != nil {
fail(w, 500, "database error")
return
}
writeJSON(w, out)
}
internal/api/admin_password_test.go
-303
View File
@@ -1,303 +0,0 @@
package api
import (
"encoding/json"
"fmt"
"golang.org/x/crypto/bcrypt"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
"time"
)
const resetPath = "/api/users/reset-password"
func TestResetPasswordConcurrentLogin(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
admin := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
start := make(chan struct{})
results := make(chan *httptest.ResponseRecorder, 8)
for i := 0; i < 8; i++ {
go func() { <-start; results <- req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil) }()
}
close(start)
w := req("POST", resetPath, resetBody("bob", resetSecret), cookieValue(admin, "dsh_csrf"), admin)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
for i := 0; i < 8; i++ {
w := <-results
if w.Code != 200 && w.Code != 401 && w.Code != 429 {
t.Fatal(w.Code, w.Body.String())
}
if w.Code == 200 {
if check := req("GET", "/api/me", "", "", w.Result().Cookies()); check.Code != 401 {
t.Fatal("old-password login retained session", check.Code)
}
}
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM sessions WHERE uid='bob'").Scan(&n); err != nil || n != 0 {
t.Fatal("old login sessions", n, err)
}
}
type resetSignalReader struct {
reader *strings.Reader
started chan struct{}
once sync.Once
}
func (r *resetSignalReader) Read(p []byte) (int, error) {
r.once.Do(func() { close(r.started) })
return r.reader.Read(p)
}
func TestResetPasswordRechecksWaitingAdmin(t *testing.T) {
for _, revoke := range []string{"session", "admin"} {
t.Run(revoke, func(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
admin := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
reader := &resetSignalReader{reader: strings.NewReader(resetBody("bob", resetSecret)), started: make(chan struct{})}
r := httptest.NewRequest("POST", resetPath, reader)
r.Header.Set("Origin", a.origin)
r.Header.Set("X-CSRF-Token", cookieValue(admin, "dsh_csrf"))
for _, c := range admin {
r.AddCookie(c)
}
a.authLoginMu.Lock()
locked := true
defer func() {
if locked {
a.authLoginMu.Unlock()
}
}()
done := make(chan *httptest.ResponseRecorder, 1)
go func() { w := httptest.NewRecorder(); a.Handler().ServeHTTP(w, r); done <- w }()
select {
case <-reader.started:
case <-time.After(5 * time.Second):
t.Fatal("request did not reach decoder")
}
select {
case w := <-done:
t.Fatal("reset bypassed login lock", w.Code)
case <-time.After(150 * time.Millisecond):
}
query := "UPDATE users SET admin=0 WHERE id='alice'"
want := 403
if revoke == "session" {
query = "DELETE FROM sessions WHERE uid='alice'"
want = 401
}
if _, err := a.s.DB.Exec(query); err != nil {
t.Fatal(err)
}
a.authLoginMu.Unlock()
locked = false
select {
case w := <-done:
if w.Code != want {
t.Fatal(w.Code, w.Body.String())
}
case <-time.After(5 * time.Second):
t.Fatal("reset did not finish")
}
hash, err := a.s.Password("bob")
if err != nil || bcrypt.CompareHashAndPassword([]byte(hash), []byte("password-123")) != nil {
t.Fatal("unauthorized reset", err)
}
})
}
}
const resetSecret = "new-password-123"
func resetBody(id, password string) string {
b, _ := json.Marshal(map[string]string{"id": id, "password": password})
return string(b)
}
func TestResetPasswordValidation(t *testing.T) {
a, req := authFixture(t)
admin := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
regular := req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil).Result().Cookies()
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
csrf := cookieValue(admin, "dsh_csrf")
for _, tc := range []struct {
name, method, body, csrf string
cookies []*http.Cookie
code int
}{
{"anonymous", "POST", resetBody("bob", resetSecret), "", nil, 401},
{"regular", "POST", resetBody("alice", resetSecret), cookieValue(regular, "dsh_csrf"), regular, 403},
{"csrf", "POST", resetBody("bob", resetSecret), "bad", admin, 403},
{"method", "GET", "", csrf, admin, 405},
{"missing id", "POST", resetBody("", resetSecret), csrf, admin, 400},
{"short", "POST", resetBody("bob", strings.Repeat("a", 11)), csrf, admin, 400},
{"long", "POST", resetBody("bob", strings.Repeat("a", 73)), csrf, admin, 400},
{"unicode long", "POST", resetBody("bob", strings.Repeat("密", 25)), csrf, admin, 400},
{"malformed", "POST", "{", csrf, admin, 400},
{"trailing", "POST", resetBody("bob", resetSecret) + "{}", csrf, admin, 400},
{"oversized", "POST", resetBody("bob", strings.Repeat("a", 4096)), csrf, admin, 400},
{"missing target", "POST", resetBody("missing", resetSecret), csrf, admin, 404},
} {
t.Run(tc.name, func(t *testing.T) {
w := req(tc.method, resetPath, tc.body, tc.csrf, tc.cookies)
if w.Code != tc.code {
t.Fatal(w.Code, w.Body.String())
}
})
}
for _, origin := range []string{"", "https://evil.test"} {
r := httptest.NewRequest("POST", resetPath, strings.NewReader(resetBody("bob", resetSecret)))
r.Header.Set("Origin", origin)
r.Header.Set("X-CSRF-Token", csrf)
for _, c := range admin {
r.AddCookie(c)
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
if w.Code != 403 {
t.Fatal("origin accepted", w.Code)
}
}
for _, password := range []string{strings.Repeat("a", 12), strings.Repeat("密", 24)} {
if w := req("POST", resetPath, resetBody("bob", password), csrf, admin); w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
hash, err := a.s.Password("bob")
if err != nil || bcrypt.CompareHashAndPassword([]byte(hash), []byte(password)) != nil {
t.Fatal("boundary hash", err)
}
}
}
func resetLogin(username, password string) string {
b, _ := json.Marshal(map[string]string{"username": username, "password": password})
return string(b)
}
func TestResetPasswordLifecycle(t *testing.T) {
for _, target := range []string{"bob", "alice"} {
t.Run(target, func(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
login := func(name string) []*http.Cookie {
w := req("POST", "/api/login", resetLogin(name, "password-123"), "", nil)
if w.Code != 200 {
t.Fatal(w.Code)
}
return w.Result().Cookies()
}
admin := login("alice")
targets := [][]*http.Cookie{login(target), login(target)}
other := "alice"
if target == "alice" {
other = "bob"
}
unaffected := login(other)
if _, err := a.s.DB.Exec("INSERT INTO auth_failures VALUES(?,5,?)", target, 9999999999); err != nil {
t.Fatal(err)
}
w := req("POST", resetPath, resetBody(target, resetSecret), cookieValue(admin, "dsh_csrf"), admin)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
var result map[string]bool
if json.Unmarshal(w.Body.Bytes(), &result) != nil || !result["ok"] || result["reauthenticate"] != (target == "alice") {
t.Fatal(w.Body.String())
}
for _, cs := range targets {
var remaining int
if err := a.s.DB.QueryRow("SELECT count(*) FROM auth_refresh WHERE sid=?", cookieValue(cs, "dsh_session")).Scan(&remaining); err != nil || remaining != 0 {
t.Fatal("old refresh row", remaining, err)
}
if w := req("GET", "/api/me", "", "", cs); w.Code != 401 {
t.Fatal("old access", w.Code)
}
if w := req("POST", "/api/refresh", "", cookieValue(cs, "dsh_csrf"), cs); w.Code != 403 {
t.Fatal("old refresh", w.Code)
}
}
for _, query := range []string{"SELECT count(*) FROM sessions WHERE uid=?", "SELECT count(*) FROM auth_failures WHERE username=?", "SELECT count(*) FROM auth_refresh WHERE sid IN (SELECT id FROM sessions WHERE uid=?)"} {
var n int
if err := a.s.DB.QueryRow(query, target).Scan(&n); err != nil || n != 0 {
t.Fatal(query, n, err)
}
}
if w := req("GET", "/api/me", "", "", unaffected); w.Code != 200 {
t.Fatal("other access", w.Code)
}
if w := req("POST", "/api/refresh", "", cookieValue(unaffected, "dsh_csrf"), unaffected); w.Code != 200 {
t.Fatal("other refresh", w.Code)
}
if w := req("POST", "/api/login", resetLogin(target, "password-123"), "", nil); w.Code != 401 {
t.Fatal("old password", w.Code)
}
if w := req("POST", "/api/login", resetLogin(target, resetSecret), "", nil); w.Code != 200 {
t.Fatal("new password", w.Code)
}
var actor, detail string
var n int
if err := a.s.DB.QueryRow("SELECT uid,detail,count(*) FROM audit WHERE action='user.reset_password' AND target=?", target).Scan(&actor, &detail, &n); err != nil || actor != "alice" || detail != "" || n != 1 {
t.Fatal("audit", actor, detail, n, err)
}
})
}
}
func TestResetPasswordRollback(t *testing.T) {
for _, step := range []struct{ table, event string }{{"users", "UPDATE"}, {"auth_refresh", "DELETE"}, {"sessions", "DELETE"}, {"auth_failures", "DELETE"}, {"audit", "INSERT"}} {
t.Run(step.table, func(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
admin := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
target := req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil).Result().Cookies()
before, err := a.s.Password("bob")
if err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO auth_failures VALUES('bob',5,9999999999)"); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec(fmt.Sprintf("CREATE TRIGGER reject_reset BEFORE %s ON %s BEGIN SELECT RAISE(ABORT,'forced failure'); END", step.event, step.table)); err != nil {
t.Fatal(err)
}
w := req("POST", resetPath, resetBody("bob", resetSecret), cookieValue(admin, "dsh_csrf"), admin)
if w.Code != 500 || strings.Contains(w.Body.String(), resetSecret) {
t.Fatal(w.Code, w.Body.String())
}
after, err := a.s.Password("bob")
if err != nil || before != after {
t.Fatal("hash changed", err)
}
for query, want := range map[string]int{"SELECT count(*) FROM sessions WHERE uid='bob'": 1, "SELECT count(*) FROM auth_refresh": 2, "SELECT count(*) FROM auth_failures WHERE username='bob' AND failures=5 AND locked_until=9999999999": 1, "SELECT count(*) FROM audit WHERE action='user.reset_password'": 0} {
var n int
if err := a.s.DB.QueryRow(query).Scan(&n); err != nil || n != want {
t.Fatal(query, n, err)
}
}
if w := req("GET", "/api/me", "", "", target); w.Code != 200 {
t.Fatal("session changed", w.Code)
}
if w := req("POST", "/api/refresh", "", cookieValue(target, "dsh_csrf"), target); w.Code != 200 {
t.Fatal("refresh changed", w.Code)
}
})
}
}
internal/api/admin_quota_test.go
-33
View File
@@ -1,33 +0,0 @@
package api
import (
"net/http"
"net/http/httptest"
"testing"
)
func TestAdminQuotaUpdate(t *testing.T) {
a, sid := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) {})
request := func() *httptest.ResponseRecorder {
w := httptest.NewRecorder()
a.users(w, runtimeRequest("PATCH", "/api/users", `{"id":"owner","quota_bytes":2048}`, sid))
return w
}
if w := request(); w.Code != 403 {
t.Fatal(w.Code)
}
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
if w := request(); w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
u, err := a.s.UserByID("owner")
if err != nil || u.Quota != 2048 {
t.Fatalf("%+v %v", u, err)
}
t.Setenv("DEFAULT_QUOTA_BYTES", "12345")
if DefaultQuota() != 12345 {
t.Fatal("default quota ignored")
}
}
internal/api/admission.go
-194
View File
@@ -1,194 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"io"
"net/http"
"strings"
"sync"
"time"
)
// Admission writes serialize with new runtime operations, including their Docker side effects.
var admissionMu sync.RWMutex
func defaultAdmission(version, recipe, platform string) string {
if recipe == "7" && platform == "linux/amd64" {
switch version {
case "0.2.0-rc.1":
return "compatible"
case "0.2.0-rc.2":
return "recommended"
}
}
return "existing_only"
}
func supportedLayout(recipe string) bool {
return recipe == imageRecipeRev
}
func positiveAdmission(state string) bool { return state == "recommended" || state == "compatible" }
func (a *API) evaluateImage(ctx context.Context, id string) (imageRecord, error) {
im := imageRecord{ID: id, ImageID: id, Status: "missing", Admission: "existing_only", AdmissionReason: "local image missing"}
actual, err := a.inspectLocalImage(ctx, id)
if err != nil || actual == nil {
return im, err
}
im.Version = actual.Config.Labels["dsh-mgr.version"]
im.RecipeRev = actual.Config.Labels["dsh-mgr.recipe"]
im.Platform = actual.platform()
im.Created = actual.Created
im.Status = "unverified"
im.AdmissionReason = "successful exact image identity verification required"
if actual.ID != id || actual.Config.Labels["dsh-mgr.role"] != "runtime" {
im.Version = ""
return im, nil
}
var count, experimental int
err = a.s.DB.QueryRowContext(ctx, "SELECT count(*),COALESCE(min(experimental),0) FROM image_builds WHERE image_id=? AND version=? AND recipe_rev=? AND platform=? AND status='success'", id, im.Version, im.RecipeRev, im.Platform).Scan(&count, &experimental)
if err != nil {
return im, err
}
var state, evidence string
err = a.s.DB.QueryRowContext(ctx, "SELECT state,evidence FROM image_admissions WHERE image_id=? AND version=? AND recipe_rev=? AND platform=?", id, im.Version, im.RecipeRev, im.Platform).Scan(&state, &evidence)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return im, err
}
if state == "blocked" {
im.Admission = state
im.Status = state
im.AdmissionReason = evidence
return im, nil
}
if count == 0 {
return im, nil
}
var trustedRelease int
err = a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM image_builds WHERE image_id=? AND version=? AND recipe_rev=? AND platform=? AND status='success' AND release_id<>'' AND release_id=? AND source_repository=? AND source_commit=?", id, im.Version, im.RecipeRev, im.Platform, actual.Config.Labels["dsh-mgr.release"], actual.Config.Labels["dsh-mgr.source"], actual.Config.Labels["dsh-mgr.commit"]).Scan(&trustedRelease)
if err != nil {
return im, err
}
im.Admission = defaultAdmission(im.Version, im.RecipeRev, im.Platform)
im.AdmissionReason = "no accepted compatibility evidence for this tuple"
if positiveAdmission(im.Admission) {
im.AdmissionReason = "built-in exact tuple compatibility"
}
if experimental != 0 {
im.Admission = "existing_only"
im.Status = "experimental"
im.AdmissionReason = "experimental validation succeeded; administrator acceptance required"
} else {
im.Status = "existing_only"
}
if trustedRelease > 0 {
im.Admission = "compatible"
im.AdmissionReason = "trusted pinned release passed exact image startup validation"
}
if state != "" {
im.Admission = state
im.AdmissionReason = evidence
im.Status = state
}
if !supportedLayout(im.RecipeRev) || im.Platform != imagePlatform {
im.Admission = "existing_only"
im.Status = "existing_only"
im.AdmissionReason = "historical validation retained; current mount layout or platform not supported"
}
im.Selectable = positiveAdmission(im.Admission)
if im.Selectable {
im.Status = "available"
}
return im, nil
}
func (a *API) imageAdmission(w http.ResponseWriter, r *http.Request) {
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
var in struct {
ImageID string `json:"image_id"`
State string `json:"state"`
Evidence string `json:"evidence"`
}
dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 8192))
dec.DisallowUnknownFields()
if dec.Decode(&in) != nil || dec.Decode(new(any)) != io.EOF || !runtimeImageID.MatchString(in.ImageID) || strings.TrimSpace(in.Evidence) == "" || (!positiveAdmission(in.State) && in.State != "existing_only" && in.State != "blocked") {
fail(w, 400, "image_id, valid state and evidence required")
return
}
if len(in.Evidence) > 4096 {
fail(w, 400, "evidence must not exceed 4096 UTF-8 bytes")
return
}
admissionMu.Lock()
defer admissionMu.Unlock()
buildQueue.Lock()
err = a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
fail(w, 500, "database error")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
im, err := a.evaluateImage(ctx, in.ImageID)
if err != nil {
fail(w, 502, "image inspection failed")
return
}
if im.Status == "missing" || im.Version == "" || im.RecipeRev == "" || im.Platform == "" {
fail(w, 400, "real runtime image identity required")
return
}
if positiveAdmission(in.State) {
var count int
err = a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM image_builds WHERE image_id=? AND version=? AND recipe_rev=? AND platform=? AND status='success'", im.ImageID, im.Version, im.RecipeRev, im.Platform).Scan(&count)
if err != nil || count == 0 || !supportedLayout(im.RecipeRev) || im.Platform != imagePlatform {
fail(w, 400, "successful exact validation and supported mount layout/platform required")
return
}
}
buildQueue.Lock()
defer buildQueue.Unlock()
tx, err := a.s.DB.BeginTx(ctx, nil)
if err != nil {
fail(w, 500, "database error")
return
}
defer tx.Rollback()
_, err = tx.ExecContext(ctx, "INSERT INTO image_admissions(image_id,version,recipe_rev,platform,state,evidence,admitted_by,admitted_at) VALUES(?,?,?,?,?,?,?,?) ON CONFLICT(image_id,version,recipe_rev,platform) DO UPDATE SET state=excluded.state,evidence=excluded.evidence,admitted_by=excluded.admitted_by,admitted_at=excluded.admitted_at", im.ImageID, im.Version, im.RecipeRev, im.Platform, in.State, strings.TrimSpace(in.Evidence), u.ID, time.Now().UTC().Format(time.RFC3339Nano))
if err != nil {
fail(w, 500, "database error")
return
}
if _, err = tx.ExecContext(ctx, "INSERT INTO audit(uid,action,target,detail,at) VALUES(?,?,?,?,?)", u.ID, "image.admission", im.ImageID, in.State+": "+in.Evidence, time.Now().Unix()); err != nil {
fail(w, 500, "audit persistence failed")
return
}
if err = tx.Commit(); err != nil {
fail(w, 500, "database error")
return
}
a.imageDeleteGeneration++
im.Admission = in.State
im.AdmissionReason = strings.TrimSpace(in.Evidence)
im.Selectable = positiveAdmission(in.State)
im.Status = in.State
if im.Selectable {
im.Status = "available"
}
writeJSON(w, im)
}
internal/api/admission_routes_test.go
-115
View File
@@ -1,115 +0,0 @@
package api
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestAdmissionEvidenceUTF8Limit(t *testing.T) {
id := "sha256:" + strings.Repeat("d", 64)
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { writeImageInspect(w, id, "0.2.0-rc.2") })
seedVerifiedImage(t, a, id, "0.2.0-rc.2")
if _, err := a.s.DB.Exec("UPDATE users SET admin=1"); err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
name, evidence string
want int
}{
{"ascii boundary", strings.Repeat("a", 4096), 200},
{"ascii over", strings.Repeat("a", 4097), 400},
{"Chinese regression", strings.Repeat("中", 1400), 400},
{"Chinese below", strings.Repeat("中", 1365), 200},
{"Chinese boundary", strings.Repeat("中", 1365) + "a", 200},
{"Chinese over", strings.Repeat("中", 1365) + "ab", 400},
{"emoji boundary", strings.Repeat("😀", 1024), 200},
{"emoji over", strings.Repeat("😀", 1024) + "a", 400},
{"two byte boundary", strings.Repeat("é", 2048), 200},
{"raw whitespace counts", " " + strings.Repeat("a", 4096), 400},
} {
t.Run(tc.name, func(t *testing.T) {
body, err := json.Marshal(map[string]string{"image_id": id, "state": "compatible", "evidence": tc.evidence})
if err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.imageAdmission(w, runtimeRequest("POST", "/api/images/admission", string(body), sid))
if w.Code != tc.want {
t.Fatal(w.Code, w.Body.String())
}
if tc.want == 400 && !strings.Contains(w.Body.String(), "4096 UTF-8 bytes") {
t.Fatal(w.Body.String())
}
if tc.want == 200 {
var evidence string
if err := a.s.DB.QueryRow("SELECT evidence FROM image_admissions WHERE image_id=?", id).Scan(&evidence); err != nil || evidence != tc.evidence {
t.Fatal(evidence, err)
}
}
})
}
}
func TestAdmissionRoutesEnforceAuthorization(t *testing.T) {
id := "sha256:" + strings.Repeat("d", 64)
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { writeImageInspect(w, id, "0.2.0-rc.2") })
seedVerifiedImage(t, a, id, "0.2.0-rc.2")
_, csrf, err := a.s.SessionUser(sid)
if err != nil {
t.Fatal(err)
}
body := fmt.Sprintf(`{"image_id":%q,"state":"compatible","evidence":"isolated validation reference"}`, id)
request := func(method, path, payload string, auth, token bool) *httptest.ResponseRecorder {
session := ""
if auth {
session = sid
}
r := runtimeRequest(method, path, payload, session)
r.Header.Set("Origin", "http://panel.test")
if token {
r.Header.Set("X-CSRF-Token", csrf)
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
return w
}
for _, path := range []string{"/api/latest", "/api/compatibility", "/api/intents", "/api/intents/old"} {
if w := request("GET", path, "", true, false); w.Code != 404 {
t.Fatal(path, w.Code, w.Body.String())
}
}
if w := request("POST", "/api/images/admission", body, true, true); w.Code != 403 {
t.Fatal("nonadmin admission", w.Code, w.Body.String())
}
if w := request("POST", "/api/builds", `{"version":"9.8.7","experimental":true}`, true, true); w.Code != 405 {
t.Fatal("nonadmin experiment", w.Code, w.Body.String())
}
if _, err = a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
if w := request("POST", "/api/images/admission", body, true, false); w.Code != 403 {
t.Fatal("missing CSRF", w.Code, w.Body.String())
}
if w := request("POST", "/api/images/admission", body, true, true); w.Code != 200 {
t.Fatal("admin admission", w.Code, w.Body.String())
}
for _, payload := range []string{
body + "{}",
body + strings.Repeat(" ", 8192),
fmt.Sprintf(`{"image_id":%q,"state":"compatible","evidence":" "}`, id),
fmt.Sprintf(`{"image_id":%q,"state":"available","evidence":"invalid state"}`, id),
fmt.Sprintf(`{"image_id":%q,"state":"compatible","evidence":"test","dockerfile":"not allowed"}`, id),
} {
if w := request("POST", "/api/images/admission", payload, true, true); w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
}
var count int
if err = a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='image.admission'").Scan(&count); err != nil || count != 1 {
t.Fatal(count, err)
}
}
internal/api/admission_test.go
-287
View File
@@ -1,287 +0,0 @@
package api
import (
"context"
"encoding/json"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestHistoricalExperimentalBuildRequiresAcceptance(t *testing.T) {
id := "sha256:" + strings.Repeat("d", 64)
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1.47/images/json":
writeJSON(w, []any{map[string]any{"Id": id, "Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "9.8.7", "dsh-mgr.recipe": imageRecipeRev}}})
case "/v1.47/images/" + id + "/json":
writeImageInspect(w, id, "9.8.7")
case "/v1.47/containers/create":
writeJSON(w, map[string]string{"Id": "check"})
case "/v1.47/containers/check/exec":
writeJSON(w, map[string]string{"Id": "probe"})
case "/v1.47/exec/probe/json":
writeJSON(w, map[string]any{"Running": false, "ExitCode": 0})
case "/v1.47/containers/check/json":
writeJSON(w, map[string]any{"State": map[string]bool{"Running": true}})
default:
w.WriteHeader(204)
}
})
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='u'"); err != nil {
t.Fatal(err)
}
seedVerifiedImage(t, a, id, "9.8.7")
if _, err := a.s.DB.Exec("DELETE FROM image_admissions; UPDATE image_builds SET experimental=1"); err != nil {
t.Fatal(err)
}
im, err := a.evaluateImage(context.Background(), id)
if err != nil || im.Selectable || im.Status != "experimental" {
t.Fatal(im, err)
}
}
func TestLegacyQueuedBuildFailsBeforeDocker(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unsupported build contacted Docker") })
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
t.Fatal(err)
}
previous := imagePlatform
imagePlatform = "linux/arm64"
defer func() { imagePlatform = previous }()
b := buildRecord{ID: "unsupported", Version: "0.2.0-rc.2", Status: "queued", RecipeRev: imageRecipeRev, Platform: imagePlatform}
if err := a.insertBuildRequest(b, "u"); err != nil {
t.Fatal(err)
}
a.runImageBuild(b)
var state, detail string
if err := a.s.DB.QueryRow("SELECT status,error FROM image_builds WHERE id=?", b.ID).Scan(&state, &detail); err != nil || state != "failed" || !strings.Contains(detail, "Git release identity required") {
t.Fatal(state, detail, err)
}
}
func TestAdmissionDowngradeWaitsForCreate(t *testing.T) {
id := "sha256:" + strings.Repeat("e", 64)
entered, release := make(chan struct{}), make(chan struct{})
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.Contains(r.URL.Path, "/images/"):
writeImageInspect(w, id, "0.2.0-rc.1")
case strings.HasSuffix(r.URL.Path, "/networks/create"):
close(entered)
<-release
w.WriteHeader(204)
case strings.HasSuffix(r.URL.Path, "/containers/create"):
writeJSON(w, map[string]string{"Id": "created"})
default:
w.WriteHeader(204)
}
})
seedVerifiedImage(t, a, id, "0.2.0-rc.1")
if _, err := a.s.DB.Exec("UPDATE users SET admin=1"); err != nil {
t.Fatal(err)
}
created := make(chan int, 1)
go func() {
w := httptest.NewRecorder()
a.instances(w, runtimeRequest("POST", "/api/instances", fmt.Sprintf(`{"name":"race","image_id":%q}`, id), sid))
created <- w.Code
}()
select {
case <-entered:
case <-time.After(3 * time.Second):
close(release)
t.Fatal("create did not reach network")
}
downgraded := make(chan int, 1)
go func() {
w := httptest.NewRecorder()
a.imageAdmission(w, runtimeRequest("POST", "/api/images/admission", fmt.Sprintf(`{"image_id":%q,"state":"blocked","evidence":"downgrade test"}`, id), sid))
downgraded <- w.Code
}()
select {
case status := <-downgraded:
close(release)
t.Fatalf("admission raced active create: %d", status)
case <-time.After(30 * time.Millisecond):
}
close(release)
if status := <-created; status != 200 {
t.Fatal(status)
}
if status := <-downgraded; status != 200 {
t.Fatal(status)
}
w := httptest.NewRecorder()
a.instances(w, runtimeRequest("POST", "/api/instances", fmt.Sprintf(`{"name":"denied","image_id":%q}`, id), sid))
if w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
}
func TestAdmissionRejectsLegacyLayouts(t *testing.T) {
for _, recipe := range []string{"", "1", "2", "3", "4", "5", "6", "8"} {
t.Run(recipe, func(t *testing.T) {
id := "sha256:" + strings.Repeat("a", 64)
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.URL.Path, "/images/") {
t.Error("unexpected Docker operation", r.Method, r.URL.Path)
}
writeJSON(w, map[string]any{"Id": id, "Os": "linux", "Architecture": "amd64", "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "1.2.3", "dsh-mgr.recipe": recipe}}})
})
seedVerifiedImage(t, a, id, "1.2.3")
for _, query := range []string{"UPDATE image_builds SET recipe_rev=?", "UPDATE image_admissions SET recipe_rev=?"} {
if _, err := a.s.DB.Exec(query, recipe); err != nil {
t.Fatal(err)
}
}
if _, err := a.s.DB.Exec("UPDATE users SET admin=1"); err != nil {
t.Fatal(err)
}
im, err := a.evaluateImage(context.Background(), id)
if err != nil || im.Selectable || im.Admission != "existing_only" {
t.Fatal(im, err)
}
for _, state := range []string{"compatible", "recommended"} {
w := httptest.NewRecorder()
a.imageAdmission(w, runtimeRequest("POST", "/api/images/admission", fmt.Sprintf(`{"image_id":%q,"state":%q,"evidence":"historical acceptance"}`, id, state), sid))
if w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
}
})
}
}
func TestAdmissionCurrentCreateAndBlock(t *testing.T) {
id := "sha256:" + strings.Repeat("a", 64)
var body map[string]any
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.Contains(r.URL.Path, "/images/"):
writeJSON(w, map[string]any{"Id": id, "Os": "linux", "Architecture": "amd64", "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "1.2.3", "dsh-mgr.recipe": imageRecipeRev}}})
case strings.HasSuffix(r.URL.Path, "/containers/create"):
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Error(err)
}
writeJSON(w, map[string]string{"Id": "created"})
default:
w.WriteHeader(204)
}
})
seedVerifiedImage(t, a, id, "1.2.3")
if _, err := a.s.DB.Exec("DELETE FROM image_admissions; UPDATE users SET admin=1"); err != nil {
t.Fatal(err)
}
im, err := a.evaluateImage(context.Background(), id)
if err != nil || im.Status != "existing_only" || im.Selectable {
t.Fatal(im, err)
}
records := []buildRecord{{ID: "unused", ImageID: id, Version: "1.2.3", RecipeRev: imageRecipeRev, Platform: imagePlatform, Status: "success"}}
if err := a.reconcileBuilds(context.Background(), records); err != nil || records[0].Status != "success" {
t.Fatal(records, err)
}
admit := func(state string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
a.imageAdmission(w, runtimeRequest("POST", "/api/images/admission", fmt.Sprintf(`{"image_id":%q,"state":%q,"evidence":"current layout explicitly accepted"}`, id, state), sid))
return w
}
if w := admit("compatible"); w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
w := httptest.NewRecorder()
a.instances(w, runtimeRequest("POST", "/api/instances", fmt.Sprintf(`{"name":"current","image_id":%q}`, id), sid))
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
host := body["HostConfig"].(map[string]any)
mounts := host["Mounts"].([]any)
if len(mounts) != 1 || mounts[0].(map[string]any)["Target"] != "/workspace" || !strings.Contains(fmt.Sprint(body["Env"]), "DSH_HOME=/workspace") {
t.Fatal(body)
}
var instance store.Instance
if err := json.Unmarshal(w.Body.Bytes(), &instance); err != nil {
t.Fatal(err)
}
if w := admit("blocked"); w.Code != 200 {
t.Fatal(w.Body.String())
}
for _, action := range []string{"restore", "backups", "profile"} {
w := httptest.NewRecorder()
a.config(w, runtimeRequest("POST", "/api/instances/"+instance.ID+"/"+action, `{}`, sid), instance, action)
if w.Code != 409 {
t.Fatal(action, w.Code, w.Body.String())
}
}
for _, action := range []string{"start", "restart"} {
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("POST", "/api/instances/"+instance.ID+"/"+action, "", sid))
if w.Code != 409 {
t.Fatal(action, w.Code, w.Body.String())
}
}
if _, err := a.s.DB.Exec("CREATE TRIGGER audit_fail BEFORE INSERT ON audit BEGIN SELECT RAISE(FAIL,'audit unavailable'); END"); err != nil {
t.Fatal(err)
}
if w := admit("compatible"); w.Code != 500 {
t.Fatal(w.Code, w.Body.String())
}
im, err = a.evaluateImage(context.Background(), id)
if err != nil || im.Admission != "blocked" {
t.Fatal(im, err)
}
}
func TestAdmissionExperimentalAndPreflight(t *testing.T) {
id := "sha256:" + strings.Repeat("b", 64)
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { writeImageInspect(w, id, "9.8.7") })
seedVerifiedImage(t, a, id, "9.8.7")
if _, err := a.s.DB.Exec("DELETE FROM image_admissions; UPDATE image_builds SET experimental=1"); err != nil {
t.Fatal(err)
}
im, err := a.evaluateImage(context.Background(), id)
if err != nil || im.Status != "experimental" || im.Selectable {
t.Fatal(im, err)
}
for _, tc := range []struct {
handler http.HandlerFunc
body string
want int
}{{a.builds, `{"version":"9.8.7"}`, 405}, {a.builds, `{"version":"9.8.7","experimental":true}`, 405}, {a.instances, fmt.Sprintf(`{"name":"unknown","image_id":%q}`, id), 400}} {
w := httptest.NewRecorder()
tc.handler(w, runtimeRequest("POST", "/api/test", tc.body, sid))
if w.Code != tc.want {
t.Fatal(w.Code, w.Body.String())
}
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_requests").Scan(&n); err != nil || n != 0 {
t.Fatal(n, err)
}
if _, err := a.s.DB.Exec("UPDATE users SET admin=1"); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.imageAdmission(w, runtimeRequest("POST", "/api/images/admission", fmt.Sprintf(`{"image_id":%q,"state":"compatible","evidence":"accepted after startup validation"}`, id), sid))
if w.Code != 200 {
t.Fatal(w.Body.String())
}
im, err = a.evaluateImage(context.Background(), id)
if err != nil || !im.Selectable {
t.Fatal(im, err)
}
if _, err := a.s.DB.Exec("UPDATE image_builds SET status='failed'"); err != nil {
t.Fatal(err)
}
im, err = a.evaluateImage(context.Background(), id)
if err != nil || im.Selectable || im.Status != "unverified" {
t.Fatal(im, err)
}
}
internal/api/api.go
-285
View File
@@ -1,285 +0,0 @@
package api
import (
"encoding/json"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
"io"
"net"
"net/http"
"os"
"strings"
"sync"
"time"
)
type API struct {
s *store.Store
origin string
root string
docker *docker.Client
readersMu sync.Mutex
workspaceReaders map[string]int
instanceLocks keyedLocks
userLocks keyedLocks
loginMu sync.Mutex
loginSources map[string]loginSource
trustedProxies []*net.IPNet
loginSlots chan struct{}
loginConfigErr error
authKey []byte
authErr error
authLoginMu sync.Mutex
// Protected by buildQueue; invalidates image observations across deletions.
imageDeleteGeneration uint64
}
func New(s *store.Store, root string) *API {
a := &API{s: s, root: root, docker: docker.New(os.Getenv("DOCKER_HOST")), workspaceReaders: make(map[string]int), loginSources: make(map[string]loginSource), loginSlots: make(chan struct{}, 8)}
a.origin, a.loginConfigErr = configuredOrigin(os.Getenv("PANEL_PUBLIC_URL"))
var proxyErr error
a.trustedProxies, proxyErr = configuredProxyCIDRs(os.Getenv("TRUSTED_PROXY_CIDRS"), a.loginConfigErr)
if a.loginConfigErr == nil {
a.loginConfigErr = proxyErr
}
if a.loginConfigErr != nil {
a.authErr = a.loginConfigErr
} else {
a.authErr = a.initAuth()
}
return a
}
func (a *API) Handler() http.Handler {
m := http.NewServeMux()
m.HandleFunc("/api/login", a.login)
m.HandleFunc("/api/refresh", a.refresh)
m.HandleFunc("/api/runtime-session.js", runtimeSessionScript)
m.HandleFunc("/api/logout", a.logout)
m.HandleFunc("/api/me", a.me)
m.HandleFunc("/api/me/password", a.changePassword)
m.HandleFunc("/api/instances", a.instances)
m.HandleFunc("/api/instances/", a.instance)
m.HandleFunc("/api/images", a.images)
m.HandleFunc("/api/images/admission", a.imageAdmission)
m.HandleFunc("/api/releases/latest", a.latestRelease)
m.HandleFunc("/api/releases/build", a.releaseBuild)
m.HandleFunc("/api/builds", a.builds)
m.HandleFunc("/api/builds/", a.buildDetail)
m.HandleFunc("/api/ws/", a.workspace)
m.HandleFunc("/api/users", a.users)
m.HandleFunc("/api/users/reset-password", a.resetPassword)
m.HandleFunc("/api/audit", a.audit)
m.HandleFunc("/api/orphans", a.orphans)
m.HandleFunc("/dsh/", a.proxy)
return a.guard(m)
}
func writeJSON(w http.ResponseWriter, v any) {
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(v)
}
func fail(w http.ResponseWriter, n int, s string) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(n)
writeJSON(w, map[string]string{"error": s})
}
func (a *API) guard(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if a.loginConfigErr != nil {
fail(w, 503, "authentication configuration invalid")
return
}
r.Body = http.MaxBytesReader(w, r.Body, 64<<20)
if strings.HasPrefix(r.URL.Path, "/dsh/") {
next.ServeHTTP(w, r)
return
}
// API bodies and downloads are bounded; runtime WS/SSE above remain streaming.
controller := http.NewResponseController(w)
_ = controller.SetReadDeadline(time.Now().Add(60 * time.Second))
_ = controller.SetWriteDeadline(time.Now().Add(5 * time.Minute))
defer controller.SetReadDeadline(time.Time{})
defer controller.SetWriteDeadline(time.Time{})
if r.Method != "GET" && r.URL.Path != "/api/login" {
if !a.panelOriginAllowed(r) {
fail(w, 403, "origin denied")
return
}
c, e := r.Cookie("dsh_session")
if e != nil {
fail(w, 401, "authentication required")
return
}
_, csrf, e := a.s.SessionUser(c.Value)
if e != nil || r.Header.Get("X-CSRF-Token") != csrf {
fail(w, 403, "csrf denied")
return
}
if r.URL.Path != "/api/refresh" && r.URL.Path != "/api/logout" {
if _, _, e := a.user(r); e != nil {
fail(w, 401, "authentication required")
return
}
}
}
next.ServeHTTP(w, r)
})
}
func (a *API) user(r *http.Request) (store.User, string, error) {
c, e := r.Cookie("dsh_session")
if e != nil {
return store.User{}, "", e
}
u, csrf, e := a.s.SessionUser(c.Value)
if e != nil {
return store.User{}, "", e
}
if e = a.verifyAccess(r, c.Value, u.ID); e != nil {
return store.User{}, "", e
}
return u, csrf, nil
}
// dummyPasswordHash keeps login latency uniform for unknown usernames so the
// response timing cannot reveal whether an account exists.
var dummyPasswordHash, _ = bcrypt.GenerateFromPassword([]byte("timing-equalization"), bcrypt.DefaultCost)
func (a *API) login(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" {
fail(w, 405, "method not allowed")
return
}
if !a.panelOriginAllowed(r) {
fail(w, 403, "origin denied")
return
}
var in struct{ Username, Password string }
r.Body = http.MaxBytesReader(w, r.Body, 4096)
decoder := json.NewDecoder(r.Body)
if decoder.Decode(&in) != nil {
fail(w, 400, "invalid json")
return
}
if decoder.Decode(new(any)) != io.EOF {
fail(w, 400, "invalid json")
return
}
if !a.allowLoginSource(a.loginSource(r)) {
fail(w, 429, "too many login attempts; retry in a minute")
return
}
if a.authErr != nil {
fail(w, 503, "authentication unavailable")
return
}
if !a.acquireLoginSlot() {
fail(w, 503, "authentication overloaded")
return
}
defer func() { <-a.loginSlots }()
a.authLoginMu.Lock()
defer a.authLoginMu.Unlock()
locked, e := a.accountLocked(in.Username)
if e != nil {
fail(w, 503, "authentication unavailable")
return
}
if locked {
fail(w, 429, "account temporarily locked; retry later")
return
}
h, e := a.s.Password(in.Username)
if e != nil {
_ = bcrypt.CompareHashAndPassword(dummyPasswordHash, []byte(in.Password))
fail(w, 401, "invalid credentials")
return
}
if bcrypt.CompareHashAndPassword([]byte(h), []byte(in.Password)) != nil {
if e = a.failedLogin(in.Username); e != nil {
fail(w, 503, "authentication unavailable")
return
}
fail(w, 401, "invalid credentials")
return
}
u, e := a.s.UserByName(in.Username)
if e != nil {
fail(w, 401, "invalid credentials")
return
}
if _, e = a.s.DB.Exec("DELETE FROM auth_failures WHERE username=?", in.Username); e != nil {
fail(w, 503, "authentication unavailable")
return
}
sid, csrf, e := a.s.Session(u.ID)
if e != nil {
fail(w, 500, "session error")
return
}
token, e := refreshToken()
if e == nil {
_, e = a.s.DB.Exec("INSERT INTO auth_refresh(sid,token_hash) VALUES(?,?)", sid, refreshHash(token))
}
if e != nil {
a.s.DeleteSession(sid)
fail(w, 500, "session error")
return
}
if e = a.authCookies(w, r, sid, u.ID, csrf, token); e != nil {
a.s.DeleteSession(sid)
fail(w, 500, "session error")
return
}
a.s.AddAudit(u.ID, "login", u.ID, "")
writeJSON(w, map[string]any{"user": u, "csrf_token": csrf})
}
func (a *API) logout(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
if c, e := r.Cookie("dsh_session"); e == nil {
if _, e = a.s.DB.Exec("DELETE FROM sessions WHERE id=?", c.Value); e != nil {
fail(w, 500, "session error")
return
}
}
for _, name := range []string{"dsh_session", "dsh_access", "dsh_refresh", "dsh_csrf"} {
path := "/"
if name == "dsh_refresh" {
path = "/api"
}
http.SetCookie(w, &http.Cookie{Name: name, MaxAge: -1, Path: path, Secure: a.secureCookies(r), HttpOnly: name != "dsh_csrf", SameSite: http.SameSiteLaxMode})
}
w.Header().Set("Cache-Control", "no-store")
writeJSON(w, map[string]bool{"ok": true})
}
func (a *API) me(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-store")
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
u, c, e := a.user(r)
if e != nil {
fail(w, 401, "authentication required")
return
}
writeJSON(w, map[string]any{"user": u, "csrf_token": c})
}
func (a *API) panelOriginAllowed(r *http.Request) bool {
if a.loginConfigErr != nil {
return false
}
expected := a.origin
if expected == "" {
scheme := "http"
if r.TLS != nil {
scheme = "https"
}
expected = scheme + "://" + r.Host
}
return r.Header.Get("Origin") == expected
}
internal/api/auth_helpers_test.go
-28
View File
@@ -1,28 +0,0 @@
package api
import (
"net/http"
"sync"
"testing"
"time"
)
var testAccessTokens sync.Map
func registerTestAccess(t *testing.T, a *API, sid string) {
t.Helper()
if a.authErr != nil {
t.Fatal(a.authErr)
}
u, _, err := a.s.SessionUser(sid)
if err != nil {
t.Fatal(err)
}
testAccessTokens.Store(sid, a.accessJWT(sid, u.ID, time.Now().Add(accessLifetime)))
t.Cleanup(func() { testAccessTokens.Delete(sid) })
}
func addTestAccess(r *http.Request, sid string) {
if token, ok := testAccessTokens.Load(sid); ok {
r.AddCookie(&http.Cookie{Name: "dsh_access", Value: token.(string)})
}
}
internal/api/auth_security.go
-166
View File
@@ -1,166 +0,0 @@
package api
import (
"errors"
"net"
"net/http"
"net/url"
"os"
"strconv"
"strings"
"time"
)
type loginSource struct {
started time.Time
count int
}
func configuredOrigin(raw string) (string, error) {
if raw == "" {
return "", nil
}
u, err := url.Parse(raw)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") || u.Host == "" || u.Hostname() == "" || u.User != nil || (u.Path != "" && u.Path != "/") || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" {
return "", errors.New("invalid PANEL_PUBLIC_URL")
}
if strings.Contains(raw, "#") || strings.HasSuffix(u.Host, ":") {
return "", errors.New("invalid PANEL_PUBLIC_URL")
}
port := u.Port()
if port != "" {
n, err := strconv.Atoi(port)
if err != nil || n < 1 || n > 65535 {
return "", errors.New("invalid PANEL_PUBLIC_URL")
}
port = strconv.Itoa(n)
}
host := strings.ToLower(u.Hostname())
if net.ParseIP(host) == nil {
if len(host) > 253 {
return "", errors.New("invalid PANEL_PUBLIC_URL")
}
for _, label := range strings.Split(strings.TrimSuffix(host, "."), ".") {
if len(label) == 0 || len(label) > 63 || label[0] == '-' || label[len(label)-1] == '-' {
return "", errors.New("invalid PANEL_PUBLIC_URL")
}
for _, c := range label {
if !(c >= 'a' && c <= 'z' || c >= '0' && c <= '9' || c == '-') {
return "", errors.New("invalid PANEL_PUBLIC_URL")
}
}
}
} else if strings.Contains(host, ":") {
host = "[" + host + "]"
}
if port != "" && !(u.Scheme == "https" && port == "443" || u.Scheme == "http" && port == "80") {
host += ":" + port
}
return u.Scheme + "://" + host, nil
}
func configuredProxyCIDRs(raw string, prior error) ([]*net.IPNet, error) {
if prior != nil || strings.TrimSpace(raw) == "" {
return nil, prior
}
var out []*net.IPNet
for _, value := range strings.Split(raw, ",") {
value = strings.TrimSpace(value)
if value == "" {
return nil, errors.New("invalid TRUSTED_PROXY_CIDRS")
}
_, network, err := net.ParseCIDR(value)
if err != nil {
return nil, errors.New("invalid TRUSTED_PROXY_CIDRS")
}
out = append(out, network)
}
return out, nil
}
func ValidateAuthConfig() error {
_, err := configuredOrigin(os.Getenv("PANEL_PUBLIC_URL"))
_, err = configuredProxyCIDRs(os.Getenv("TRUSTED_PROXY_CIDRS"), err)
return err
}
func (a *API) ConfigError() error { return a.loginConfigErr }
func (a *API) acquireLoginSlot() bool {
a.loginMu.Lock()
defer a.loginMu.Unlock()
if a.loginSlots == nil {
a.loginSlots = make(chan struct{}, 8)
}
select {
case a.loginSlots <- struct{}{}:
return true
default:
return false
}
}
func (a *API) loginSource(r *http.Request) string {
peer, _, err := net.SplitHostPort(r.RemoteAddr)
if err != nil {
peer = r.RemoteAddr
}
ip := net.ParseIP(peer)
if ip == nil {
return peer
}
trusted := false
for _, network := range a.trustedProxies {
if network.Contains(ip) {
trusted = true
break
}
}
if trusted {
parts := strings.Split(r.Header.Get("X-Forwarded-For"), ",")
for i := len(parts) - 1; i >= 0; i-- {
candidate := net.ParseIP(strings.TrimSpace(parts[i]))
if candidate == nil {
return ip.String()
}
isProxy := false
for _, network := range a.trustedProxies {
if network.Contains(candidate) {
isProxy = true
break
}
}
if !isProxy {
return candidate.String()
}
}
}
return ip.String()
}
func (a *API) allowLoginSource(source string) bool {
now := time.Now()
a.loginMu.Lock()
defer a.loginMu.Unlock()
if a.loginSources == nil {
a.loginSources = make(map[string]loginSource)
}
for key, entry := range a.loginSources {
if now.Sub(entry.started) >= time.Minute {
delete(a.loginSources, key)
}
}
entry, ok := a.loginSources[source]
if !ok {
if len(a.loginSources) >= 1024 {
return false
}
entry = loginSource{started: now}
}
if entry.count >= 30 {
return false
}
entry.count++
a.loginSources[source] = entry
return true
}
internal/api/auth_security_test.go
-271
View File
@@ -1,271 +0,0 @@
package api
import (
"fmt"
"io"
"net"
"net/http"
"net/http/cookiejar"
"net/http/httptest"
"net/url"
"strings"
"sync"
"testing"
"time"
)
func TestConfiguredOriginRejectsInvalidOrigins(t *testing.T) {
for _, raw := range []string{"https://", "https://:8443", "https://panel.test?", "https://panel.test:bad", "ftp://panel.test", "https://panel.test/path"} {
if _, err := configuredOrigin(raw); err == nil {
t.Errorf("accepted invalid origin %q", raw)
}
}
for _, raw := range []string{"https://panel.test", "https://panel.test:8443", "http://127.0.0.1:8088"} {
if got, err := configuredOrigin(raw); err != nil || got != raw {
t.Errorf("origin %q => %q, %v", raw, got, err)
}
}
}
func TestLoginSourceNilMapAndInvalidForwardedChain(t *testing.T) {
a := &API{}
if !a.allowLoginSource("127.0.0.1") {
t.Fatal("nil source map rejected")
}
a.trustedProxies = mustCIDRs(t, "10.0.0.0/8")
r := httptest.NewRequest("POST", "http://panel.test/api/login", nil)
r.RemoteAddr = "10.0.0.1:1234"
r.Header.Set("X-Forwarded-For", "198.51.100.4,not-an-ip,10.0.0.2")
if got := a.loginSource(r); got != "10.0.0.1" {
t.Fatalf("invalid chain crossed trust boundary: %s", got)
}
}
func mustCIDRs(t *testing.T, raw string) []*net.IPNet {
t.Helper()
proxies, err := configuredProxyCIDRs(raw, nil)
if err != nil {
t.Fatal(err)
}
return proxies
}
func TestSecurityConfigValidation(t *testing.T) {
for _, raw := range []string{"https://panel.test:0", "https://panel.test:65536", "https://panel.test:", "https://panel.test#", "https://bad_host", "https://[bad]", "https://[email protected]"} {
if _, err := configuredOrigin(raw); err == nil {
t.Errorf("accepted %q", raw)
}
}
for raw, want := range map[string]string{"https://PANEL.EXAMPLE:443/": "https://panel.example", "http://PANEL.EXAMPLE:80": "http://panel.example", "https://[::1]:443": "https://[::1]"} {
if got, err := configuredOrigin(raw); err != nil || got != want {
t.Errorf("%q => %q, %v", raw, got, err)
}
}
for _, raw := range []string{"bad", "10.0.0.1", "10.0.0.0/8,", ",", "10.0.0.0/33"} {
if _, err := configuredProxyCIDRs(raw, nil); err == nil {
t.Errorf("accepted CIDRs %q", raw)
}
}
for _, key := range []string{"PANEL_PUBLIC_URL", "TRUSTED_PROXY_CIDRS"} {
t.Run(key, func(t *testing.T) {
t.Setenv("PANEL_PUBLIC_URL", "")
t.Setenv("TRUSTED_PROXY_CIDRS", "")
t.Setenv(key, "invalid")
if ValidateAuthConfig() == nil {
t.Fatal("configuration accepted")
}
a := New(nil, t.TempDir())
if a.ConfigError() == nil {
t.Fatal("missing config error")
}
for _, path := range []string{"/api/login", "/api/me", "/api/refresh", "/dsh/example/"} {
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, httptest.NewRequest("POST", path, strings.NewReader("{}")))
if w.Code != 503 {
t.Fatalf("%s: %d", path, w.Code)
}
}
})
}
}
func TestSecurityTLSCookieJarLifecycle(t *testing.T) {
a, _ := authFixture(t)
a.origin = ""
a.loginSlots = nil
a.loginSources = nil
srv := httptest.NewTLSServer(a.Handler())
defer srv.Close()
client := srv.Client()
client.Jar, _ = cookiejar.New(nil)
u, _ := url.Parse(srv.URL + "/api/me")
request := func(method, path, body string, status int) []*http.Cookie {
t.Helper()
r, _ := http.NewRequest(method, srv.URL+path, strings.NewReader(body))
r.Header.Set("Origin", srv.URL)
r.Header.Set("X-CSRF-Token", cookieValue(client.Jar.Cookies(u), "dsh_csrf"))
resp, err := client.Do(r)
if err != nil {
t.Fatal(err)
}
defer resp.Body.Close()
data, _ := io.ReadAll(resp.Body)
if resp.StatusCode != status {
t.Fatalf("%s: %d %s", path, resp.StatusCode, data)
}
return resp.Cookies()
}
check := func(cookies []*http.Cookie, deleting bool) {
t.Helper()
if len(cookies) != 4 {
t.Fatalf("cookies: %v", cookies)
}
for _, c := range cookies {
path := "/"
if c.Name == "dsh_refresh" {
path = "/api"
}
if !c.Secure || c.HttpOnly != (c.Name != "dsh_csrf") || c.Path != path || c.SameSite != http.SameSiteLaxMode || (deleting && c.MaxAge != -1) {
t.Fatalf("cookie: %+v", c)
}
}
}
check(request("POST", "/api/login", `{"username":"alice","password":"password-123"}`, 200), false)
request("GET", "/api/me", "", 200)
old := cookieValue(client.Jar.Cookies(u), "dsh_refresh")
check(request("POST", "/api/refresh", "", 200), false)
if old == cookieValue(client.Jar.Cookies(u), "dsh_refresh") {
t.Fatal("refresh not rotated")
}
request("GET", "/api/me", "", 200)
check(request("POST", "/api/logout", "", 200), true)
if len(client.Jar.Cookies(u)) != 0 {
t.Fatal("logout left cookies")
}
request("GET", "/api/me", "", 401)
}
func TestSecurityLoginBudgetsAndUntrustedHeaders(t *testing.T) {
a, _ := authFixture(t)
a.origin = ""
request := func(peer, origin, body string) int {
r := httptest.NewRequest("POST", "http://panel.test/api/login", strings.NewReader(body))
r.RemoteAddr = peer + ":1234"
r.Header.Set("Origin", origin)
r.Header.Set("X-Forwarded-For", fmt.Sprintf("198.51.100.%d", len(a.loginSources)))
r.Header.Set("X-Forwarded-Proto", "https")
r.Header.Set("X-Forwarded-Host", "evil.test")
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
return w.Code
}
for i := 0; i < 35; i++ {
if got := request("192.0.2.1", "https://evil.test", "{}"); got != 403 {
t.Fatal(got)
}
for _, body := range []string{"{", "{} garbage", "{} {}", strings.Repeat(" ", 4097)} {
if got := request("192.0.2.1", "http://panel.test", body); got != 400 {
t.Fatal(got)
}
}
}
if len(a.loginSources) != 0 {
t.Fatal("invalid requests consumed budget")
}
for i := 0; i < 30; i++ {
if got := request("192.0.2.1", "http://panel.test", "{}"); got != 401 {
t.Fatal(i, got)
}
}
if got := request("192.0.2.1", "http://panel.test", "{}"); got != 429 {
t.Fatal(got)
}
if got := request("192.0.2.2", "http://panel.test", "{}"); got != 401 {
t.Fatal("other source blocked", got)
}
if len(a.loginSources) != 2 {
t.Fatal("forwarded headers changed source")
}
r := httptest.NewRequest("POST", "http://panel.test", nil)
r.Header.Set("X-Forwarded-Proto", "https")
if a.secureCookies(r) {
t.Fatal("untrusted proto changed cookie policy")
}
a.origin = "https://panel.test"
if !a.secureCookies(r) {
t.Fatal("configured HTTPS not secure")
}
}
func TestSecurityLoginSlotOverloadRecovery(t *testing.T) {
a, req := authFixture(t)
for i := 0; i < 8; i++ {
if !a.acquireLoginSlot() {
t.Fatal("slot unavailable")
}
}
if w := req("POST", "/api/login", "{}", "", nil); w.Code != 503 {
t.Fatal("overload not rejected", w.Code)
}
<-a.loginSlots
if w := req("POST", "/api/login", "{}", "", nil); w.Code != 401 {
t.Fatal("released slot not usable", w.Code)
}
if len(a.loginSlots) != 7 {
t.Fatal("request leaked slot")
}
}
func TestSecurityConcurrentSlotsAndBoundedBuckets(t *testing.T) {
a := &API{}
var wg sync.WaitGroup
results := make(chan bool, 64)
for i := 0; i < 64; i++ {
wg.Add(1)
go func() { defer wg.Done(); results <- a.acquireLoginSlot() }()
}
wg.Wait()
close(results)
count := 0
for ok := range results {
if ok {
count++
}
}
if count != 8 {
t.Fatalf("slots: %d", count)
}
<-a.loginSlots
if !a.acquireLoginSlot() {
t.Fatal("released slot unavailable")
}
for i := 0; i < 1024; i++ {
if !a.allowLoginSource(fmt.Sprint(i)) {
t.Fatal(i)
}
}
if a.allowLoginSource("overflow") || len(a.loginSources) != 1024 {
t.Fatal("unbounded buckets")
}
a.loginSources["0"] = loginSource{started: time.Now().Add(-time.Minute), count: 30}
if !a.allowLoginSource("overflow") || len(a.loginSources) != 1024 {
t.Fatal("expired bucket not reclaimed")
}
b := &API{}
accepted := make(chan bool, 64)
for i := 0; i < 64; i++ {
wg.Add(1)
go func() { defer wg.Done(); accepted <- b.allowLoginSource("same") }()
}
wg.Wait()
close(accepted)
count = 0
for ok := range accepted {
if ok {
count++
}
}
if count != 30 {
t.Fatalf("concurrent budget: %d", count)
}
}
internal/api/auth_test.go
-93
View File
@@ -1,93 +0,0 @@
package api
import (
"encoding/json"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
)
func TestAuthenticationAndAdmin(t *testing.T) {
s, e := store.Open(filepath.Join(t.TempDir(), "test.sqlite"))
if e != nil {
t.Fatal(e)
}
defer s.Close()
h, _ := bcrypt.GenerateFromPassword([]byte("test-password-123"), bcrypt.MinCost)
if e = s.CreateUser(store.User{ID: "admin", Username: "admin", Admin: true, Quota: 1024}, string(h)); e != nil {
t.Fatal(e)
}
a := New(s, t.TempDir())
a.origin = "http://panel.test"
var access *http.Cookie
request := func(method, path, body, csrf string, cookie *http.Cookie) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, path, strings.NewReader(body))
r.Header.Set("Origin", a.origin)
r.Header.Set("X-CSRF-Token", csrf)
if cookie != nil {
r.AddCookie(cookie)
if access != nil {
r.AddCookie(access)
}
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
return w
}
if w := request("GET", "/api/me", "", "", nil); w.Code != 401 {
t.Fatal(w.Code)
}
w := request("POST", "/api/login", `{"username":"admin","password":"test-password-123"}`, "", nil)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
var login struct {
CSRF string `json:"csrf_token"`
}
json.Unmarshal(w.Body.Bytes(), &login)
var cookie *http.Cookie
for _, c := range w.Result().Cookies() {
if c.Name == "dsh_access" {
access = c
}
if c.Name == "dsh_session" {
cookie = c
}
}
if cookie == nil || !cookie.HttpOnly || cookie.Secure {
t.Fatal("wrong local cookie settings")
}
if w = request("GET", "/api/users", "", "", cookie); w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
if w = request("POST", "/api/users", `{"username":"member","password":"member-password-123"}`, "", cookie); w.Code != 403 {
t.Fatal("CSRF accepted", w.Code)
}
if w = request("POST", "/api/users", `{"username":"member","password":"member-password-123"}`, login.CSRF, cookie); w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
if w = request("POST", "/api/logout", "", login.CSRF, cookie); w.Code != 200 {
t.Fatal(w.Code)
}
if w = request("GET", "/api/me", "", "", cookie); w.Code != 401 {
t.Fatal("session not revoked")
}
}
func TestOriginRequiredWithoutPublicURL(t *testing.T) {
a := &API{}
for _, tc := range []struct {
origin string
want bool
}{{"", false}, {"http://evil.test", false}, {"http://panel.test", true}} {
r := httptest.NewRequest("POST", "http://panel.test/api/login", nil)
r.Header.Set("Origin", tc.origin)
if a.panelOriginAllowed(r) != tc.want {
t.Fatalf("origin %q", tc.origin)
}
}
}
internal/api/auth_tokens.go
-299
View File
@@ -1,299 +0,0 @@
package api
import (
"context"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"database/sql"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"log"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"time"
)
const accessLifetime = 5 * time.Minute
var errAuthentication = errors.New("authentication required")
func (a *API) initAuth() error {
if err := os.MkdirAll(a.root, 0700); err != nil {
return err
}
path := filepath.Join(a.root, "auth-signing.key")
f, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
if err == nil {
key := make([]byte, 32)
if _, err = rand.Read(key); err == nil {
_, err = f.Write(key)
}
if err == nil {
err = f.Sync()
}
closeErr := f.Close()
if err != nil {
return err
}
if closeErr != nil {
return closeErr
}
} else if !os.IsExist(err) {
return err
}
info, err := os.Lstat(path)
if err != nil {
return err
}
if !info.Mode().IsRegular() || info.Mode().Perm() != 0600 {
return errors.New("unsafe signing key permissions")
}
a.authKey, err = os.ReadFile(path)
if err != nil {
return err
}
if len(a.authKey) != 32 {
return errors.New("invalid signing key")
}
_, err = a.s.DB.Exec(`CREATE TABLE IF NOT EXISTS auth_refresh(sid TEXT PRIMARY KEY REFERENCES sessions(id) ON DELETE CASCADE, token_hash TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS auth_failures(username TEXT PRIMARY KEY, failures INTEGER NOT NULL, locked_until INTEGER NOT NULL);`)
if err == nil {
_, err = a.s.DB.Exec(`CREATE TABLE IF NOT EXISTS auth_refresh_previous(sid TEXT PRIMARY KEY REFERENCES auth_refresh(sid) ON DELETE CASCADE, token_hash TEXT NOT NULL);`)
}
return err
}
// Derivation makes retries return the same successor without storing plaintext
// credentials. Domain and session binding prevent cross-session token reuse.
func (a *API) nextRefresh(sid, old string) string {
mac := hmac.New(sha256.New, a.authKey)
mac.Write([]byte("refresh-successor\x00" + sid + "\x00" + old))
return base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
}
func (a *API) rotateRefresh(sid, old string) (string, error) {
tx, err := a.s.DB.Begin()
if err != nil {
return "", err
}
defer tx.Rollback()
var current, previous string
err = tx.QueryRow(`SELECT r.token_hash, COALESCE(p.token_hash, '')
FROM auth_refresh r JOIN sessions s ON s.id=r.sid
LEFT JOIN auth_refresh_previous p ON p.sid=r.sid
WHERE r.sid=? AND s.expires>?`, sid, time.Now().Unix()).Scan(&current, &previous)
if err != nil {
return "", err
}
hash := refreshHash(old)
next := a.nextRefresh(sid, old)
if hash == previous && refreshHash(next) == current {
return next, tx.Commit()
}
if hash != current {
return "", errAuthentication
}
if _, err = tx.Exec("UPDATE auth_refresh SET token_hash=? WHERE sid=?", refreshHash(next), sid); err != nil {
return "", err
}
if _, err = tx.Exec("INSERT INTO auth_refresh_previous(sid,token_hash) VALUES(?,?) ON CONFLICT(sid) DO UPDATE SET token_hash=excluded.token_hash", sid, hash); err != nil {
return "", err
}
return next, tx.Commit()
}
type accessClaims struct {
Subject string `json:"sub"`
Session string `json:"sid"`
Issuer string `json:"iss"`
Audience string `json:"aud"`
Expires int64 `json:"exp"`
}
const jwtHeader = "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9"
func (a *API) accessJWT(sid, uid string, expires time.Time) string {
payload, _ := json.Marshal(accessClaims{uid, sid, "dsh-panel", "dsh-panel", expires.Unix()})
body := jwtHeader + "." + base64.RawURLEncoding.EncodeToString(payload)
mac := hmac.New(sha256.New, a.authKey)
mac.Write([]byte(body))
return body + "." + base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
}
func (a *API) verifyAccess(r *http.Request, sid, uid string) error {
if a.authErr != nil || len(a.authKey) != 32 {
return errAuthentication
}
c, err := r.Cookie("dsh_access")
if err != nil {
return errAuthentication
}
p := strings.Split(c.Value, ".")
if len(p) != 3 || p[0] != jwtHeader {
return errAuthentication
}
sig, err := base64.RawURLEncoding.DecodeString(p[2])
if err != nil {
return errAuthentication
}
mac := hmac.New(sha256.New, a.authKey)
mac.Write([]byte(p[0] + "." + p[1]))
if !hmac.Equal(sig, mac.Sum(nil)) {
return errAuthentication
}
data, err := base64.RawURLEncoding.DecodeString(p[1])
if err != nil {
return errAuthentication
}
var claims accessClaims
if json.Unmarshal(data, &claims) != nil || claims.Subject != uid || claims.Session != sid || claims.Issuer != "dsh-panel" || claims.Audience != "dsh-panel" || claims.Expires <= time.Now().Unix() || claims.Expires > time.Now().Add(accessLifetime).Unix() {
return errAuthentication
}
return nil
}
func refreshToken() (string, error) {
b := make([]byte, 32)
_, err := rand.Read(b)
return base64.RawURLEncoding.EncodeToString(b), err
}
func refreshHash(token string) string {
h := sha256.Sum256([]byte(token))
return hex.EncodeToString(h[:])
}
func (a *API) secureCookies(r *http.Request) bool {
secure := r.TLS != nil
if u, err := url.Parse(a.origin); err == nil && u.Scheme == "https" {
secure = true
}
return secure
}
func (a *API) authCookies(w http.ResponseWriter, r *http.Request, sid, uid, csrf, refresh string) error {
// Persist recovery cookies across mobile browser process eviction, but never
// extend them beyond the database session's absolute expiry on refresh.
var expires int64
if err := a.s.DB.QueryRow("SELECT expires FROM sessions WHERE id=?", sid).Scan(&expires); err != nil {
return err
}
now := time.Now()
remaining := int(expires - now.Unix())
if remaining <= 0 {
return errAuthentication
}
for _, c := range []*http.Cookie{
{Name: "dsh_session", Value: sid, Path: "/", HttpOnly: true, MaxAge: remaining},
{Name: "dsh_access", Value: a.accessJWT(sid, uid, time.Now().Add(accessLifetime)), Path: "/", HttpOnly: true, MaxAge: int(accessLifetime.Seconds())},
{Name: "dsh_refresh", Value: refresh, Path: "/api", HttpOnly: true, MaxAge: remaining},
{Name: "dsh_csrf", Value: csrf, Path: "/", MaxAge: remaining},
} {
c.Expires = time.Unix(expires, 0)
if c.Name == "dsh_access" {
c.Expires = now.Add(accessLifetime)
}
c.Secure = a.secureCookies(r)
c.SameSite = http.SameSiteLaxMode
http.SetCookie(w, c)
}
w.Header().Set("Cache-Control", "no-store")
return nil
}
func (a *API) refresh(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
if a.authErr != nil {
fail(w, 503, "authentication unavailable")
return
}
sid, err := r.Cookie("dsh_session")
if err != nil {
fail(w, 401, "authentication required")
return
}
var uid, csrf string
err = a.s.DB.QueryRow("SELECT uid,csrf FROM sessions WHERE id=? AND expires>?", sid.Value, time.Now().Unix()).Scan(&uid, &csrf)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
fail(w, 401, "authentication required")
} else {
fail(w, 500, "session error")
}
return
}
u, err := a.s.UserByID(uid)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
fail(w, 401, "authentication required")
} else {
fail(w, 500, "session error")
}
return
}
old, err := r.Cookie("dsh_refresh")
if err != nil {
fail(w, 401, "authentication required")
return
}
token, err := a.rotateRefresh(sid.Value, old.Value)
if err != nil {
if errors.Is(err, sql.ErrNoRows) || errors.Is(err, errAuthentication) {
fail(w, 401, "authentication required")
} else {
fail(w, 500, "session error")
}
return
}
if err = a.authCookies(w, r, sid.Value, u.ID, csrf, token); err != nil {
fail(w, 500, "session error")
return
}
writeJSON(w, map[string]any{"user": u, "csrf_token": csrf})
}
func (a *API) accountLocked(username string) (bool, error) {
var until int64
err := a.s.DB.QueryRow("SELECT locked_until FROM auth_failures WHERE username=?", username).Scan(&until)
if err == sql.ErrNoRows {
return false, nil
}
return until > time.Now().Unix(), err
}
// RunSessionCleanup removes expired sessions, their refresh tokens and elapsed
// login locks every ten minutes; without it the sessions table grows without
// bound because SessionUser only filters rows on read.
func (a *API) RunSessionCleanup(ctx context.Context) {
ticker := time.NewTicker(10 * time.Minute)
defer ticker.Stop()
for {
a.cleanupSessions(time.Now().Unix())
select {
case <-ctx.Done():
return
case <-ticker.C:
}
}
}
func (a *API) cleanupSessions(now int64) {
for _, q := range []struct{ name, sql string }{
{"refresh tokens", "DELETE FROM auth_refresh WHERE sid IN (SELECT id FROM sessions WHERE expires<=?)"},
{"sessions", "DELETE FROM sessions WHERE expires<=?"},
{"login locks", "DELETE FROM auth_failures WHERE locked_until>0 AND locked_until<=?"},
} {
if _, err := a.s.DB.Exec(q.sql, now); err != nil {
log.Printf("session cleanup (%s): %v", q.name, err)
}
}
}
func (a *API) failedLogin(username string) error {
_, err := a.s.DB.Exec(`INSERT INTO auth_failures(username,failures,locked_until) VALUES(?,1,0)
ON CONFLICT(username) DO UPDATE SET
failures=CASE WHEN locked_until>0 AND locked_until<=? THEN 1 ELSE failures+1 END,
locked_until=CASE WHEN locked_until>0 AND locked_until<=? THEN 0 WHEN failures+1>=5 THEN ? ELSE 0 END`, username, time.Now().Unix(), time.Now().Unix(), time.Now().Add(15*time.Minute).Unix())
return err
}
internal/api/auth_tokens_test.go
-348
View File
@@ -1,348 +0,0 @@
package api
import (
"bytes"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
)
func authFixture(t *testing.T) (*API, func(string, string, string, string, []*http.Cookie) *httptest.ResponseRecorder) {
t.Helper()
s, e := store.Open(filepath.Join(t.TempDir(), "db"))
if e != nil {
t.Fatal(e)
}
t.Cleanup(s.Close)
h, _ := bcrypt.GenerateFromPassword([]byte("password-123"), bcrypt.MinCost)
for _, id := range []string{"alice", "bob"} {
if e = s.CreateUser(store.User{ID: id, Username: id}, string(h)); e != nil {
t.Fatal(e)
}
}
a := New(s, t.TempDir())
a.origin = "https://panel.test"
return a, func(method, path, body, csrf string, cookies []*http.Cookie) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, path, strings.NewReader(body))
r.Header.Set("Origin", a.origin)
r.Header.Set("X-CSRF-Token", csrf)
for _, c := range cookies {
r.AddCookie(c)
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
return w
}
}
func cookieValue(cs []*http.Cookie, name string) string {
for _, c := range cs {
if c.Name == name {
return c.Value
}
}
return ""
}
func TestRefreshSessionLookupFailureIsServerError(t *testing.T) {
a, req := authFixture(t)
w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil)
cookies := w.Result().Cookies()
a.s.Close()
r := httptest.NewRequest("POST", "/api/refresh", nil)
for _, c := range cookies {
r.AddCookie(c)
}
w = httptest.NewRecorder()
a.refresh(w, r)
if w.Code != 500 || len(w.Result().Cookies()) != 0 {
t.Fatal(w.Code, w.Header(), w.Body.String())
}
}
func TestAccessRefreshLifecycle(t *testing.T) {
a, req := authFixture(t)
w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
cs := w.Result().Cookies()
sid := cookieValue(cs, "dsh_session")
csrf := cookieValue(cs, "dsh_csrf")
for _, name := range []string{"dsh_access", "dsh_refresh"} {
v := cookieValue(cs, name)
if v == "" || strings.Contains(w.Body.String(), v) {
t.Fatal("missing/leaked", name)
}
}
for _, c := range cs {
if !c.Secure || (c.Name != "dsh_csrf" && !c.HttpOnly) {
t.Fatal("unsafe cookie", c.Name)
}
}
if w = req("GET", "/api/me", "", "", []*http.Cookie{{Name: "dsh_session", Value: sid}}); w.Code != 401 {
t.Fatal("missing access accepted", w.Code)
}
expired := append([]*http.Cookie{}, cs...)
for i, c := range expired {
if c.Name == "dsh_access" {
expired[i] = &http.Cookie{Name: c.Name, Value: a.accessJWT(sid, "alice", time.Now().Add(-time.Minute))}
}
}
if w = req("GET", "/api/me", "", "", expired); w.Code != 401 {
t.Fatal("expired access accepted")
}
if w = req("POST", "/api/refresh", "", "", expired); w.Code != 403 {
t.Fatal("refresh bypassed csrf", w.Code)
}
w = req("POST", "/api/refresh", "", csrf, expired)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
fresh := w.Result().Cookies()
if cookieValue(fresh, "dsh_refresh") == cookieValue(cs, "dsh_refresh") {
t.Fatal("not rotated")
}
if w = req("POST", "/api/refresh", "", csrf, cs); w.Code != 200 {
t.Fatal("retry rejected", w.Code)
}
if cookieValue(w.Result().Cookies(), "dsh_refresh") != cookieValue(fresh, "dsh_refresh") {
t.Fatal("retry must return the same successor")
}
if w = req("GET", "/api/me", "", "", fresh); w.Code != 200 {
t.Fatal(w.Code)
}
if w = req("POST", "/api/logout", "", csrf, fresh); w.Code != 200 {
t.Fatal(w.Code)
}
if w = req("GET", "/api/me", "", "", fresh); w.Code != 401 {
t.Fatal("logout access valid")
}
if w = req("POST", "/api/refresh", "", csrf, fresh); w.Code != 403 {
t.Fatal("logout refresh valid")
}
}
func TestPersistentCookiesRecoverAfterBrowserRestart(t *testing.T) {
a, req := authFixture(t)
w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil)
if w.Code != 200 {
t.Fatal(w.Code)
}
cookies := w.Result().Cookies()
sid := cookieValue(cookies, "dsh_session")
var expires int64
if err := a.s.DB.QueryRow("SELECT expires FROM sessions WHERE id=?", sid).Scan(&expires); err != nil {
t.Fatal(err)
}
var restored []*http.Cookie
for _, c := range cookies {
if c.MaxAge <= 0 || c.Expires.IsZero() {
t.Fatalf("nonpersistent cookie: %s", c.Name)
}
if c.Name == "dsh_access" {
continue
} // Access has expired while backgrounded.
if c.Expires.Unix() != expires {
t.Fatalf("wrong expiry: %s", c.Name)
}
restored = append(restored, c)
}
if w = req("GET", "/api/me", "", "", restored); w.Code != 401 {
t.Fatal(w.Code)
}
// Simulate an older session without sleeping; rotation must not reset its TTL.
expires = time.Now().Add(time.Hour).Unix()
if _, err := a.s.DB.Exec("UPDATE sessions SET expires=? WHERE id=?", expires, sid); err != nil {
t.Fatal(err)
}
w = req("POST", "/api/refresh", "", cookieValue(restored, "dsh_csrf"), restored)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
fresh := w.Result().Cookies()
for _, c := range fresh {
if c.Name == "dsh_access" {
if c.MaxAge != int(accessLifetime.Seconds()) {
t.Fatal("access lifetime changed")
}
continue
}
if c.Expires.Unix() != expires || c.MaxAge <= 0 || c.MaxAge > 3600 {
t.Fatalf("session extended: %s", c.Name)
}
}
if w = req("GET", "/api/me", "", "", fresh); w.Code != 200 {
t.Fatal(w.Code)
}
w = req("POST", "/api/logout", "", cookieValue(fresh, "dsh_csrf"), fresh)
if w.Code != 200 {
t.Fatal(w.Code)
}
for _, c := range w.Result().Cookies() {
if c.MaxAge >= 0 {
t.Fatalf("cookie not cleared: %s", c.Name)
}
}
if w = req("GET", "/api/me", "", "", fresh); w.Code != 401 {
t.Fatal("revoked session accepted")
}
}
func TestAccountLockAndPersistentKey(t *testing.T) {
a, req := authFixture(t)
for i := 0; i < 5; i++ {
if w := req("POST", "/api/login", `{"username":"alice","password":"wrong"}`, "", nil); w.Code != 401 {
t.Fatal(w.Code)
}
}
if w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil); w.Code != 429 {
t.Fatal("not locked", w.Code)
}
if w := req("POST", "/api/login", `{"username":"bob","password":"password-123"}`, "", nil); w.Code != 200 {
t.Fatal("other account locked", w.Code)
}
b := New(a.s, a.root)
if !bytes.Equal(a.authKey, b.authKey) || b.authErr != nil {
t.Fatal("key not persistent", b.authErr)
}
fi, e := os.Stat(filepath.Join(a.root, "auth-signing.key"))
if e != nil || fi.Mode().Perm() != 0600 {
t.Fatal("unsafe key")
}
if locked, e := b.accountLocked("alice"); e != nil || !locked {
t.Fatal("lock not persistent", e)
}
if _, e = a.s.DB.Exec("UPDATE auth_failures SET locked_until=? WHERE username='alice'", time.Now().Add(-time.Second).Unix()); e != nil {
t.Fatal(e)
}
if w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil); w.Code != 200 {
t.Fatal("lock did not expire", w.Code)
}
}
func TestRefreshOriginConcurrentRotationAndJWTBinding(t *testing.T) {
a, req := authFixture(t)
w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil)
cs := w.Result().Cookies()
sid := cookieValue(cs, "dsh_session")
csrf := cookieValue(cs, "dsh_csrf")
var stored string
if err := a.s.DB.QueryRow("SELECT token_hash FROM auth_refresh WHERE sid=?", sid).Scan(&stored); err != nil || stored != refreshHash(cookieValue(cs, "dsh_refresh")) {
t.Fatal("refresh not hashed", err)
}
for _, token := range []string{cookieValue(cs, "dsh_access") + "x", a.accessJWT("other", "alice", time.Now().Add(accessLifetime)), a.accessJWT(sid, "bob", time.Now().Add(accessLifetime)), "eyJhbGciOiJub25lIn0.e30."} {
if w = req("GET", "/api/me", "", "", []*http.Cookie{{Name: "dsh_session", Value: sid}, {Name: "dsh_access", Value: token}}); w.Code != 401 {
t.Fatal("invalid JWT accepted")
}
}
for _, origin := range []string{"", "https://evil.test"} {
r := httptest.NewRequest("POST", "/api/refresh", nil)
r.Header.Set("Origin", origin)
r.Header.Set("X-CSRF-Token", csrf)
for _, c := range cs {
r.AddCookie(c)
}
w = httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
if w.Code != 403 {
t.Fatal("origin accepted", w.Code)
}
}
var wg sync.WaitGroup
codes := make(chan int, 2)
for i := 0; i < 2; i++ {
wg.Add(1)
go func() { defer wg.Done(); codes <- req("POST", "/api/refresh", "", csrf, cs).Code }()
}
wg.Wait()
close(codes)
counts := map[int]int{}
for code := range codes {
counts[code]++
}
if counts[200] != 2 {
t.Fatal(counts)
}
b := New(a.s, a.root)
r := httptest.NewRequest("GET", "/api/me", nil)
for _, c := range cs {
r.AddCookie(c)
}
w = httptest.NewRecorder()
b.Handler().ServeHTTP(w, r)
if w.Code != 200 {
t.Fatal("restart rejected access", w.Code)
}
a.s.DeleteSession(sid)
if w = req("GET", "/api/me", "", "", cs); w.Code != 401 {
t.Fatal("deleted binding accepted")
}
}
func TestUnsafeAuthKeyFailsClosed(t *testing.T) {
a, req := authFixture(t)
if err := os.Chmod(filepath.Join(a.root, "auth-signing.key"), 0644); err != nil {
t.Fatal(err)
}
b := New(a.s, a.root)
if b.authErr == nil {
t.Fatal("unsafe key accepted")
}
a.authErr = b.authErr
if w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil); w.Code != 503 {
t.Fatal(w.Code)
}
}
func TestSessionCleanupRemovesExpiredRows(t *testing.T) {
a, _ := authFixture(t)
expired, _, e := a.s.Session("alice")
if e != nil {
t.Fatal(e)
}
current, _, e := a.s.Session("alice")
if e != nil {
t.Fatal(e)
}
past := time.Now().Add(-time.Minute).Unix()
if _, e = a.s.DB.Exec("UPDATE sessions SET expires=? WHERE id=?", past, expired); e != nil {
t.Fatal(e)
}
token, e := refreshToken()
if e != nil {
t.Fatal(e)
}
if _, e = a.s.DB.Exec("INSERT INTO auth_refresh(sid,token_hash) VALUES(?,?)", expired, refreshHash(token)); e != nil {
t.Fatal(e)
}
future := time.Now().Add(time.Hour).Unix()
if _, e = a.s.DB.Exec("INSERT INTO auth_failures(username,failures,locked_until) VALUES('expired-lock',5,?)", past); e != nil {
t.Fatal(e)
}
if _, e = a.s.DB.Exec("INSERT INTO auth_failures(username,failures,locked_until) VALUES('active-lock',5,?)", future); e != nil {
t.Fatal(e)
}
a.cleanupSessions(time.Now().Unix())
var n int
for _, q := range []string{
"SELECT count(*) FROM sessions WHERE id='" + expired + "'",
"SELECT count(*) FROM auth_refresh WHERE sid='" + expired + "'",
"SELECT count(*) FROM auth_failures WHERE username='expired-lock'",
} {
if err := a.s.DB.QueryRow(q).Scan(&n); err != nil || n != 0 {
t.Fatalf("expired row retained: %s (%d, %v)", q, n, err)
}
}
for _, q := range []string{
"SELECT count(*) FROM sessions WHERE id='" + current + "'",
"SELECT count(*) FROM auth_failures WHERE username='active-lock'",
} {
if err := a.s.DB.QueryRow(q).Scan(&n); err != nil || n != 1 {
t.Fatalf("live row removed: %s (%d, %v)", q, n, err)
}
}
}
internal/api/backup_management.go
-315
View File
@@ -1,315 +0,0 @@
package api
import (
"context"
"database/sql"
"errors"
"io/fs"
"net/http"
"os"
"regexp"
"strings"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
var snapshotIDPattern = regexp.MustCompile(`^(pre-restore-[a-f0-9]{32}|failed-restore-[a-f0-9]{32}|upgrade-[a-f0-9]{32}-snapshot|failed-home-[a-f0-9]{32})$`)
var errUnsafeManagementPath = errors.New("unsafe storage path")
func managementPathError(w http.ResponseWriter, err error) {
if errors.Is(err, errUnsafeManagementPath) {
fail(w, 400, "unsafe storage path")
return
}
fail(w, 500, "storage unavailable")
}
type snapshotInfo struct {
ID string `json:"id"`
Size int64 `json:"size"`
CreatedAt time.Time `json:"created_at"`
Protected bool `json:"protected"`
ProtectionReason string `json:"protection_reason"`
SizeError string `json:"size_error,omitempty"`
Restorable bool `json:"restorable"`
RestoreReason string `json:"restore_reason"`
SourceVersion string `json:"source_version,omitempty"`
SourceImage string `json:"source_image,omitempty"`
}
func managementRoot(base string, parts ...string) (*os.Root, error) {
root, err := os.OpenRoot(base)
if err != nil {
return nil, err
}
for _, part := range parts {
if part == "" || part == "." || part == ".." || strings.ContainsAny(part, "/\\") {
root.Close()
return nil, errUnsafeManagementPath
}
st, err := root.Lstat(part)
if err != nil {
root.Close()
return nil, err
}
if !st.IsDir() {
root.Close()
return nil, errUnsafeManagementPath
}
next, err := root.OpenRoot(part)
root.Close()
if err != nil {
return nil, err
}
actual, err := next.Stat(".")
if err != nil {
next.Close()
return nil, err
}
if !os.SameFile(st, actual) {
next.Close()
return nil, errUnsafeManagementPath
}
root = next
}
return root, nil
}
func (a *API) backupProtection(i store.Instance) string {
if i.Status == "error" {
return "instance recovery required"
}
var exists, pending int
if err := a.s.DB.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='upgrade_journal'").Scan(&exists); err != nil {
return "recovery journal unavailable"
}
if exists != 0 {
if err := a.s.DB.QueryRow("SELECT count(*) FROM upgrade_journal WHERE instance_id=? AND (phase IS NULL OR phase NOT IN ('complete','rolled_back'))", i.ID).Scan(&pending); err != nil {
return "recovery journal unavailable"
}
if pending != 0 {
return "instance recovery required"
}
}
return ""
}
func snapshotBytes(ctx context.Context, root *os.Root, name string) (int64, error) {
var total int64
count := 0
err := fs.WalkDir(root.FS(), name, func(p string, d fs.DirEntry, err error) error {
if ctx.Err() != nil {
return ctx.Err()
}
if err != nil {
return err
}
count++
if count > 100000 || strings.Count(p, "/") > 128 {
return errors.New("tree limit")
}
st, err := d.Info()
if err != nil {
return err
}
if st.Mode().IsRegular() {
total += st.Size()
}
return nil
})
return total, err
}
func (a *API) backupManagement(w http.ResponseWriter, r *http.Request, i store.Instance, action string) {
w, flush := deferJSONResponse(w)
defer flush()
if r.Method != http.MethodDelete && !(action == "snapshots" && r.Method == http.MethodGet) {
fail(w, 405, "method not allowed")
return
}
id := r.URL.Query().Get("id")
if r.Method == http.MethodDelete && ((action == "snapshots" && !snapshotIDPattern.MatchString(id)) || (action == "backups" && !backupIDPattern.MatchString(id))) {
fail(w, 400, "invalid ID")
return
}
lock := a.instanceLocks.Acquire(i.ID)
defer lock.Release()
user := a.userLocks.Acquire(i.UID)
defer user.Release()
current, err := a.s.GetInstance(i.ID)
if err != nil || current.UID != i.UID {
fail(w, 404, "not found")
return
}
i = current
reason := a.backupProtection(i)
if r.Method == http.MethodDelete && reason != "" {
fail(w, 409, reason)
return
}
parts := []string{"workspaces", i.UID, "instances", i.ID}
if action == "backups" {
parts = []string{"backups", i.UID, i.ID}
}
root, err := managementRoot(a.root, parts...)
if errors.Is(err, os.ErrNotExist) {
if action == "backups" {
a.deleteBackup(w, i, nil, id)
return
}
if r.Method == http.MethodGet {
writeJSON(w, []snapshotInfo{})
return
}
fail(w, 404, "not found")
return
}
if err != nil {
managementPathError(w, err)
return
}
defer root.Close()
if r.Method == http.MethodGet {
f, err := root.Open(".")
if err != nil {
fail(w, 500, "snapshot listing failed")
return
}
entries, err := f.ReadDir(-1)
f.Close()
if err != nil {
fail(w, 500, "snapshot listing failed")
return
}
out := []snapshotInfo{}
for _, entry := range entries {
if !snapshotIDPattern.MatchString(entry.Name()) || !entry.IsDir() {
continue
}
st, err := entry.Info()
if err != nil {
fail(w, 500, "snapshot metadata unavailable")
return
}
s := snapshotInfo{ID: entry.Name(), CreatedAt: st.ModTime(), Protected: reason != "", ProtectionReason: reason}
s.SourceVersion, s.SourceImage, s.RestoreReason = a.snapshotCompatibility(i, s.ID)
s.Restorable = s.RestoreReason == "" || s.RestoreReason == snapshotUnknownReason
if reason != "" {
s.Restorable = false
s.RestoreReason = reason
}
s.Size, err = snapshotBytes(r.Context(), root, entry.Name())
if err != nil {
s.Size = -1
s.SizeError = "snapshot size unavailable: " + err.Error()
s.Restorable = false
s.RestoreReason = "snapshot size unavailable"
}
out = append(out, s)
}
writeJSON(w, out)
return
}
if action == "backups" {
a.deleteBackup(w, i, root, id)
return
}
st, err := root.Lstat(id)
if errors.Is(err, os.ErrNotExist) {
fail(w, 404, "snapshot not found")
return
}
if err != nil {
managementPathError(w, err)
return
}
if !st.IsDir() {
fail(w, 400, "unsafe snapshot path")
return
}
a.deleteSnapshot(w, i, id, func() error { return root.RemoveAll(id) })
}
func (a *API) deleteSnapshot(w http.ResponseWriter, i store.Instance, id string, remove func() error) {
audit := func(state string) error {
_, err := a.s.DB.Exec("INSERT INTO audit(uid,action,target,detail,at) VALUES(?,?,?,?,?)", i.UID, "snapshot.delete."+state, i.ID, id, time.Now().Unix())
return err
}
if err := audit("started"); err != nil {
fail(w, 500, "audit unavailable")
return
}
if err := remove(); err != nil {
if err := audit("failed"); err != nil {
fail(w, 500, "snapshot deletion failed; audit unavailable")
return
}
fail(w, 500, "snapshot deletion failed")
return
}
if err := audit("completed"); err != nil {
fail(w, 500, "snapshot removed; completion audit unavailable")
return
}
writeJSON(w, map[string]bool{"ok": true})
}
func (a *API) deleteBackup(w http.ResponseWriter, i store.Instance, root *os.Root, id string) {
name := id + ".dshbak"
var st os.FileInfo
statErr := error(os.ErrNotExist)
if root != nil {
st, statErr = root.Lstat(name)
}
if statErr != nil && !errors.Is(statErr, os.ErrNotExist) {
fail(w, 500, "backup unavailable")
return
}
if statErr == nil && !st.Mode().IsRegular() {
fail(w, 400, "unsafe backup path")
return
}
tx, err := a.s.DB.Begin()
if err != nil {
fail(w, 500, "backup index unavailable")
return
}
defer tx.Rollback()
var exists, indexed int
err = tx.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='backups'").Scan(&exists)
if err == nil && exists != 0 {
var result sql.Result
result, err = tx.Exec("DELETE FROM backups WHERE id=? AND uid=? AND instance_id=?", id, i.UID, i.ID)
if err == nil {
n, e := result.RowsAffected()
err = e
indexed = int(n)
}
}
if err != nil {
fail(w, 500, "backup index unavailable")
return
}
if statErr != nil && indexed == 0 {
fail(w, 404, "backup not found")
return
}
if _, err = tx.Exec("INSERT INTO audit(uid,action,target,detail,at) VALUES(?,?,?,?,?)", i.UID, "backup.delete", i.ID, id, time.Now().Unix()); err != nil {
fail(w, 500, "audit unavailable")
return
}
if statErr == nil {
if err = root.Remove(name); err != nil {
fail(w, 500, "backup deletion failed")
return
}
}
// Filesystem and SQLite cannot commit atomically; a retry clears a stale index after a failed commit.
if err = tx.Commit(); err != nil {
fail(w, 500, "backup removed; index update failed, retry deletion")
return
}
writeJSON(w, map[string]bool{"ok": true})
}
internal/api/backup_management_test.go
-413
View File
@@ -1,413 +0,0 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestSnapshotManagement(t *testing.T) {
a, i, home := configTestAPI(t, false)
a.docker = nil // These operations must never inspect or stop Docker.
parent := filepath.Dir(home)
suffix := strings.Repeat("a", 32)
names := []string{"pre-restore-" + suffix, "failed-restore-" + suffix, "upgrade-" + suffix + "-snapshot", "failed-home-" + suffix}
for _, name := range append(append([]string{}, names...), "pre-restore-no", "upgrade-"+suffix, "other") {
if err := os.Mkdir(filepath.Join(parent, name), 0700); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(parent, name, "data"), "abc")
}
outside := t.TempDir()
configWrite(t, filepath.Join(outside, "keep"), "outside")
if err := os.Symlink(outside, filepath.Join(parent, names[0], "link")); err != nil {
t.Fatal(err)
}
if err := os.Link(filepath.Join(parent, names[0], "data"), filepath.Join(parent, names[0], "hardlink")); err != nil {
t.Fatal(err)
}
w := configCall(a, i, "GET", "snapshots", "")
var got []snapshotInfo
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &got) != nil || len(got) != 4 {
t.Fatal(w.Code, w.Body.String())
}
for _, s := range got {
want := int64(3)
if s.ID == names[0] {
want = 6
}
if s.Size != want || s.Protected || s.ProtectionReason != "" || s.SizeError != "" || s.CreatedAt.IsZero() {
t.Fatalf("%+v", s)
}
}
for _, name := range names {
w = configCall(a, i, "DELETE", "snapshots?id="+name, "")
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
if _, err := os.Stat(filepath.Join(parent, name)); !os.IsNotExist(err) {
t.Fatal(err)
}
}
for _, p := range []string{home, filepath.Join(parent, "other"), filepath.Join(outside, "keep")} {
if _, err := os.Stat(p); err != nil {
t.Fatal(err)
}
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='snapshot.delete.completed' AND uid=? AND target=?", i.UID, i.ID).Scan(&count); err != nil || count != 4 {
t.Fatal(count, err)
}
if w = configCall(a, i, "DELETE", "snapshots?id="+names[0], ""); w.Code != 404 {
t.Fatal(w.Code)
}
}
func TestManagementInvalidPaths(t *testing.T) {
a, i, home := configTestAPI(t, false)
a.docker = nil
for _, action := range []string{"snapshots", "backups"} {
for _, id := range []string{"", "home", "..", "../home", "/home", strings.Repeat("A", 32), "pre-restore-" + strings.Repeat("a", 32) + "/../home"} {
if w := configCall(a, i, "DELETE", action+"?id="+url.QueryEscape(id), ""); w.Code != 400 {
t.Fatal(action, id, w.Code)
}
}
}
id := "failed-home-" + strings.Repeat("b", 32)
outside := t.TempDir()
configWrite(t, filepath.Join(outside, "keep"), "safe")
if err := os.Symlink(outside, filepath.Join(filepath.Dir(home), id)); err != nil {
t.Fatal(err)
}
if w := configCall(a, i, "DELETE", "snapshots?id="+id, ""); w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
if w := configCall(a, i, "GET", "snapshots", ""); w.Code != 200 || strings.TrimSpace(w.Body.String()) != "[]" {
t.Fatal(w.Code, w.Body.String())
}
if err := os.Rename(filepath.Dir(home), filepath.Dir(home)+"-saved"); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Dir(home)); err != nil {
t.Fatal(err)
}
for _, method := range []string{"GET", "DELETE"} {
if w := configCall(a, i, method, "snapshots?id="+id, ""); w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
}
if _, err := os.Stat(filepath.Join(outside, "keep")); err != nil {
t.Fatal(err)
}
}
func TestManagementProtection(t *testing.T) {
for _, state := range []string{"error", "pending", "journal-error", "complete", "rolled_back"} {
t.Run(state, func(t *testing.T) {
a, i, home := configTestAPI(t, false)
id := "pre-restore-" + strings.Repeat("c", 32)
if err := os.Mkdir(filepath.Join(filepath.Dir(home), id), 0700); err != nil {
t.Fatal(err)
}
var err error
if state == "error" {
_, err = a.s.DB.Exec("UPDATE instances SET status='error' WHERE id=?", i.ID)
} else if state == "journal-error" {
_, err = a.s.DB.Exec("CREATE TABLE upgrade_journal(broken TEXT)")
} else {
_, err = a.s.DB.Exec("CREATE TABLE upgrade_journal(instance_id TEXT,phase TEXT)")
if err == nil {
_, err = a.s.DB.Exec("INSERT INTO upgrade_journal VALUES(?,?)", i.ID, state)
}
}
if err != nil {
t.Fatal(err)
}
protected := state != "complete" && state != "rolled_back"
w := configCall(a, i, "GET", "snapshots", "")
var got []snapshotInfo
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &got) != nil || len(got) != 1 || got[0].Protected != protected || (got[0].ProtectionReason != "") != protected {
t.Fatal(w.Code, w.Body.String())
}
w = configCall(a, i, "DELETE", "snapshots?id="+id, "")
want := 200
if protected {
want = 409
}
if w.Code != want {
t.Fatal(w.Code, w.Body.String())
}
if protected {
w = configCall(a, i, "DELETE", "backups?id="+strings.Repeat("a", 32), "")
if w.Code != 409 {
t.Fatal(w.Code, w.Body.String())
}
}
})
}
}
func TestSnapshotSizeError(t *testing.T) {
a, i, home := configTestAPI(t, false)
id := "failed-restore-" + strings.Repeat("d", 32)
deep := filepath.Join(filepath.Dir(home), id)
for n := 0; n < 130; n++ {
deep = filepath.Join(deep, "d")
}
if err := os.MkdirAll(deep, 0700); err != nil {
t.Fatal(err)
}
w := configCall(a, i, "GET", "snapshots", "")
var got []snapshotInfo
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &got) != nil || len(got) != 1 || got[0].Size != -1 || got[0].SizeError == "" {
t.Fatal(w.Code, w.Body.String())
}
}
func TestBackupDelete(t *testing.T) {
for _, mode := range []string{"indexed", "legacy", "stale", "missing-parent", "index-error", "symlink"} {
t.Run(mode, func(t *testing.T) {
a, i, _ := configTestAPI(t, false)
a.docker = nil
id := strings.Repeat("e", 32)
dir := filepath.Join(a.root, "backups", i.UID, i.ID)
if err := os.MkdirAll(dir, 0700); err != nil {
t.Fatal(err)
}
file := filepath.Join(dir, id+".dshbak")
if mode != "stale" && mode != "symlink" {
configWrite(t, file, "encrypted")
}
if mode == "symlink" {
if err := os.Symlink(t.TempDir(), file); err != nil {
t.Fatal(err)
}
}
if mode == "indexed" || mode == "stale" || mode == "missing-parent" {
if err := a.indexBackup(i, id, "manual", 2, 9); err != nil {
t.Fatal(err)
}
}
if mode == "index-error" {
if _, err := a.s.DB.Exec("CREATE TABLE backups(broken TEXT)"); err != nil {
t.Fatal(err)
}
}
if mode == "missing-parent" {
if err := os.RemoveAll(filepath.Join(a.root, "backups")); err != nil {
t.Fatal(err)
}
}
w := configCall(a, i, "DELETE", "backups?id="+id, "")
want := 200
if mode == "index-error" {
want = 500
}
if mode == "symlink" {
want = 400
}
if w.Code != want {
t.Fatal(w.Code, w.Body.String())
}
if want != 200 {
if _, err := os.Lstat(file); err != nil {
t.Fatal(err)
}
return
}
if _, err := os.Stat(file); !os.IsNotExist(err) {
t.Fatal(err)
}
if mode != "legacy" {
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM backups WHERE id=?", id).Scan(&count); err != nil || count != 0 {
t.Fatal(count, err)
}
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='backup.delete' AND target=? AND detail=?", i.ID, id).Scan(&count); err != nil || count != 1 {
t.Fatal(count, err)
}
})
}
}
func TestManagementOwnershipRecheck(t *testing.T) {
a, i, _ := configTestAPI(t, false)
if _, err := a.s.DB.Exec("UPDATE instances SET uid='someone-else' WHERE id=?", i.ID); err != nil {
t.Fatal(err)
}
for _, action := range []string{"snapshots", "snapshots?id=pre-restore-" + strings.Repeat("a", 32), "backups?id=" + strings.Repeat("a", 32)} {
method := "DELETE"
if action == "snapshots" {
method = "GET"
}
if w := configCall(a, i, method, action, ""); w.Code != 404 {
t.Fatal(w.Code, w.Body.String())
}
}
}
func TestManagementAuditFailure(t *testing.T) {
a, i, home := configTestAPI(t, false)
id := strings.Repeat("f", 32)
snapshot := filepath.Join(filepath.Dir(home), "pre-restore-"+id)
if err := os.Mkdir(snapshot, 0700); err != nil {
t.Fatal(err)
}
dir := filepath.Join(a.root, "backups", i.UID, i.ID)
if err := os.MkdirAll(dir, 0700); err != nil {
t.Fatal(err)
}
file := filepath.Join(dir, id+".dshbak")
configWrite(t, file, "encrypted")
if err := a.indexBackup(i, id, "manual", 2, 9); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("DROP TABLE audit"); err != nil {
t.Fatal(err)
}
for _, action := range []string{"snapshots?id=pre-restore-" + id, "backups?id=" + id} {
if w := configCall(a, i, "DELETE", action, ""); w.Code != 500 {
t.Fatal(w.Code, w.Body.String())
}
}
for _, p := range []string{snapshot, file} {
if _, err := os.Stat(p); err != nil {
t.Fatal(err)
}
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM backups WHERE id=?", id).Scan(&count); err != nil || count != 1 {
t.Fatal(count, err)
}
}
func TestSnapshotScanCancelled(t *testing.T) {
root, err := os.OpenRoot(t.TempDir())
if err != nil {
t.Fatal(err)
}
defer root.Close()
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err := snapshotBytes(ctx, root, "."); err != context.Canceled {
t.Fatal(err)
}
}
func TestSnapshotDeletionReleasesSQLConnection(t *testing.T) {
a, i, _ := configTestAPI(t, false)
a.s.DB.SetMaxOpenConns(1)
entered, release, done := make(chan struct{}), make(chan struct{}), make(chan struct{})
w := httptest.NewRecorder()
go func() {
defer close(done)
a.deleteSnapshot(w, i, "snapshot", func() error {
close(entered)
<-release
return nil
})
}()
defer func() { close(release); <-done }()
select {
case <-entered:
case <-time.After(5 * time.Second):
t.Fatal("removal did not start")
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
var count int
if err := a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM audit WHERE action='snapshot.delete.started'").Scan(&count); err != nil || count != 1 {
t.Fatalf("SQL blocked during removal or initial audit not persisted: count=%d err=%v", count, err)
}
}
func TestSnapshotDeletionAuditOutcomes(t *testing.T) {
for _, mode := range []string{"success", "remove-failed", "start-failed", "completion-audit-failed", "failure-audit-failed"} {
t.Run(mode, func(t *testing.T) {
a, i, _ := configTestAPI(t, false)
if mode == "start-failed" {
if _, err := a.s.DB.Exec("DROP TABLE audit"); err != nil {
t.Fatal(err)
}
}
w := httptest.NewRecorder()
called := false
a.deleteSnapshot(w, i, "snapshot", func() error {
called = true
if strings.Contains(mode, "audit-failed") {
if _, err := a.s.DB.Exec(`CREATE TRIGGER reject_terminal BEFORE INSERT ON audit WHEN NEW.action != 'snapshot.delete.started' BEGIN SELECT RAISE(FAIL, 'audit unavailable'); END`); err != nil {
t.Fatal(err)
}
}
if mode == "remove-failed" || mode == "failure-audit-failed" {
return os.ErrPermission
}
return nil
})
want := 500
if mode == "success" {
want = 200
}
if w.Code != want || called != (mode != "start-failed") {
t.Fatal(w.Code, w.Body.String(), called)
}
if mode == "start-failed" {
return
}
for _, state := range []string{"started", "completed", "failed"} {
wantCount := 0
if state == "started" || (state == "completed" && mode == "success") || (state == "failed" && mode == "remove-failed") {
wantCount = 1
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action=? AND uid=? AND target=? AND detail=?", "snapshot.delete."+state, i.UID, i.ID, "snapshot").Scan(&count); err != nil || count != wantCount {
t.Fatal(state, count, err)
}
}
})
}
}
func TestManagementPathErrorClassification(t *testing.T) {
for _, tc := range []struct {
name string
err error
want int
}{
{"unsafe", errUnsafeManagementPath, 400},
{"wrapped-unsafe", fmt.Errorf("path: %w", errUnsafeManagementPath), 400},
{"permission", &os.PathError{Op: "lstat", Path: "snapshot", Err: os.ErrPermission}, 500},
{"closed", os.ErrClosed, 500},
} {
t.Run(tc.name, func(t *testing.T) {
w := httptest.NewRecorder()
managementPathError(w, tc.err)
if w.Code != tc.want {
t.Fatal(w.Code, w.Body.String())
}
})
}
a, i, _ := configTestAPI(t, false)
base := t.TempDir()
configWrite(t, filepath.Join(base, "file"), "not a directory")
a.root = filepath.Join(base, "file", "child")
if w := configCall(a, i, "GET", "snapshots", ""); w.Code != 500 {
t.Fatal(w.Code, w.Body.String())
}
a.root = filepath.Join(base, "missing")
if w := configCall(a, i, "GET", "snapshots", ""); w.Code != 200 || strings.TrimSpace(w.Body.String()) != "[]" {
t.Fatal(w.Code, w.Body.String())
}
if w := configCall(a, i, "DELETE", "snapshots?id=pre-restore-"+strings.Repeat("a", 32), ""); w.Code != 404 {
t.Fatal(w.Code, w.Body.String())
}
}
internal/api/builds_read.go
-112
View File
@@ -1,112 +0,0 @@
package api
import (
"context"
"database/sql"
"errors"
"net/http"
"strconv"
"strings"
"time"
)
// Summary queries never load log bodies; history cannot inflate a page response.
const buildSummaryColumns = "id,version,status,error,image_id,created_at,requested_by,COALESCE(recipe_rev,''),COALESCE(platform,''),COALESCE(finished_at,''),experimental,release_id,source_repository,source_commit"
func scanBuildSummary(row interface{ Scan(...any) error }, b *buildRecord) error {
return row.Scan(&b.ID, &b.Version, &b.Status, &b.Error, &b.ImageID, &b.Created, &b.RequestedBy, &b.RecipeRev, &b.Platform, &b.Finished, &b.Experimental, &b.ReleaseID, &b.SourceRepository, &b.SourceCommit)
}
func (a *API) listBuilds(w http.ResponseWriter, r *http.Request) {
limit, offset := 20, 0
for key, dest := range map[string]*int{"limit": &limit, "offset": &offset} {
if values, ok := r.URL.Query()[key]; ok {
if len(values) != 1 {
fail(w, 400, "invalid pagination")
return
}
value, err := strconv.Atoi(values[0])
if err != nil || value < 0 || (key == "limit" && (value < 1 || value > 100)) {
fail(w, 400, "invalid pagination")
return
}
*dest = value
}
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
fail(w, 500, "database error")
return
}
rows, err := a.s.DB.QueryContext(ctx, "SELECT "+buildSummaryColumns+" FROM image_builds ORDER BY created_at DESC,id DESC LIMIT ? OFFSET ?", limit+1, offset)
if err != nil {
fail(w, 500, "database error")
return
}
result := []buildRecord{}
for rows.Next() {
var b buildRecord
if err = scanBuildSummary(rows, &b); err != nil {
break
}
result = append(result, b)
}
if err == nil {
err = rows.Err()
}
rows.Close()
if err != nil {
fail(w, 500, "database error")
return
}
hasMore := len(result) > limit
if hasMore {
result = result[:limit]
}
if err := a.reconcileBuilds(ctx, result); err != nil {
fail(w, 502, "docker image reconciliation failed")
return
}
a.startBuildWorker()
w.Header().Set("X-Has-More", strconv.FormatBool(hasMore))
writeJSON(w, result)
}
func (a *API) buildDetail(w http.ResponseWriter, r *http.Request) {
if _, _, err := a.user(r); err != nil {
fail(w, 401, "authentication required")
return
}
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
id := strings.TrimPrefix(r.URL.Path, "/api/builds/")
if id == "" || len(id) > 128 || strings.Contains(id, "/") {
fail(w, 404, "build not found")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
var b buildRecord
if err == nil {
err = a.s.DB.QueryRowContext(ctx, "SELECT "+buildSummaryColumns+",logs FROM image_builds WHERE id=?", id).Scan(&b.ID, &b.Version, &b.Status, &b.Error, &b.ImageID, &b.Created, &b.RequestedBy, &b.RecipeRev, &b.Platform, &b.Finished, &b.Experimental, &b.ReleaseID, &b.SourceRepository, &b.SourceCommit, &b.Logs)
}
if errors.Is(err, sql.ErrNoRows) {
fail(w, 404, "build not found")
return
}
if err != nil {
fail(w, 500, "database error")
return
}
b.Logs = sanitizeBuildLog(b.Logs)
writeJSON(w, b)
}
internal/api/builds_read_test.go
-167
View File
@@ -1,167 +0,0 @@
package api
import (
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestBuildHistoryPaginationAndOnDemandLogs(t *testing.T) {
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Errorf("unexpected Docker request: %s", r.URL) })
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
t.Fatal(err)
}
for n := 0; n < 125; n++ {
_, err := a.s.DB.Exec("INSERT INTO image_builds(id,version,status,logs,created_at) VALUES(?,?,?,?,?)", fmt.Sprintf("task-%03d", n), "1.2.3", "failed", "token=secret-value"+string(rune(10))+strings.Repeat("x", buildLogLimit), "2026-01-01")
if err != nil {
t.Fatal(err)
}
}
request := func(target, session, method string) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, target, nil)
if session != "" {
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: session})
addTestAccess(r, session)
}
w := httptest.NewRecorder()
if strings.HasPrefix(r.URL.Path, "/api/builds/") {
a.buildDetail(w, r)
} else {
a.builds(w, r)
}
return w
}
for _, tc := range []struct {
query string
count int
first, more string
}{
{"", 20, "task-124", "true"}, {"?limit=100", 100, "task-124", "true"},
{"?limit=20&offset=120", 5, "task-004", "false"}, {"?offset=125", 0, "", "false"},
{"?limit=20&offset=105", 20, "task-019", "false"}, {"?limit=20&offset=104", 20, "task-020", "true"},
} {
w := request("/api/builds"+tc.query, sid, "GET")
var records []buildRecord
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &records) != nil || len(records) != tc.count {
t.Fatalf("%s: %d %s", tc.query, w.Code, w.Body.String())
}
if tc.count > 0 && records[0].ID != tc.first {
t.Fatal(records[0].ID)
}
if w.Header().Get("X-Has-More") != tc.more || strings.Contains(w.Body.String(), "logs") || strings.Contains(w.Body.String(), "secret-value") {
t.Fatal("unbounded or invalid summary response", w.Body.String())
}
}
for _, query := range []string{"?limit=0", "?limit=101", "?limit=-1", "?offset=-1", "?offset=x", "?limit=1&limit=2", "?offset=9999999999999999999999999"} {
if w := request("/api/builds"+query, sid, "GET"); w.Code != 400 {
t.Fatal(query, w.Code)
}
}
w := request("/api/builds/task-000", sid, "GET")
var b buildRecord
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &b) != nil || b.ID != "task-000" || b.Logs == "" || len(b.Logs) > buildLogLimit || strings.Contains(b.Logs, "secret-value") {
t.Fatal(w.Code, w.Body.String())
}
for _, path := range []string{"/api/builds/missing", "/api/builds/", "/api/builds/task-000/extra"} {
if w := request(path, sid, "GET"); w.Code != 404 {
t.Fatal(path, w.Code)
}
}
if w := request("/api/builds/task-000", "", "GET"); w.Code != 401 {
t.Fatal(w.Code)
}
if w := request("/api/builds", "", "GET"); w.Code != 401 {
t.Fatal(w.Code)
}
if w := request("/api/builds/task-000", sid, "POST"); w.Code != 405 {
t.Fatal(w.Code)
}
}
func TestBuildDetailRouteAndAccess(t *testing.T) {
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker call") })
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
t.Fatal(err)
}
if _, err = a.s.DB.Exec("INSERT INTO image_builds(id,version,status,created_at) VALUES('task','1.2.3','failed','now')"); err != nil {
t.Fatal(err)
}
_, csrf, err := a.s.SessionUser(sid)
if err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
path, method string
access bool
status int
}{
{"/api/builds/task", "GET", true, 200}, {"/api/builds/task", "GET", false, 401},
{"/api/builds/task", "POST", true, 405}, {"/api/builds/", "GET", true, 404},
{"/api/builds/task%2Fextra", "GET", true, 404}, {"/api/builds/task/extra", "GET", true, 404},
} {
r := httptest.NewRequest(tc.method, tc.path, nil)
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: sid})
if tc.access {
addTestAccess(r, sid)
}
if tc.method != "GET" {
r.Header.Set("Origin", "http://example.com")
r.Header.Set("X-CSRF-Token", csrf)
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
if w.Code != tc.status {
t.Fatalf("%s %s: %d %s", tc.method, tc.path, w.Code, w.Body.String())
}
}
}
func TestBuildPageReconcilesOnlySelectedRecords(t *testing.T) {
unavailable := false
calls := 0
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
calls++
if unavailable {
w.WriteHeader(500)
return
}
if !strings.HasPrefix(r.URL.Path, "/v1.47/images/") {
t.Errorf("unexpected request %s", r.URL)
}
w.WriteHeader(404)
})
seedVerifiedImage(t, a, "first-image", "1.2.3")
seedVerifiedImage(t, a, "second-image", "1.2.4")
if _, err := a.s.DB.Exec("UPDATE image_builds SET created_at=CASE version WHEN '1.2.3' THEN 'z' ELSE 'a' END"); err != nil {
t.Fatal(err)
}
request := func(offset int) *httptest.ResponseRecorder {
r := httptest.NewRequest("GET", fmt.Sprintf("/api/builds?limit=1&offset=%d", offset), nil)
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: sid})
addTestAccess(r, sid)
w := httptest.NewRecorder()
a.builds(w, r)
return w
}
w := request(0)
if w.Code != 200 || calls != 1 || !strings.Contains(w.Body.String(), "missing") {
t.Fatal(w.Code, calls, w.Body.String())
}
var state string
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE version='1.2.4'").Scan(&state); err != nil || state != "success" {
t.Fatal(state, err)
}
unavailable = true
if w = request(1); w.Code != 502 {
t.Fatal(w.Code, w.Body.String())
}
}
internal/api/config.go
-699
View File
@@ -1,699 +0,0 @@
package api
import (
"bytes"
"context"
"encoding/json"
"errors"
"git.misaka.ren/m1saka/dsh-web/internal/backup"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"gopkg.in/yaml.v3"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"syscall"
"time"
)
var backupIDPattern = regexp.MustCompile("^[a-f0-9]{32}$")
var configBodySlots = make(chan struct{}, 4)
func (a *API) config(w http.ResponseWriter, r *http.Request, i store.Instance, action string) {
if action == "snapshots" && r.Method == http.MethodGet && r.URL.Query().Get("download") != "" {
a.snapshotDownload(w, r, i)
return
}
if action == "snapshots" && r.Method == http.MethodPost {
a.snapshotRestore(w, r, i)
return
}
if action == "snapshots" || (action == "backups" && r.Method == http.MethodDelete) {
a.backupManagement(w, r, i, action)
return
}
controller := http.NewResponseController(w)
if r.Method != http.MethodGet || action != "backups" || r.URL.Query().Get("download") == "" {
var flush func()
w, flush = deferJSONResponse(w)
defer flush()
}
if r.Method != http.MethodGet {
select {
case configBodySlots <- struct{}{}:
defer func() {
r.Body = http.NoBody
<-configBodySlots
}()
default:
fail(w, 503, "configuration uploads busy; retry later")
return
}
limit := int64(4096)
multipartUpload := action == "backups" && strings.HasPrefix(r.Header.Get("Content-Type"), "multipart/form-data")
if action == "profile" {
limit = 2 << 20
} else if multipartUpload {
limit = min(64<<20, backup.MaxEncryptedBytes+16384)
}
if !configReadBody(w, r, controller, limit) {
return
}
if multipartUpload {
defer func() {
if r.MultipartForm != nil {
r.MultipartForm.RemoveAll()
}
}()
if r.ParseMultipartForm(1<<20) != nil {
fail(w, 400, "invalid or oversized backup upload")
return
}
}
}
inst := a.instanceLocks.Acquire(i.ID)
locked := inst
defer func() {
if locked != nil {
locked.Release()
}
}()
if r.Method != http.MethodGet {
a.readersMu.Lock()
readers := a.workspaceReaders[i.ID]
a.readersMu.Unlock()
if readers > 0 {
fail(w, 409, "instance files busy; retry later")
return
}
user := a.userLocks.Acquire(i.UID)
defer user.Release()
}
current, e := a.s.GetInstance(i.ID)
if e != nil || current.UID != i.UID {
fail(w, 404, "not found")
return
}
i = current
if r.Method != http.MethodGet {
admissionMu.RLock()
defer admissionMu.RUnlock()
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
fail(w, 500, "database error")
return
}
var blocked int
if err = a.s.DB.QueryRowContext(r.Context(), "SELECT count(*) FROM image_admissions WHERE image_id=? AND state='blocked'", i.ImageID).Scan(&blocked); err != nil {
fail(w, 500, "database error")
return
}
if blocked > 0 {
fail(w, 409, "image is blocked; configuration mutations and automatic restarts denied")
return
}
}
if i.Status == "error" || a.upgradeRecoveryRequired(i) {
fail(w, 409, "instance recovery required")
return
}
home := filepath.Join(a.root, "workspaces", i.UID, "instances", i.ID, "home")
dir := filepath.Join(a.root, "backups", i.UID, i.ID)
storage, e := os.OpenRoot(a.root)
if e != nil {
fail(w, 500, "storage unavailable")
return
}
defer storage.Close()
if configCheckPath(storage, filepath.ToSlash(filepath.Join("backups", i.UID, i.ID, "archive"))) != nil || configCheckPath(storage, filepath.ToSlash(filepath.Join("workspaces", i.UID, "instances", i.ID, "home", "profile"))) != nil {
fail(w, 400, "unsafe storage path")
return
}
if e = os.MkdirAll(dir, 0700); e != nil {
fail(w, 500, "backup storage unavailable")
return
}
wasRunning := false
if r.Method != "GET" {
ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
defer cancel()
c, e := a.docker.Inspect(ctx, i.ContainerID)
if e != nil {
fail(w, 502, "container inspection failed")
return
}
wasRunning = c.State.Running
if c.State.Paused || c.State.Restarting {
fail(w, 409, "unsupported container state")
return
}
if c.State.Running && action == "profile" {
fail(w, 409, "stop instance before changing configuration or backing up")
return
}
}
switch action {
case "profile":
parent, e := upgradeRoot(a.root, i)
if e != nil {
fail(w, 400, "unsafe home")
return
}
defer parent.Close()
if e = configCheckPath(parent, "home/profiles/web/cordis.patch.yml"); e != nil {
fail(w, 400, "unsafe profile")
return
}
root, e := configProfileRoot(parent, r.Method == "PUT")
if e != nil {
if r.Method == "GET" && errors.Is(e, os.ErrNotExist) {
writeJSON(w, map[string]string{"content": ""})
return
}
fail(w, 404, "home unavailable")
return
}
defer root.Close()
const path = "cordis.patch.yml"
if r.Method == "GET" {
f, e := root.OpenFile(path, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if errors.Is(e, os.ErrNotExist) {
writeJSON(w, map[string]string{"content": ""})
return
}
if e != nil {
fail(w, 400, "profile unavailable")
return
}
defer f.Close()
st, e := f.Stat()
if e != nil || !st.Mode().IsRegular() || st.Size() > 1<<20 {
fail(w, 400, "invalid profile")
return
}
b, e := io.ReadAll(io.LimitReader(f, 1<<20))
if e != nil {
fail(w, 500, "read failed")
return
}
writeJSON(w, map[string]string{"content": string(b)})
return
}
if r.Method != "PUT" {
fail(w, 405, "method not allowed")
return
}
var in struct {
Content string `json:"content"`
Restart bool `json:"restart"`
}
if json.NewDecoder(http.MaxBytesReader(w, r.Body, 2<<20)).Decode(&in) != nil || len(in.Content) > 1<<20 {
fail(w, 400, "invalid profile")
return
}
var document any
if strings.TrimSpace(in.Content) == "" {
in.Content = "[]\n"
}
if yaml.Unmarshal([]byte(in.Content), &document) != nil {
fail(w, 400, "invalid YAML")
return
}
var previous int64
if st, err := root.Lstat(path); err == nil {
if !st.Mode().IsRegular() {
fail(w, 400, "invalid profile")
return
}
previous = st.Size()
} else if !errors.Is(err, os.ErrNotExist) {
fail(w, 400, "profile unavailable")
return
}
if e = a.quotaGrowth(i.UID, int64(len(in.Content))-previous); e != nil {
fail(w, 413, "quota exceeded or usage unavailable")
return
}
tmp := ".panel-" + store.ID()
f, e := root.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
if e != nil {
fail(w, 500, "profile write failed")
return
}
defer root.Remove(tmp)
_, e = f.WriteString(in.Content)
if e == nil {
e = f.Chown(1000, 1000)
}
if e == nil {
e = f.Sync()
}
f.Close()
if e == nil {
e = root.Rename(tmp, path)
}
if e != nil {
fail(w, 500, "profile write failed")
return
}
a.s.AddAudit(i.UID, "profile.save", i.ID, "")
if in.Restart {
if a.configRestart(i) != nil {
fail(w, 502, "profile saved but restart failed")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 45*time.Second)
defer cancel()
if a.upgradeReady(ctx, i.ContainerID, i.ID, a.runtimeHost(r)) != nil {
a.s.SetInstance(i.ID, "error")
fail(w, 502, "profile saved but runtime readiness failed")
return
}
a.s.SetInstance(i.ID, "running")
}
writeJSON(w, map[string]bool{"ok": true})
case "backups":
if r.Method == "GET" {
if id := r.URL.Query().Get("download"); id != "" {
if !backupIDPattern.MatchString(id) {
fail(w, 400, "invalid backup ID")
return
}
w.Header().Set("Content-Type", "application/octet-stream")
w.Header().Set("Content-Disposition", "attachment; filename="+id+".dshbak")
root, e := os.OpenRoot(dir)
if e != nil {
fail(w, 404, "backup unavailable")
return
}
defer root.Close()
f, e := root.OpenFile(id+".dshbak", os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if e != nil {
fail(w, 404, "backup unavailable")
return
}
defer f.Close()
st, e := f.Stat()
if e != nil || !st.Mode().IsRegular() {
fail(w, 400, "invalid backup")
return
}
locked.Release()
locked = nil
http.ServeContent(w, r, id+".dshbak", st.ModTime(), f)
return
}
entries, e := os.ReadDir(dir)
if e != nil {
fail(w, 500, "backup listing failed")
return
}
out := []map[string]any{}
for _, entry := range entries {
if entry.IsDir() || filepath.Ext(entry.Name()) != ".dshbak" {
continue
}
st, e := entry.Info()
if e != nil || !st.Mode().IsRegular() {
continue
}
id := entry.Name()[:len(entry.Name())-7]
kind, schema := "legacy", 0
_ = a.s.DB.QueryRow("SELECT kind,schema FROM backups WHERE id=? AND uid=? AND instance_id=?", id, i.UID, i.ID).Scan(&kind, &schema)
out = append(out, map[string]any{"id": id, "size": st.Size(), "created_at": st.ModTime(), "kind": kind, "schema": schema})
}
writeJSON(w, out)
return
}
if r.Method != "POST" {
fail(w, 405, "method not allowed")
return
}
entries, _ := os.ReadDir(dir)
if len(entries) >= 20 {
fail(w, 409, "backup retention limit reached (20); administrator cleanup required")
return
}
id := store.ID()
path := filepath.Join(dir, id+".dshbak")
kind, schema := "manual", 2
if strings.HasPrefix(r.Header.Get("Content-Type"), "multipart/form-data") {
kind, schema = "imported", 0
source, _, err := r.FormFile("file")
if err != nil {
fail(w, 400, "backup file required")
return
}
defer source.Close()
dest, err := os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0600)
if err != nil {
fail(w, 500, "backup storage unavailable")
return
}
n, err := io.Copy(dest, io.LimitReader(source, backup.MaxEncryptedBytes+1))
closeErr := dest.Close()
if err != nil || closeErr != nil || n > backup.MaxEncryptedBytes {
os.Remove(path)
fail(w, 400, "invalid or oversized backup upload")
return
}
key, _ := a.configKey()
_, cleanup, err := backup.Prepare(path, r.FormValue("password"), key)
clear(key)
if err != nil {
os.Remove(path)
fail(w, 400, "invalid backup or original backup password")
return
}
cleanup()
} else {
var in struct {
Password string `json:"password"`
}
if json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&in) != nil || len(in.Password) == 0 || len(in.Password) > 72 {
fail(w, 400, "login password required")
return
}
if !a.reauthenticate(w, i.UID, in.Password) {
return
}
if wasRunning {
if e = a.configStop(r.Context(), i.ContainerID); e != nil {
a.s.SetInstance(i.ID, "error")
fail(w, 502, "runtime stop failed")
return
}
}
if e = backup.CreateWithPassword(home, path, in.Password, backup.Metadata{DSHVersion: i.Version, CreatedAt: time.Now().UTC()}); e != nil {
if wasRunning {
a.configRestart(i)
}
fail(w, 400, "backup failed: unsafe or oversized configuration")
return
}
}
st, _ := os.Stat(path)
if e = a.indexBackup(i, id, kind, schema, st.Size()); e != nil {
os.Remove(path)
if wasRunning && kind == "manual" {
a.configRestart(i)
}
fail(w, 500, "backup index unavailable")
return
}
if wasRunning && kind == "manual" && a.configRestart(i) != nil {
fail(w, 502, "backup saved; runtime restart requires recovery")
return
}
a.s.AddAudit(i.UID, "backup.create", i.ID, id)
writeJSON(w, map[string]any{"id": id, "size": st.Size(), "created_at": st.ModTime(), "kind": kind, "schema": schema})
case "restore":
if r.Method != "POST" {
fail(w, 405, "method not allowed")
return
}
var in struct {
ID string `json:"backup_id"`
Password string `json:"password"`
}
if json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096)).Decode(&in) != nil || !backupIDPattern.MatchString(in.ID) {
fail(w, 400, "invalid backup ID")
return
}
key, _ := a.configKey()
restore, cleanup, e := backup.Prepare(filepath.Join(dir, in.ID+".dshbak"), in.Password, key)
clear(key)
if e != nil {
fail(w, 400, "invalid backup or original backup password")
return
}
defer cleanup()
staging, e := os.MkdirTemp("", "dsh-restore-quota-")
if e != nil {
fail(w, 500, "restore staging unavailable")
return
}
defer os.RemoveAll(staging)
if e = restore(staging); e != nil {
fail(w, 400, "invalid restore contents")
return
}
staged, e := os.OpenRoot(staging)
if e != nil {
fail(w, 500, "restore staging unavailable")
return
}
restoredBytes, e := quotaTreeBytes(staged, ".")
staged.Close()
if e != nil {
fail(w, 413, "restore size unavailable")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Minute)
defer cancel()
parent, e := upgradeRoot(a.root, i)
if e != nil {
fail(w, 500, "storage unavailable")
return
}
defer parent.Close()
f, e := parent.Open(".")
if e != nil {
fail(w, 500, "snapshot storage unavailable")
return
}
entries, e := f.ReadDir(-1)
f.Close()
count := 0
for _, entry := range entries {
if strings.HasPrefix(entry.Name(), "pre-restore-") || strings.HasPrefix(entry.Name(), "failed-restore-") {
count++
}
}
if e != nil || count >= 3 {
fail(w, 409, "restore snapshot limit reached; administrator cleanup required")
return
}
snapshot := "pre-restore-" + store.ID()
failed := "failed-restore-" + store.ID()
if e = a.recordSnapshot(i, snapshot, i.Version, i.ImageID); e != nil {
fail(w, 500, "snapshot metadata unavailable")
return
}
if e = a.recordSnapshot(i, failed, i.Version, i.ImageID); e != nil {
fail(w, 500, "snapshot metadata unavailable")
return
}
if _, e = a.quotaSnapshot(i.UID, parent, restoredBytes); e != nil {
fail(w, 413, "quota exceeded or snapshot size unavailable")
return
}
if wasRunning {
if e = a.configStop(ctx, i.ContainerID); e != nil {
a.s.SetInstance(i.ID, "error")
fail(w, 502, "runtime stop failed")
return
}
}
snapshotBytes, quotaErr := a.quotaSnapshot(i.UID, parent, restoredBytes)
if quotaErr != nil {
if wasRunning {
a.configRestart(i)
}
fail(w, 413, "quota exceeded or snapshot size unavailable")
return
}
if e = upgradeCopy(r.Context(), parent, "home", snapshot, snapshotBytes); e != nil {
cleanupErr := parent.RemoveAll(snapshot)
if wasRunning {
a.configRestart(i)
}
if cleanupErr != nil {
a.s.AddAudit(i.UID, "backup.snapshot.cleanup_failed", i.ID, snapshot)
fail(w, 500, "restore snapshot cleanup requires administrator recovery")
return
}
fail(w, 400, "cannot snapshot home for restore")
return
}
if _, e = a.s.DB.Exec("UPDATE instances SET status='error' WHERE id=?", i.ID); e != nil {
if wasRunning {
a.configRestart(i)
}
fail(w, 500, "cannot record restore state")
return
}
restoreErr := restore(home)
if restoreErr == nil {
var root *os.Root
root, restoreErr = os.OpenRoot(home)
if restoreErr == nil {
for _, p := range []string{"profiles", ".credentials.yaml", "cordis.patch.yml", "dsh-ssh.json", "skins", "pets", "agent-presets", ".agent-presets"} {
if restoreErr = configChownTree(root, p); restoreErr != nil {
break
}
}
root.Close()
}
}
if restoreErr == nil {
restoreErr = a.docker.Do(ctx, "POST", "/containers/"+url.PathEscape(i.ContainerID)+"/start", nil, nil)
}
if restoreErr == nil {
restoreErr = a.upgradeReady(ctx, i.ContainerID, i.ID, a.runtimeHost(r))
}
if restoreErr == nil && !wasRunning {
restoreErr = a.configStop(ctx, i.ContainerID)
}
if restoreErr != nil {
rollbackCtx, done := context.WithTimeout(context.Background(), 45*time.Second)
defer done()
if a.configStop(rollbackCtx, i.ContainerID) != nil {
fail(w, 500, "restore rollback requires runtime stop")
return
}
if parent.Rename("home", failed) != nil || parent.Rename(snapshot, "home") != nil || upgradeSync(parent) != nil {
a.s.SetInstance(i.ID, "error")
fail(w, 500, "restore rollback requires administrator recovery")
return
}
if wasRunning && a.configRestart(i) != nil {
fail(w, 500, "restore rollback restart requires recovery")
return
}
status := "exited"
if wasRunning {
status = "running"
}
if _, e = a.s.DB.Exec("UPDATE instances SET status=? WHERE id=?", status, i.ID); e != nil {
fail(w, 500, "restore state requires administrator recovery")
return
}
fail(w, 400, "restore rejected; previous home restored")
return
}
status := "exited"
if wasRunning {
status = "running"
}
if _, e = a.s.DB.Exec("UPDATE instances SET status=? WHERE id=?", status, i.ID); e != nil {
fail(w, 500, "restore state requires administrator recovery")
return
}
runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
a.s.AddAudit(i.UID, "backup.restore", i.ID, in.ID)
writeJSON(w, map[string]bool{"ok": true})
default:
fail(w, 404, "not found")
}
}
func configReadBody(w http.ResponseWriter, r *http.Request, controller *http.ResponseController, limit int64) bool {
if r.ContentLength > limit {
fail(w, 413, "configuration request too large")
return false
}
if err := controller.SetReadDeadline(time.Now().Add(30 * time.Second)); err != nil && !errors.Is(err, http.ErrNotSupported) {
fail(w, 500, "cannot set configuration read deadline")
return false
}
defer controller.SetReadDeadline(time.Time{})
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, limit))
if err != nil {
var oversized *http.MaxBytesError
switch {
case errors.As(err, &oversized):
fail(w, 413, "configuration request too large")
case errors.Is(err, os.ErrDeadlineExceeded):
fail(w, 408, "configuration upload timed out")
default:
fail(w, 400, "cannot read configuration request")
}
return false
}
r.Body = io.NopCloser(bytes.NewReader(body))
return true
}
func (a *API) configKey() ([]byte, error) {
path := filepath.Join(a.root, "backup.key")
f, e := os.OpenFile(path, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if e == nil {
defer f.Close()
st, err := f.Stat()
if err != nil || !st.Mode().IsRegular() {
return nil, errors.New("invalid key")
}
b, e := io.ReadAll(io.LimitReader(f, 33))
if e != nil || len(b) != 32 {
return nil, errors.New("invalid key")
}
return b, nil
}
return nil, e
}
func configCheckPath(root *os.Root, name string) error {
parts := strings.Split(name, "/")
for n := range parts {
st, e := root.Lstat(strings.Join(parts[:n+1], "/"))
if errors.Is(e, os.ErrNotExist) {
return nil
}
if e != nil {
return e
}
if st.Mode()&os.ModeSymlink != 0 || (n < len(parts)-1 && !st.IsDir()) {
return errors.New("unsafe configuration path")
}
}
return nil
}
func configProfileRoot(parent *os.Root, create bool) (*os.Root, error) {
root, err := parent.OpenRoot(".")
if err != nil {
return nil, err
}
for _, name := range []string{"home", "profiles", "web"} {
created := false
st, e := root.Lstat(name)
if errors.Is(e, os.ErrNotExist) && create && name != "home" {
e = root.Mkdir(name, 0750)
if e == nil {
created = true
st, e = root.Lstat(name)
}
}
if e != nil {
root.Close()
return nil, e
}
if !st.IsDir() {
root.Close()
return nil, errors.New("unsafe profile directory")
}
next, e := root.OpenRoot(name)
root.Close()
if e != nil {
return nil, e
}
actual, e := next.Stat(".")
if e != nil || !os.SameFile(st, actual) {
next.Close()
return nil, errors.New("profile directory changed")
}
if created {
if e = next.Chown(".", 1000, 1000); e != nil {
next.Close()
return nil, e
}
}
root = next
}
return root, nil
}
internal/api/config_body_test.go
-200
View File
@@ -1,200 +0,0 @@
package api
import (
"bytes"
"io"
"mime/multipart"
"net/http/httptest"
"os"
"strings"
"sync"
"testing"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
type configPausedBody struct {
io.Reader
started chan struct{}
release chan struct{}
once sync.Once
}
func (b *configPausedBody) Read(p []byte) (int, error) {
b.once.Do(func() { close(b.started) })
<-b.release
return b.Reader.Read(p)
}
func TestConfigSlowBodyRevalidatesWithoutLocks(t *testing.T) {
for _, action := range []string{"profile", "backups", "restore", "multipart"} {
for _, change := range []string{"lease", "deleted"} {
t.Run(action+"/"+change, func(t *testing.T) {
a, i, _ := configTestAPI(t, false)
a.workspaceReaders = make(map[string]int)
body := &configPausedBody{Reader: strings.NewReader(`{}`), started: make(chan struct{}), release: make(chan struct{})}
r := httptest.NewRequest("POST", "/", body)
if action == "profile" {
r.Method = "PUT"
}
target := action
if action == "multipart" {
target = "backups"
var buf bytes.Buffer
form := multipart.NewWriter(&buf)
part, _ := form.CreateFormFile("file", "test.dshbak")
part.Write(bytes.Repeat([]byte("x"), 2<<20))
form.Close()
body.Reader = &buf
r.Header.Set("Content-Type", form.FormDataContentType())
t.Setenv("TMPDIR", t.TempDir())
}
w := httptest.NewRecorder()
done := make(chan struct{})
go func() { defer close(done); a.config(w, r, i, target) }()
defer func() {
select {
case <-body.release:
default:
close(body.release)
}
<-done
}()
select {
case <-body.started:
case <-time.After(3 * time.Second):
t.Fatal("body read did not start")
}
if a.instanceLocks.held() || a.userLocks.held() {
t.Fatal("slow body holds API locks")
}
if change == "lease" {
a.readersMu.Lock()
a.workspaceReaders[i.ID]++
a.readersMu.Unlock()
}
want := 409
if change == "deleted" {
if _, err := a.s.DB.Exec("DELETE FROM instances WHERE id=?", i.ID); err != nil {
t.Fatal(err)
}
want = 404
}
close(body.release)
select {
case <-done:
case <-time.After(3 * time.Second):
t.Fatal("config did not finish")
}
if w.Code != want {
t.Fatalf("status %d: %s", w.Code, w.Body.String())
}
if action == "multipart" {
entries, err := os.ReadDir(os.Getenv("TMPDIR"))
if err != nil || len(entries) != 0 {
t.Fatalf("multipart temp files remain: %v %v", entries, err)
}
}
})
}
}
}
type configDeadlineWriter struct {
*httptest.ResponseRecorder
deadlines []time.Time
}
func (w *configDeadlineWriter) SetReadDeadline(deadline time.Time) error {
w.deadlines = append(w.deadlines, deadline)
return nil
}
type configErrorBody struct{}
func (configErrorBody) Read([]byte) (int, error) { return 0, os.ErrDeadlineExceeded }
func TestConfigBodyErrorsReturnWithoutLocks(t *testing.T) {
for _, tc := range []struct {
name string
body io.Reader
want int
}{
{"oversized", strings.NewReader(strings.Repeat("x", 4097)), 413},
{"timeout", configErrorBody{}, 408},
{"multipart", strings.NewReader("bad multipart"), 400},
} {
t.Run(tc.name, func(t *testing.T) {
a := &API{}
r := httptest.NewRequest("POST", "/", tc.body)
r.ContentLength = -1
if tc.name == "multipart" {
r.Header.Set("Content-Type", "multipart/form-data; boundary=test")
}
w := &configDeadlineWriter{ResponseRecorder: httptest.NewRecorder()}
inst := a.instanceLocks.Acquire("")
user := a.userLocks.Acquire("")
done := make(chan struct{})
go func() { defer close(done); a.config(w, r, store.Instance{}, "backups") }()
select {
case <-done:
user.Release()
inst.Release()
case <-time.After(3 * time.Second):
user.Release()
inst.Release()
t.Fatal("body error waited for locks")
}
if w.Code != tc.want {
t.Fatalf("status %d: %s", w.Code, w.Body.String())
}
if len(w.deadlines) != 2 || w.deadlines[0].IsZero() || !w.deadlines[1].IsZero() {
t.Fatalf("deadlines: %v", w.deadlines)
}
})
}
}
func TestConfigBodySlotsRejectWithoutReading(t *testing.T) {
for n := 0; n < cap(configBodySlots); n++ {
configBodySlots <- struct{}{}
}
defer func() {
for n := 0; n < cap(configBodySlots); n++ {
<-configBodySlots
}
}()
w := httptest.NewRecorder()
(&API{}).config(w, httptest.NewRequest("POST", "/", configErrorBody{}), store.Instance{}, "backups")
if w.Code != 503 {
t.Fatalf("status %d", w.Code)
}
}
func TestConfigSlowResponseReleasesLocks(t *testing.T) {
for _, tc := range []struct{ method, action string }{{"POST", "restore"}, {"GET", "profile"}, {"GET", "backups"}} {
t.Run(tc.method+"/"+tc.action, func(t *testing.T) {
a, i, _ := configTestAPI(t, false)
w := &pausedDownload{ResponseRecorder: httptest.NewRecorder(), started: make(chan struct{}), release: make(chan struct{})}
done := make(chan struct{})
go func() {
defer close(done)
a.config(w, httptest.NewRequest(tc.method, "/", strings.NewReader(`{}`)), i, tc.action)
}()
defer func() { close(w.release); <-done }()
select {
case <-w.started:
case <-time.After(3 * time.Second):
t.Fatal("response write did not start")
}
if a.instanceLocks.held() {
t.Fatal("slow response holds instance lock")
}
if a.userLocks.held() {
t.Fatal("slow response holds user lock")
}
})
}
}
internal/api/config_helpers.go
-85
View File
@@ -1,85 +0,0 @@
package api
import (
"context"
"errors"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"net/http"
"net/url"
"os"
"path/filepath"
"time"
)
func (a *API) indexBackup(i store.Instance, id, kind string, schema int, size int64) error {
_, err := a.s.DB.Exec("CREATE TABLE IF NOT EXISTS backups (id TEXT PRIMARY KEY, uid TEXT NOT NULL, instance_id TEXT NOT NULL, kind TEXT NOT NULL, schema INTEGER NOT NULL, size INTEGER NOT NULL, created_at INTEGER NOT NULL)")
if err != nil {
return err
}
_, err = a.s.DB.Exec("INSERT INTO backups(id,uid,instance_id,kind,schema,size,created_at) VALUES(?,?,?,?,?,?,?)", id, i.UID, i.ID, kind, schema, size, time.Now().Unix())
return err
}
func configChownTree(root *os.Root, name string) error {
st, err := root.Lstat(name)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
if st.Mode()&os.ModeSymlink != 0 {
return root.Lchown(name, 1000, 1000)
}
if err = root.Chown(name, 1000, 1000); err != nil {
return err
}
if !st.IsDir() {
return nil
}
f, err := root.Open(name)
if err != nil {
return err
}
entries, err := f.ReadDir(-1)
f.Close()
if err != nil {
return err
}
for _, entry := range entries {
if err = configChownTree(root, filepath.Join(name, entry.Name())); err != nil {
return err
}
}
return nil
}
func (a *API) configStop(ctx context.Context, id string) error {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
if err := a.docker.Do(ctx, "POST", "/containers/"+url.PathEscape(id)+"/stop?t=10", nil, nil); err != nil {
var apiErr *docker.APIError
if !errors.As(err, &apiErr) || apiErr.StatusCode != http.StatusNotModified {
return err
}
}
c, err := a.docker.Inspect(ctx, id)
if err != nil {
return err
}
if c.State.Running || c.State.Paused || c.State.Restarting {
return errors.New("runtime did not stop")
}
return nil
}
func (a *API) configRestart(i store.Instance) error {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
err := a.docker.Do(ctx, "POST", "/containers/"+url.PathEscape(i.ContainerID)+"/start", nil, nil)
if err != nil {
a.s.SetInstance(i.ID, "error")
}
return err
}
internal/api/config_test.go
-446
View File
@@ -1,446 +0,0 @@
package api
import (
"bytes"
"context"
"encoding/json"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
"mime/multipart"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"syscall"
"testing"
"time"
)
func TestConfigStopAlreadyStopped(t *testing.T) {
for _, state := range []string{`{}`, `{"Running":true}`, `{"Paused":true}`, `{"Restarting":true}`} {
t.Run(state, func(t *testing.T) {
inspected := false
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" {
w.WriteHeader(http.StatusNotModified)
return
}
inspected = true
fmt.Fprintf(w, `{"State":%s}`, state)
}))
defer srv.Close()
a := &API{docker: docker.New(srv.URL)}
err := a.configStop(context.Background(), "test")
if !inspected || (err == nil) != (state == `{}`) {
t.Fatalf("inspected=%v error=%v", inspected, err)
}
})
}
}
func TestConfigEmptyProfileRoundTrip(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
w := configCall(a, i, "GET", "profile", "")
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
w = configCall(a, i, "PUT", "profile", w.Body.String())
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
content, err := os.ReadFile(filepath.Join(home, "profiles/web/cordis.patch.yml"))
if err != nil || string(content) != "[]\n" {
t.Fatalf("content=%q error=%v", content, err)
}
}
func TestConfigPasswordBackupImport(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
hash, _ := bcrypt.GenerateFromPassword([]byte("original-password"), bcrypt.MinCost)
if _, err := a.s.DB.Exec("UPDATE users SET pass_hash=? WHERE id=?", string(hash), i.UID); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(home, ".credentials.yaml"), "secret: original")
for _, body := range []string{`{}`, `{"password":"wrong"}`} {
if w := configCall(a, i, "POST", "backups", body); w.Code < 400 {
t.Fatal("accepted wrong password")
}
}
w := configCall(a, i, "POST", "backups", `{"password":"original-password"}`)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
var out struct {
ID string `json:"id"`
}
json.Unmarshal(w.Body.Bytes(), &out)
if _, err := os.Stat(filepath.Join(a.root, "backup.key")); !os.IsNotExist(err) {
t.Fatal("new backup persisted key")
}
encrypted, err := os.ReadFile(filepath.Join(a.root, "backups", i.UID, i.ID, out.ID+".dshbak"))
if err != nil {
t.Fatal(err)
}
if bytes.Contains(encrypted, []byte("secret: original")) {
t.Fatal("plaintext archive")
}
hash, _ = bcrypt.GenerateFromPassword([]byte("changed-password"), bcrypt.MinCost)
a.s.DB.Exec("UPDATE users SET pass_hash=? WHERE id=?", string(hash), i.UID)
configWrite(t, filepath.Join(home, ".credentials.yaml"), "secret: changed")
w = configCall(a, i, "POST", "restore", `{"backup_id":"`+out.ID+`","password":"wrong"}`)
if w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
entries, _ := os.ReadDir(filepath.Dir(home))
for _, entry := range entries {
if strings.HasPrefix(entry.Name(), "pre-restore-") {
t.Fatal("bad password created snapshot")
}
}
var body bytes.Buffer
form := multipart.NewWriter(&body)
part, _ := form.CreateFormFile("file", "external.dshbak")
part.Write(encrypted)
form.WriteField("password", "original-password")
form.Close()
req := httptest.NewRequest("POST", "/api/instances/one/backups", &body)
req.Header.Set("Content-Type", form.FormDataContentType())
w = httptest.NewRecorder()
a.config(w, req, i, "backups")
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
w = configCall(a, i, "POST", "restore", `{"backup_id":"`+out.ID+`","password":"original-password"}`)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
restored, _ := os.ReadFile(filepath.Join(home, ".credentials.yaml"))
if string(restored) != "secret: original" {
t.Fatal(string(restored))
}
}
func configTestAPI(t *testing.T, running bool, failures ...string) (*API, store.Instance, string) {
t.Helper()
var mu sync.Mutex
httpFailure := false
backupRestart := running
failure := ""
if len(failures) != 0 {
failure = failures[0]
}
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
defer mu.Unlock()
if httpFailure {
httpFailure = false
running = false
w.WriteHeader(503)
return
}
if r.URL.Query().Get("token") == "test-token" {
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: "ready"})
w.WriteHeader(303)
return
}
w.WriteHeader(200)
}))
t.Cleanup(upstream.Close)
transport := runtimeTransport
runtimeTransport = transport.Clone()
runtimeTransport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, network, strings.TrimPrefix(upstream.URL, "http://"))
}
t.Cleanup(func() { runtimeTransport.CloseIdleConnections(); runtimeTransport = transport })
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
defer mu.Unlock()
switch {
case strings.HasSuffix(r.URL.Path, "/stop"):
running = false
w.WriteHeader(204)
case strings.HasSuffix(r.URL.Path, "/start"):
if backupRestart {
backupRestart = false
running = true
w.WriteHeader(204)
return
}
running = failure != "reconcile"
httpFailure = failure == "readiness"
failure = ""
w.WriteHeader(204)
case strings.Contains(r.URL.Path, "/exec"), strings.HasSuffix(r.URL.Path, "/restart"):
t.Error("restore must use entrypoint prestart, not exec/restart")
w.WriteHeader(500)
case strings.HasSuffix(r.URL.Path, "/logs"):
fmt.Fprint(w, "http://runtime/?token=test-token")
default:
writeJSON(w, map[string]any{"State": map[string]any{"Running": running}, "NetworkSettings": map[string]any{"Networks": map[string]any{"dsh-inst-one": map[string]string{"IPAddress": "127.0.0.1"}}}})
}
})
hash, _ := bcrypt.GenerateFromPassword([]byte("original-password"), bcrypt.MinCost)
a.s.DB.Exec("UPDATE users SET pass_hash=?,quota=10737418240 WHERE id='owner'", string(hash))
i := store.Instance{ID: "one", UID: "owner", Name: "one", ContainerID: "old", Status: "exited", Created: time.Now()}
if e := a.s.AddInstance(i); e != nil {
t.Fatal(e)
}
home := filepath.Join(a.root, "workspaces", "owner", "instances", "one", "home")
if e := os.MkdirAll(home, 0750); e != nil {
t.Fatal(e)
}
return a, i, home
}
func configCall(a *API, i store.Instance, method, action, body string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
r := httptest.NewRequest(method, "/api/instances/one/"+action, strings.NewReader(body))
a.config(w, r, i, strings.Split(action, "?")[0])
return w
}
func configWrite(t *testing.T, path, content string) {
t.Helper()
if e := os.WriteFile(path, []byte(content), 0600); e != nil {
t.Fatal(e)
}
}
func TestConfigProfile(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
for _, tc := range []struct {
method, body string
code int
}{
{"GET", "", 200}, {"PUT", `{"content":"foo: ["}`, 400}, {"PUT", `{"content":"foo: bar\n"}`, 200}, {"GET", "", 200},
} {
w := configCall(a, i, tc.method, "profile", tc.body)
if w.Code != tc.code {
t.Fatalf("%s: %d %s", tc.method, w.Code, w.Body.String())
}
}
for _, path := range []string{"profiles", "profiles/web", "profiles/web/cordis.patch.yml"} {
st, e := os.Stat(filepath.Join(home, path))
if e != nil {
t.Fatal(e)
}
stat := st.Sys().(*syscall.Stat_t)
if stat.Uid != 1000 || stat.Gid != 1000 {
t.Fatalf("ownership of %s: %+v", path, stat)
}
}
w := configCall(a, i, "GET", "profile", "")
if !strings.Contains(w.Body.String(), "foo: bar") {
t.Fatal(w.Body.String())
}
}
func TestConfigProfileSymlinks(t *testing.T) {
for _, level := range []string{"profiles", "web", "leaf", "home"} {
t.Run(level, func(t *testing.T) {
a, i, home := configTestAPI(t, false)
secret := "credential-secret-never-return"
configWrite(t, filepath.Join(home, ".credentials.yaml"), secret)
if e := os.MkdirAll(filepath.Join(home, "profiles", "web"), 0750); e != nil {
t.Fatal(e)
}
var link, target string
switch level {
case "profiles":
link = filepath.Join(home, "profiles")
target = "private"
if e := os.MkdirAll(filepath.Join(home, "private", "web"), 0750); e != nil {
t.Fatal(e)
}
configWrite(t, filepath.Join(home, "private", "web", "cordis.patch.yml"), secret)
case "web":
link = filepath.Join(home, "profiles", "web")
target = "../private"
if e := os.Mkdir(filepath.Join(home, "private"), 0750); e != nil {
t.Fatal(e)
}
configWrite(t, filepath.Join(home, "private", "cordis.patch.yml"), secret)
case "leaf":
link = filepath.Join(home, "profiles", "web", "cordis.patch.yml")
target = "../../.credentials.yaml"
case "home":
link = home
target = home + "-other"
if e := os.Rename(home, target); e != nil {
t.Fatal(e)
}
}
if level != "home" {
if e := os.RemoveAll(link); e != nil {
t.Fatal(e)
}
}
if e := os.Symlink(target, link); e != nil {
t.Fatal(e)
}
for _, method := range []string{"GET", "PUT"} {
w := configCall(a, i, method, "profile", `{"content":"foo: bar"}`)
if w.Code != 400 || strings.Contains(w.Body.String(), secret) {
t.Fatalf("%s: %d %s", method, w.Code, w.Body.String())
}
}
})
}
}
func TestConfigRunningRejected(t *testing.T) {
a, i, _ := configTestAPI(t, true)
for _, tc := range []struct{ method, action string }{{"PUT", "profile"}} {
w := configCall(a, i, tc.method, tc.action, `{}`)
if w.Code != 409 {
t.Fatalf("%s: %d", tc.action, w.Code)
}
}
}
func TestConfigBackupRestore(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
const secret = "private-credential-token"
configWrite(t, filepath.Join(home, ".credentials.yaml"), secret)
if w := configCall(a, i, "PUT", "profile", `{"content":"foo: old"}`); w.Code != 200 {
t.Fatal(w.Body.String())
}
w := configCall(a, i, "POST", "backups", `{"password":"original-password"}`)
var result struct {
ID string `json:"id"`
}
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &result) != nil || !backupIDPattern.MatchString(result.ID) {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
w = configCall(a, i, "GET", "backups?download="+result.ID, "")
if w.Code != 200 || !bytes.HasPrefix(w.Body.Bytes(), []byte("DSHPASS")) || bytes.Contains(w.Body.Bytes(), []byte(secret)) {
t.Fatalf("unsafe download: %d", w.Code)
}
if e := os.RemoveAll(filepath.Join(home, "profiles")); e != nil {
t.Fatal(e)
}
configWrite(t, filepath.Join(home, ".credentials.yaml"), "changed")
body := fmt.Sprintf(`{"backup_id":%q,"password":"original-password"}`, result.ID)
for n := 0; n < 3; n++ {
w = configCall(a, i, "POST", "restore", body)
if w.Code != 200 {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
}
w = configCall(a, i, "POST", "restore", body)
if w.Code != 409 {
t.Fatalf("snapshot retention: %d", w.Code)
}
got, e := os.ReadFile(filepath.Join(home, ".credentials.yaml"))
if e != nil || string(got) != secret {
t.Fatalf("restore: %q %v", got, e)
}
for _, p := range []string{"profiles", "profiles/web", "profiles/web/cordis.patch.yml"} {
st, e := os.Stat(filepath.Join(home, p))
if e != nil {
t.Fatal(e)
}
if st.Sys().(*syscall.Stat_t).Uid != 1000 {
t.Fatalf("not owned by runtime: %s", p)
}
}
state, e := a.s.GetInstance(i.ID)
if e != nil || state.Status != "exited" {
t.Fatalf("state: %+v %v", state, e)
}
id := store.ID()
if e := os.Symlink(filepath.Join(a.root, "backup.key"), filepath.Join(a.root, "backups", i.UID, i.ID, id+".dshbak")); e != nil {
t.Fatal(e)
}
w = configCall(a, i, "GET", "backups?download="+id, "")
if w.Code == 200 {
t.Fatal("symlink download exposed key")
}
}
func TestConfigRestoreInvalidArchive(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
configWrite(t, filepath.Join(home, "cordis.patch.yml"), "original")
id := store.ID()
dir := filepath.Join(a.root, "backups", i.UID, i.ID)
if e := os.MkdirAll(dir, 0700); e != nil {
t.Fatal(e)
}
configWrite(t, filepath.Join(dir, id+".dshbak"), "corrupt")
w := configCall(a, i, "POST", "restore", fmt.Sprintf(`{"backup_id":%q}`, id))
if w.Code != 400 || !strings.Contains(w.Body.String(), "invalid backup") {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
got, e := os.ReadFile(filepath.Join(home, "cordis.patch.yml"))
if e != nil || string(got) != "original" {
t.Fatalf("rollback: %q %v", got, e)
}
a.s.SetInstance(i.ID, "error")
w = configCall(a, i, "POST", "restore", fmt.Sprintf(`{"backup_id":%q}`, id))
if w.Code != 409 {
t.Fatalf("recovery guard: %d", w.Code)
}
}
func TestConfigRestoreOrchestration(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
for _, running := range []bool{false, true} {
for _, failure := range []string{"", "reconcile", "readiness"} {
t.Run(fmt.Sprintf("running=%v/%s", running, failure), func(t *testing.T) {
a, i, home := configTestAPI(t, running, failure)
configWrite(t, filepath.Join(home, ".credentials.yaml"), "archive")
if err := os.MkdirAll(filepath.Join(home, "pets", "nested"), 0750); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(home, "pets", "nested", "pet.json"), "{}")
w := configCall(a, i, "POST", "backups", `{"password":"original-password"}`)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
var out struct{ ID string }
json.Unmarshal(w.Body.Bytes(), &out)
var kind string
var schema int
if err := a.s.DB.QueryRow("SELECT kind,schema FROM backups WHERE id=?", out.ID).Scan(&kind, &schema); err != nil || kind != "manual" || schema != 2 {
t.Fatalf("index: %s %d %v", kind, schema, err)
}
configWrite(t, filepath.Join(home, ".credentials.yaml"), "before-restore")
w = configCall(a, i, "POST", "restore", fmt.Sprintf(`{"backup_id":%q,"password":"original-password"}`, out.ID))
wantCode, wantContent := 200, "archive"
if failure != "" {
wantCode, wantContent = 400, "before-restore"
}
if w.Code != wantCode {
t.Fatal(w.Code, w.Body.String())
}
b, _ := os.ReadFile(filepath.Join(home, ".credentials.yaml"))
if string(b) != wantContent {
t.Fatalf("home: %s", b)
}
c, err := a.docker.Inspect(context.Background(), i.ContainerID)
if err != nil || c.State.Running != running {
t.Fatalf("runtime state: %+v %v", c, err)
}
st, err := os.Stat(filepath.Join(home, "pets", "nested", "pet.json"))
if err != nil || st.Sys().(*syscall.Stat_t).Uid != 1000 {
t.Fatalf("resource ownership: %v", err)
}
})
}
}
}
internal/api/download_filename_test.go
-101
View File
@@ -1,101 +0,0 @@
package api
import (
"archive/zip"
"bytes"
"io"
"mime"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"testing"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func TestWorkspaceDownloadFilename(t *testing.T) {
a, sid := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) { t.Error("unexpected Docker call") })
if err := a.s.AddInstance(store.Instance{ID: "one", UID: "owner"}); err != nil {
t.Fatal(err)
}
home := filepath.Join(a.root, "workspaces/owner/instances/one/home")
checkFilename := func(t *testing.T, w *httptest.ResponseRecorder, want string) {
t.Helper()
header := w.Header().Get("Content-Disposition")
mediaType, params, err := mime.ParseMediaType(header)
if err != nil {
t.Fatalf("parse Content-Disposition %q: %v", header, err)
}
if mediaType != "attachment" || params["filename"] != want {
t.Fatalf("Content-Disposition = %q, want attachment with filename %q", header, want)
}
}
const content = "0123456789\nunchanged 下载 content\x00\xff"
for _, name := range []string{"report.txt", "中文文件.txt", "report with spaces.txt", `report";100%.txt`, "README"} {
t.Run(name, func(t *testing.T) {
p := "parent/nested/" + name
file := filepath.Join(home, filepath.FromSlash(p))
if err := os.MkdirAll(filepath.Dir(file), 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(file, []byte(content), 0600); err != nil {
t.Fatal(err)
}
for _, partial := range []bool{false, true} {
r := runtimeRequest(http.MethodGet, "/api/ws/download?instance=one&path="+url.QueryEscape(p), "", sid)
wantStatus, wantBody := http.StatusOK, content
if partial {
r.Header.Set("Range", "bytes=2-6")
wantStatus, wantBody = http.StatusPartialContent, content[2:7]
}
w := httptest.NewRecorder()
a.workspace(w, r)
if w.Code != wantStatus {
t.Fatalf("partial=%v: status = %d, want %d: %s", partial, w.Code, wantStatus, w.Body.String())
}
checkFilename(t, w, name)
if w.Body.String() != wantBody {
t.Errorf("partial=%v: body = %q, want %q", partial, w.Body.String(), wantBody)
}
}
})
}
t.Run("directory zip", func(t *testing.T) {
const name = `资料 ";100%`
p := "parent/" + name
dir := filepath.Join(home, filepath.FromSlash(p))
if err := os.MkdirAll(dir, 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "entry.txt"), []byte(content), 0600); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest(http.MethodGet, "/api/ws/download?instance=one&path="+url.QueryEscape(p), "", sid))
if w.Code != http.StatusOK {
t.Fatalf("status = %d: %s", w.Code, w.Body.String())
}
checkFilename(t, w, name+".zip")
archive, err := zip.NewReader(bytes.NewReader(w.Body.Bytes()), int64(w.Body.Len()))
if err != nil {
t.Fatal(err)
}
if len(archive.File) != 1 || archive.File[0].Name != "entry.txt" {
t.Fatalf("unexpected archive entries: %v", archive.File)
}
r, err := archive.File[0].Open()
if err != nil {
t.Fatal(err)
}
defer r.Close()
got, err := io.ReadAll(r)
if err != nil {
t.Fatal(err)
}
if string(got) != content {
t.Errorf("archive body = %q, want %q", got, content)
}
})
}
internal/api/files.go
-988
View File
@@ -1,988 +0,0 @@
package api
import (
"archive/zip"
"context"
"crypto/sha256"
"encoding/json"
"errors"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"mime"
"net/http"
"os"
"path"
"sort"
"strings"
"syscall"
"time"
"unicode"
"unicode/utf8"
"golang.org/x/sys/unix"
)
const maxUpload = 32 << 20
var uploadSlots = make(chan struct{}, 4)
const (
maxZipEntries = 10000
maxZipDepth = 64
maxZipBytes int64 = 10 << 30
zipTimeout = 2 * time.Minute
)
var zipSlots = make(chan struct{}, 4)
var errZipLimit = errors.New("archive limit exceeded")
func renameNoReplace(root *os.Root, old, next string) error {
anchor, err := root.Open(".")
if err != nil {
return err
}
defer anchor.Close()
openParent := func(p string) (int, error) {
return unix.Openat2(int(anchor.Fd()), path.Dir(p), &unix.OpenHow{
Flags: unix.O_RDONLY | unix.O_DIRECTORY | unix.O_CLOEXEC,
Resolve: unix.RESOLVE_BENEATH | unix.RESOLVE_NO_SYMLINKS,
})
}
src, err := openParent(old)
if err != nil {
return err
}
defer unix.Close(src)
dst, err := openParent(next)
if err != nil {
return err
}
defer unix.Close(dst)
var st unix.Stat_t
if err = unix.Fstatat(src, path.Base(old), &st, unix.AT_SYMLINK_NOFOLLOW); err != nil {
return err
}
if st.Mode&unix.S_IFMT == unix.S_IFLNK {
return errors.New("unsafe path")
}
return unix.Renameat2(src, path.Base(old), dst, path.Base(next), unix.RENAME_NOREPLACE)
}
func safeRel(p string) (string, error) {
p = strings.TrimSpace(strings.ReplaceAll(p, `\`, `/`))
if p == "" {
return ".", nil
}
if strings.HasPrefix(p, "/") {
return "", errors.New("invalid path")
}
c := path.Clean(p)
if c == ".." || strings.HasPrefix(c, "../") {
return "", errors.New("invalid path")
}
return c, nil
}
func (a *API) openWorkspace(u store.User, instanceID string) (*os.Root, error) {
if instanceID == "" {
return nil, errors.New("instance required")
}
base, err := os.OpenRoot(a.root)
if err != nil {
return nil, err
}
defer base.Close()
rel := path.Join("workspaces", u.ID, "instances", instanceID, "home")
if err := rootSafe(base, rel, false); err != nil {
return nil, err
}
return base.OpenRoot(rel)
}
func rootSafe(root *os.Root, rel string, finalMayMissing bool) error {
if rel == "." {
return nil
}
parts := strings.Split(rel, "/")
for i := range parts {
q := strings.Join(parts[:i+1], "/")
st, err := root.Lstat(q)
if err != nil {
if finalMayMissing && i == len(parts)-1 && errors.Is(err, os.ErrNotExist) {
return nil
}
return err
}
if st.Mode()&os.ModeSymlink != 0 || (i < len(parts)-1 && !st.IsDir()) {
return errors.New("unsafe path")
}
}
return nil
}
func (a *API) workspace(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/ws/download" && r.URL.Path != "/api/ws/image" {
var flush func()
w, flush = deferJSONResponse(w)
defer flush()
}
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
instanceID := strings.TrimSpace(r.URL.Query().Get("instance"))
if instanceID == "" && r.URL.Path != "/api/ws/usage" {
fail(w, http.StatusGone, "shared files removed; instance required")
return
}
if instanceID != "" {
// Pin this instance's home without holding the instance lifecycle lock
// across client reads/writes. Lifecycle mutations reject active leases.
inst := a.instanceLocks.Acquire(instanceID)
i, e := a.s.GetInstance(instanceID)
if e != nil || i.UID != u.ID {
inst.Release()
fail(w, 404, "instance not found")
return
}
if r.Method != http.MethodGet && (i.Status == "error" || a.upgradeRecoveryRequired(i)) {
inst.Release()
fail(w, 409, "instance requires recovery")
return
}
a.readersMu.Lock()
if a.workspaceReaders == nil {
a.workspaceReaders = make(map[string]int)
}
a.workspaceReaders[instanceID]++
a.readersMu.Unlock()
inst.Release()
defer func() {
a.readersMu.Lock()
if a.workspaceReaders != nil {
a.workspaceReaders[instanceID]--
if a.workspaceReaders[instanceID] == 0 {
delete(a.workspaceReaders, instanceID)
}
}
a.readersMu.Unlock()
}()
}
if r.URL.Path == "/api/ws/usage" {
a.wsUsage(w, r, u)
return
}
user := a.userLocks.Acquire(u.ID)
root, err := a.openWorkspace(u, instanceID)
user.Release()
if err != nil {
fail(w, 500, "storage unavailable")
return
}
defer root.Close()
switch strings.TrimPrefix(r.URL.Path, "/api/ws/") {
case "tree":
a.wsTree(w, r, root, u)
case "mkdir":
a.wsMkdir(w, r, root, u)
case "rename":
a.wsRename(w, r, root, u)
case "delete":
a.wsDelete(w, r, root, u)
case "upload":
a.wsUpload(w, r, root, u)
case "download":
a.wsDownload(w, r, root)
case "image":
a.wsImage(w, r, root)
case "text":
a.wsText(w, r, root, u)
default:
fail(w, 404, "not found")
}
}
const maxTextBytes = 1 << 20
func textStatus(content string) int {
if len(content) > maxTextBytes {
return 413
}
if !utf8.ValidString(content) {
return 415
}
for _, c := range content {
if unicode.IsControl(c) && c != 9 && c != 10 && c != 13 {
return 415
}
}
return 0
}
func textVersion(content string, st os.FileInfo) string {
s := st.Sys().(*syscall.Stat_t)
return fmt.Sprintf("%x", sha256.Sum256([]byte(fmt.Sprintf("%d:%d:%d:%d:%d:%d:%s", s.Dev, s.Ino, st.ModTime().UnixNano(), s.Mode, s.Uid, s.Gid, content))))
}
func readText(parent *os.File, name string) (string, string, os.FileInfo, int) {
f, err := openDownload(parent, name)
if err != nil {
return "", "", nil, 404
}
defer f.Close()
st, err := f.Stat()
if err != nil || !st.Mode().IsRegular() {
return "", "", nil, 415
}
if st.Size() > maxTextBytes {
return "", "", nil, 413
}
b, err := io.ReadAll(io.LimitReader(f, maxTextBytes+1))
if err != nil {
return "", "", nil, 500
}
if code := textStatus(string(b)); code != 0 {
return "", "", nil, code
}
after, err := f.Stat()
if err != nil || st.Size() != after.Size() || !st.ModTime().Equal(after.ModTime()) || st.Sys().(*syscall.Stat_t).Ctim != after.Sys().(*syscall.Stat_t).Ctim {
return "", "", nil, 409
}
return string(b), textVersion(string(b), after), after, 0
}
func (a *API) wsText(w http.ResponseWriter, r *http.Request, root *os.Root, u store.User) {
if r.Method != http.MethodGet && r.Method != http.MethodPut {
fail(w, 405, "method not allowed")
return
}
p := r.URL.Query().Get("path")
if p == "" || path.IsAbs(p) || p == "." || p == ".." || strings.HasPrefix(p, "../") || path.Clean(p) != p || strings.ContainsRune(p, 0) {
fail(w, 400, "invalid path")
return
}
var x struct {
Content *string `json:"content"`
Version string `json:"version"`
}
if r.Method == http.MethodPut {
b, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 6*maxTextBytes+4096))
if err != nil {
fail(w, 413, "text request too large")
return
}
if !utf8.Valid(b) {
fail(w, 415, "invalid UTF-8")
return
}
if json.Unmarshal(b, &x) != nil || x.Content == nil || x.Version == "" {
fail(w, 400, "content and version required")
return
}
if code := textStatus(*x.Content); code != 0 {
fail(w, code, "unsupported text or text too large")
return
}
}
user := a.userLocks.Acquire(u.ID)
defer user.Release()
anchor, err := root.Open(".")
if err != nil {
fail(w, 500, "storage unavailable")
return
}
defer anchor.Close()
fd, err := unix.Openat2(int(anchor.Fd()), path.Dir(p), &unix.OpenHow{
Flags: unix.O_RDONLY | unix.O_DIRECTORY | unix.O_CLOEXEC,
Resolve: unix.RESOLVE_BENEATH | unix.RESOLVE_NO_SYMLINKS | unix.RESOLVE_NO_XDEV,
})
if err != nil {
code := 404
if r.Method == http.MethodPut {
code = 409
}
fail(w, code, "parent unavailable")
return
}
parent := os.NewFile(uintptr(fd), path.Dir(p))
defer parent.Close()
content, version, st, code := readText(parent, path.Base(p))
if code != 0 {
if r.Method == http.MethodPut && code == 404 {
code = 409
}
fail(w, code, "text file unavailable")
return
}
if r.Method == http.MethodGet {
writeJSON(w, map[string]string{"content": content, "version": version})
return
}
if version != x.Version {
fail(w, 409, "file changed; reload before saving")
return
}
if err := a.s.DB.QueryRow("SELECT quota FROM users WHERE id=?", u.ID).Scan(&u.Quota); err != nil {
fail(w, 413, "quota unavailable")
return
}
used, err := a.userUsage(u)
if err != nil || int64(len(*x.Content)) > u.Quota-used {
fail(w, 413, "quota exceeded or usage unavailable")
return
}
tmp := ".text-" + store.ID()
tfd, err := unix.Openat2(fd, tmp, &unix.OpenHow{Flags: unix.O_WRONLY | unix.O_CREAT | unix.O_EXCL | unix.O_CLOEXEC, Mode: 0600, Resolve: unix.RESOLVE_BENEATH | unix.RESOLVE_NO_SYMLINKS})
if err != nil {
fail(w, 500, "cannot prepare text save")
return
}
f := os.NewFile(uintptr(tfd), tmp)
defer f.Close()
defer unix.Unlinkat(fd, tmp, 0)
_, err = io.WriteString(f, *x.Content)
s := st.Sys().(*syscall.Stat_t)
if err == nil {
err = f.Chown(int(s.Uid), int(s.Gid))
}
if err == nil {
err = unix.Fchmod(tfd, s.Mode&07777)
}
if err == nil {
err = f.Sync()
}
next, statErr := f.Stat()
if err != nil || statErr != nil {
fail(w, 500, "cannot prepare text save")
return
}
nextVersion := textVersion(*x.Content, next)
_, latest, _, code := readText(parent, path.Base(p))
if code != 0 || latest != version {
fail(w, 409, "file changed; reload before saving")
return
}
// External writers do not take the user lock; this is not a filesystem CAS.
if err := unix.Renameat(fd, tmp, fd, path.Base(p)); err != nil {
fail(w, 500, "text save failed")
return
}
a.s.AddAudit(u.ID, "workspace.text", p, "")
writeJSON(w, map[string]string{"version": nextVersion})
}
func entry(root *os.Root, rel string) (os.FileInfo, error) {
if err := rootSafe(root, rel, false); err != nil {
return nil, err
}
return root.Lstat(rel)
}
func (a *API) wsUsage(w http.ResponseWriter, r *http.Request, u store.User) {
if r.Method != "GET" {
fail(w, 405, "method not allowed")
return
}
user := a.userLocks.Acquire(u.ID)
n, e := a.userUsage(u)
user.Release()
if e != nil {
fail(w, 500, "usage unavailable")
return
}
writeJSON(w, map[string]int64{"used_bytes": n, "quota_bytes": u.Quota})
}
func (a *API) wsTree(w http.ResponseWriter, r *http.Request, root *os.Root, u store.User) {
user := a.userLocks.Acquire(u.ID)
locked := user
defer func() {
if locked != nil {
locked.Release()
}
}()
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
rel, err := safeRel(r.URL.Query().Get("path"))
if err != nil {
fail(w, 400, "invalid path")
return
}
st, err := entry(root, rel)
if err != nil || !st.IsDir() {
fail(w, 404, "not found")
return
}
es, err := readRootDir(root, rel)
if err != nil {
if errors.Is(err, errDirectoryLimit) {
fail(w, 413, "directory entry limit exceeded")
return
}
fail(w, 404, "not found")
return
}
out := make([]map[string]any, 0, len(es))
for _, de := range es {
p := de.Name()
if rel != "." {
p = rel + "/" + p
}
st, err := entry(root, p)
if err != nil || (!st.IsDir() && !st.Mode().IsRegular()) {
continue
}
sz := int64(0)
if !st.IsDir() {
sz = st.Size()
}
out = append(out, map[string]any{"name": de.Name(), "path": p, "is_dir": st.IsDir(), "size": sz, "mod_time": st.ModTime()})
}
sort.Slice(out, func(i, j int) bool { return out[i]["name"].(string) < out[j]["name"].(string) })
locked.Release()
locked = nil
writeJSON(w, out)
}
func (a *API) wsMkdir(w http.ResponseWriter, r *http.Request, root *os.Root, u store.User) {
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
var x struct {
Path string `json:"path"`
}
if json.NewDecoder(r.Body).Decode(&x) != nil {
fail(w, 400, "invalid json")
return
}
p, e := safeRel(x.Path)
user := a.userLocks.Acquire(u.ID)
locked := user
defer func() {
if locked != nil {
locked.Release()
}
}()
if e != nil || p == "." || rootSafe(root, path.Dir(p), false) != nil {
fail(w, 400, "cannot create directory")
return
}
if e = root.Mkdir(p, 0750); e != nil {
fail(w, 400, "cannot create directory")
return
}
if f, e := root.OpenFile(p, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0); e == nil {
_ = f.Chown(1000, 1000)
f.Close()
}
locked.Release()
locked = nil
a.s.AddAudit(u.ID, "workspace.mkdir", p, "")
writeJSON(w, map[string]bool{"ok": true})
}
func (a *API) wsRename(w http.ResponseWriter, r *http.Request, root *os.Root, u store.User) {
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
var x struct {
Path string `json:"path"`
NewPath string `json:"new_path"`
}
if json.NewDecoder(r.Body).Decode(&x) != nil {
fail(w, 400, "invalid json")
return
}
p, e := safeRel(x.Path)
q, e2 := safeRel(x.NewPath)
user := a.userLocks.Acquire(u.ID)
locked := user
defer func() {
if locked != nil {
locked.Release()
}
}()
if e != nil || e2 != nil || p == "." || q == "." || rootSafe(root, p, false) != nil || rootSafe(root, path.Dir(q), false) != nil {
fail(w, 400, "rename failed")
return
}
st, e := root.Lstat(p)
if e != nil || st.Mode()&os.ModeSymlink != 0 {
fail(w, 400, "rename failed")
return
}
if _, e = root.Lstat(q); e == nil {
fail(w, 400, "rename failed")
return
}
if e = renameNoReplace(root, p, q); e != nil {
fail(w, 400, "rename failed")
return
}
locked.Release()
locked = nil
a.s.AddAudit(u.ID, "workspace.rename", p, q)
writeJSON(w, map[string]bool{"ok": true})
}
func (a *API) wsDelete(w http.ResponseWriter, r *http.Request, root *os.Root, u store.User) {
user := a.userLocks.Acquire(u.ID)
locked := user
defer func() {
if locked != nil {
locked.Release()
}
}()
if r.Method != http.MethodDelete {
fail(w, 405, "method not allowed")
return
}
p, e := safeRel(r.URL.Query().Get("path"))
if e != nil || p == "." {
fail(w, 400, "delete failed")
return
}
st, e := entry(root, p)
if e != nil || st.Mode()&os.ModeSymlink != 0 {
fail(w, 400, "delete failed")
return
}
if st.IsDir() {
es, e := readRootDir(root, p)
if e != nil || len(es) != 0 {
fail(w, 400, "delete failed")
return
}
}
if e = root.Remove(p); e != nil {
fail(w, 400, "delete failed")
return
}
locked.Release()
locked = nil
a.s.AddAudit(u.ID, "workspace.delete", p, "")
writeJSON(w, map[string]bool{"ok": true})
}
func (a *API) wsUpload(w http.ResponseWriter, r *http.Request, root *os.Root, u store.User) {
if r.Method != "POST" {
fail(w, 405, "method not allowed")
return
}
select {
case uploadSlots <- struct{}{}:
defer func() { <-uploadSlots }()
default:
fail(w, 429, "too many concurrent uploads")
return
}
usedBefore, usageErr := a.userUsage(u)
if usageErr != nil || usedBefore >= u.Quota {
fail(w, 413, "quota exceeded or usage unavailable")
return
}
r.Body = http.MaxBytesReader(w, r.Body, maxUpload+(1<<20))
controller := http.NewResponseController(w)
if err := controller.SetReadDeadline(time.Now().Add(30 * time.Second)); err != nil && !errors.Is(err, http.ErrNotSupported) {
fail(w, 400, "upload unavailable")
return
}
defer controller.SetReadDeadline(time.Time{})
if r.ParseMultipartForm(1<<20) != nil {
fail(w, 400, "invalid upload")
return
}
defer r.MultipartForm.RemoveAll()
user := a.userLocks.Acquire(u.ID)
locked := user
defer func() {
if locked != nil {
locked.Release()
}
}()
dir, e := safeRel(r.FormValue("path"))
if e != nil || rootSafe(root, dir, false) != nil {
fail(w, 400, "invalid directory")
return
}
input, h, e := r.FormFile("file")
if e != nil {
fail(w, 400, "file required")
return
}
defer input.Close()
name, e := safeRel(h.Filename)
if e != nil || name == "." || path.Base(name) != name {
fail(w, 400, "invalid filename")
return
}
if !uploadExtensionAllowed(name, os.Getenv("UPLOAD_ALLOWED_EXTENSIONS")) {
fail(w, 415, "upload file type denied")
return
}
target := path.Join(dir, name)
if _, e = root.Lstat(target); !errors.Is(e, os.ErrNotExist) {
fail(w, 409, "target exists")
return
}
if err := a.s.DB.QueryRow("SELECT quota FROM users WHERE id=?", u.ID).Scan(&u.Quota); err != nil {
fail(w, 413, "quota unavailable")
return
}
used, e := a.userUsage(u)
if e != nil || h.Size > maxUpload || used+h.Size > u.Quota {
fail(w, 413, "quota or upload limit exceeded")
return
}
tmp := path.Join(dir, ".upload-"+store.ID())
f, e := root.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
if e != nil {
fail(w, 400, "upload directory unavailable")
return
}
defer root.Remove(tmp)
n, e := io.Copy(f, io.LimitReader(input, maxUpload+1))
if e == nil && (n > maxUpload || used+n > u.Quota) {
e = errors.New("quota exceeded")
}
if e == nil {
e = f.Chown(1000, 1000)
}
if e == nil {
e = f.Sync()
}
f.Close()
if e == nil {
e = renameNoReplace(root, tmp, target)
}
if errors.Is(e, os.ErrExist) {
fail(w, 409, "target exists")
return
}
if e != nil {
fail(w, 400, "upload failed")
return
}
locked.Release()
locked = nil
a.s.AddAudit(u.ID, "workspace.upload", target, "")
writeJSON(w, map[string]bool{"ok": true})
}
// Empty policy permits arbitrary workspace files; configured entries are exact
// case-insensitive extensions, not MIME claims supplied by the client.
func uploadExtensionAllowed(name, policy string) bool {
if strings.TrimSpace(policy) == "" {
return true
}
ext := strings.ToLower(path.Ext(name))
for _, entry := range strings.Split(policy, ",") {
entry = strings.ToLower(strings.TrimSpace(entry))
if strings.HasPrefix(entry, ".") && len(entry) > 1 && ext == entry {
return true
}
}
return false
}
func (a *API) userUsage(u store.User) (int64, error) {
base, e := os.OpenRoot(a.root)
if e != nil {
return 0, e
}
defer base.Close()
root, e := base.OpenRoot(path.Join("workspaces", u.ID))
if errors.Is(e, os.ErrNotExist) {
return 0, nil
}
if e != nil {
return 0, e
}
defer root.Close()
var total int64
count := 0
var walk func(string, int) error
walk = func(p string, depth int) error {
count++
if count > 100000 || depth > 128 {
return errors.New("tree limit")
}
st, e := root.Lstat(p)
if e != nil {
return e
}
if st.Mode().IsRegular() {
total += st.Size()
return nil
}
if !st.IsDir() {
return nil
}
f, e := root.OpenFile(p, os.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW, 0)
if e != nil {
return e
}
defer f.Close()
for {
ds, readErr := f.ReadDir(128)
for _, d := range ds {
if e = walk(path.Join(p, d.Name()), depth+1); e != nil {
return e
}
}
if readErr == io.EOF {
return nil
}
if readErr != nil {
return readErr
}
if count >= 100000 {
return errors.New("tree limit")
}
}
}
e = walk(".", 0)
return total, e
}
func (a *API) wsDownload(w http.ResponseWriter, r *http.Request, root *os.Root) {
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
p, e := safeRel(r.URL.Query().Get("path"))
if e != nil {
fail(w, 400, "invalid path")
return
}
anchor, e := root.Open(".")
if e != nil {
fail(w, 404, "not found")
return
}
defer anchor.Close()
f, e := openDownload(anchor, p)
if e != nil {
fail(w, 404, "not found")
return
}
defer f.Close()
st, e := f.Stat()
if e == nil && st.IsDir() {
a.wsDownloadZip(w, r, f, p)
return
}
if e != nil || !st.Mode().IsRegular() {
fail(w, 400, "invalid file")
return
}
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": path.Base(p)}))
http.ServeContent(w, r, path.Base(p), st.ModTime(), f)
}
func openDownload(anchor *os.File, p string) (*os.File, error) {
fd, err := unix.Openat2(int(anchor.Fd()), p, &unix.OpenHow{
Flags: unix.O_RDONLY | unix.O_NONBLOCK | unix.O_CLOEXEC,
Resolve: unix.RESOLVE_BENEATH | unix.RESOLVE_NO_SYMLINKS | unix.RESOLVE_NO_XDEV,
})
if err != nil {
return nil, err
}
f := os.NewFile(uintptr(fd), p)
st, err := f.Stat()
if err == nil && (!st.IsDir() && !st.Mode().IsRegular() || st.Mode().IsRegular() && st.Sys().(*syscall.Stat_t).Nlink != 1) {
err = errors.New("unsafe file")
}
if err != nil {
f.Close()
return nil, err
}
return f, nil
}
type zipEntry struct {
name string
info os.FileInfo
}
func zipManifest(ctx context.Context, anchor *os.File) ([]zipEntry, error) {
var entries []zipEntry
var total int64
var walk func(string, int) error
walk = func(p string, depth int) error {
if err := ctx.Err(); err != nil {
return err
}
if depth > maxZipDepth || len(entries) >= maxZipEntries {
return errZipLimit
}
f, err := openDownload(anchor, p)
if err != nil {
return err
}
defer f.Close()
st, err := f.Stat()
if err != nil {
return err
}
if p != "." {
if st.Size() < 0 || st.Mode().IsRegular() && st.Size() > maxZipBytes-total {
return errZipLimit
}
if st.Mode().IsRegular() {
total += st.Size()
}
entries = append(entries, zipEntry{p, st})
}
if !st.IsDir() {
return nil
}
for {
ds, err := f.ReadDir(128)
if err != nil && err != io.EOF {
return err
}
for _, d := range ds {
if strings.ContainsAny(d.Name(), "\\:") {
return errors.New("unsafe archive name")
}
if e := walk(path.Join(p, d.Name()), depth+1); e != nil {
return e
}
}
if err == io.EOF {
return nil
}
}
}
err := walk(".", 0)
return entries, err
}
type zipContextWriter struct {
ctx context.Context
w io.Writer
}
func (w zipContextWriter) Write(p []byte) (int, error) {
if err := w.ctx.Err(); err != nil {
return 0, err
}
return w.w.Write(p)
}
func streamZip(ctx context.Context, w io.Writer, anchor *os.File, entries []zipEntry) error {
z := zip.NewWriter(zipContextWriter{ctx, w})
buf := make([]byte, 32<<10)
for _, item := range entries {
if err := ctx.Err(); err != nil {
return err
}
f, err := openDownload(anchor, item.name)
if err != nil {
return err
}
err = func() error {
defer f.Close()
st, err := f.Stat()
if err != nil {
return err
}
if !os.SameFile(st, item.info) || st.Mode() != item.info.Mode() || st.Size() != item.info.Size() || !st.ModTime().Equal(item.info.ModTime()) {
return errors.New("file changed")
}
h, err := zip.FileInfoHeader(st)
if err != nil {
return err
}
h.Name = item.name
if st.IsDir() {
h.Name += "/"
}
dst, err := z.CreateHeader(h)
if err != nil || st.IsDir() {
return err
}
n, err := io.CopyBuffer(dst, io.LimitReader(f, item.info.Size()), buf)
if err == nil && n != item.info.Size() {
err = io.ErrUnexpectedEOF
}
if err == nil {
after, statErr := f.Stat()
if statErr != nil {
return statErr
}
if after.Size() != st.Size() || !after.ModTime().Equal(st.ModTime()) {
return errors.New("file changed")
}
}
return err
}()
if err != nil {
return err
}
}
return z.Close()
}
func (a *API) wsDownloadZip(w http.ResponseWriter, r *http.Request, anchor *os.File, p string) {
select {
case zipSlots <- struct{}{}:
defer func() { <-zipSlots }()
default:
fail(w, 429, "too many archive downloads")
return
}
ctx, cancel := context.WithTimeout(r.Context(), zipTimeout)
defer cancel()
controller := http.NewResponseController(w)
if err := controller.SetWriteDeadline(time.Now().Add(zipTimeout)); err != nil && !errors.Is(err, http.ErrNotSupported) {
fail(w, 500, "download unavailable")
return
}
defer controller.SetWriteDeadline(time.Time{})
entries, err := zipManifest(ctx, anchor)
if err != nil {
code := 400
if errors.Is(err, errZipLimit) {
code = 413
}
fail(w, code, "cannot archive directory")
return
}
name := path.Base(p)
if p == "." {
name = "workspace"
}
w.Header().Set("Content-Type", "application/zip")
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": name + ".zip"}))
if err := streamZip(ctx, w, anchor, entries); err != nil {
// A failed stream must not become a successful, silently incomplete archive.
panic(http.ErrAbortHandler)
}
}
const maxDirectoryEntries = 10000
var errDirectoryLimit = errors.New("directory entry limit exceeded")
func readRootDir(root *os.Root, p string) ([]os.DirEntry, error) {
f, e := root.OpenFile(p, os.O_RDONLY|syscall.O_DIRECTORY|syscall.O_NOFOLLOW, 0)
if e != nil {
return nil, e
}
defer f.Close()
entries, err := f.ReadDir(maxDirectoryEntries + 1)
if len(entries) > maxDirectoryEntries {
return nil, errDirectoryLimit
}
if err == io.EOF {
err = nil
}
return entries, err
}
internal/api/files_test.go
-729
View File
@@ -1,729 +0,0 @@
package api
import (
"archive/zip"
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
)
func TestWorkspaceText(t *testing.T) {
a, sid := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) { t.Error("unexpected Docker call") })
for _, i := range []store.Instance{{ID: "one", UID: "owner"}, {ID: "foreign", UID: "other"}} {
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
}
dir := filepath.Join(a.root, "workspaces/owner/instances/one/home")
if err := os.MkdirAll(dir, 0750); err != nil {
t.Fatal(err)
}
file := filepath.Join(dir, "file")
putFile := func(content string) {
t.Helper()
if err := os.WriteFile(file, []byte(content), 0751); err != nil {
t.Fatal(err)
}
}
putFile("original")
if _, err := a.s.DB.Exec("UPDATE users SET quota=? WHERE id='owner'", 32<<20); err != nil {
t.Fatal(err)
}
request := func(method, query, body, session string, want int) map[string]string {
t.Helper()
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest(method, "/api/ws/text?"+query, body, session))
if w.Code != want {
t.Fatalf("%s %s: got %d want %d: %s", method, query, w.Code, want, w.Body.String())
}
out := map[string]string{}
if want == 200 {
if err := json.Unmarshal(w.Body.Bytes(), &out); err != nil {
t.Fatal(err)
}
}
return out
}
q := "instance=one&path=file"
body := func(content, version string) string {
b, _ := json.Marshal(map[string]string{"content": content, "version": version})
return string(b)
}
request("GET", q, "", "", 401)
request("GET", "instance=foreign&path=file", "", sid, 404)
request("GET", "path=file", "", sid, 410)
for _, p := range []string{"../file", "..", "/etc/passwd", ".", "", "\x00", "./file", "a/../file", "a/./file", "a//file", "file/"} {
for _, method := range []string{"GET", "PUT"} {
request(method, "instance=one&path="+url.QueryEscape(p), "", sid, 400)
}
}
for _, tc := range []struct{ exact, counterpart string }{
{"config.yaml ", "config.yaml"},
{" config.yaml", "config.yaml"},
{`a\b`, "a/b"},
} {
for name, content := range map[string]string{tc.exact: "exact", tc.counterpart: "counterpart"} {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, name), []byte(content), 0600); err != nil {
t.Fatal(err)
}
}
exactQuery := "instance=one&path=" + url.QueryEscape(tc.exact)
got := request("GET", exactQuery, "", sid, 200)
if got["content"] != "exact" {
t.Fatalf("GET %q returned %q", tc.exact, got["content"])
}
request("PUT", exactQuery, body("updated", got["version"]), sid, 200)
if got := request("GET", exactQuery, "", sid, 200); got["content"] != "updated" {
t.Fatalf("GET %q after PUT returned %q", tc.exact, got["content"])
}
for name, want := range map[string]string{tc.exact: "updated", tc.counterpart: "counterpart"} {
got, err := os.ReadFile(filepath.Join(dir, name))
if err != nil || string(got) != want {
t.Fatalf("file %q: got %q, want %q: %v", name, got, want, err)
}
}
}
request("POST", q, "", sid, 405)
v := request("GET", q, "", sid, 200)["version"]
if v == "" {
t.Fatal("missing version")
}
request("PUT", q, "{}", sid, 400)
request("PUT", q, body("next", "stale"), sid, 409)
request("PUT", q, body("next", ""), sid, 400)
for _, content := range []string{"nul\x00", "del\x7f", "control\x01", "invalid\xff"} {
putFile(content)
request("GET", q, "", sid, 415)
}
putFile(strings.Repeat("x", maxTextBytes+1))
request("GET", q, "", sid, 413)
putFile(strings.Repeat("x", maxTextBytes))
request("GET", q, "", sid, 200)
putFile("original")
v = request("GET", q, "", sid, 200)["version"]
request("PUT", q, body(strings.Repeat("x", maxTextBytes+1), v), sid, 413)
request("PUT", q, body("bad\x00", v), sid, 415)
request("PUT", q, "{\"content\":\"\xff\",\"version\":\"v\"}", sid, 415)
request("PUT", q, strings.Repeat(" ", 6*maxTextBytes+4097), sid, 413)
if err := os.Symlink(file, filepath.Join(dir, "link")); err != nil {
t.Fatal(err)
}
if err := os.Symlink(dir, filepath.Join(dir, "parent")); err != nil {
t.Fatal(err)
}
for _, p := range []string{"link", "parent/file"} {
request("GET", "instance=one&path="+p, "", sid, 404)
request("PUT", "instance=one&path="+p, body("next", v), sid, 409)
}
if err := os.Link(file, filepath.Join(dir, "hard")); err != nil {
t.Fatal(err)
}
request("GET", q, "", sid, 404)
request("PUT", q, body("next", v), sid, 409)
if err := os.Remove(filepath.Join(dir, "hard")); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("UPDATE users SET quota=8 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
request("PUT", q, body("next", v), sid, 413)
if _, err := a.s.DB.Exec("UPDATE users SET quota=? WHERE id='owner'", 32<<20); err != nil {
t.Fatal(err)
}
before, err := os.Stat(file)
if err != nil {
t.Fatal(err)
}
next := "你好\tline\r\n"
saved := request("PUT", q, body(next, v), sid, 200)["version"]
got := request("GET", q, "", sid, 200)
if got["content"] != next || got["version"] != saved || saved == v {
t.Fatal(got, saved)
}
after, err := os.Stat(file)
if err != nil {
t.Fatal(err)
}
bs, as := before.Sys().(*syscall.Stat_t), after.Sys().(*syscall.Stat_t)
if bs.Mode != as.Mode || bs.Uid != as.Uid || bs.Gid != as.Gid {
t.Fatal("metadata changed")
}
request("PUT", q, body("stale save", v), sid, 409)
results := make(chan int, 2)
for range 2 {
go func() {
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest("PUT", "/api/ws/text?"+q, body("concurrent", saved), sid))
results <- w.Code
}()
}
first, second := <-results, <-results
if !(first == 200 && second == 409 || first == 409 && second == 200) {
t.Fatalf("concurrent saves: %d %d", first, second)
}
saved = request("GET", q, "", sid, 200)["version"]
putFile("external")
request("PUT", q, body("stale save", saved), sid, 409)
if err := os.Remove(file); err != nil {
t.Fatal(err)
}
request("PUT", q, body("recreate", saved), sid, 409)
request("GET", q, "", sid, 404)
putFile("restored")
if _, err := a.s.DB.Exec("UPDATE instances SET status='error' WHERE id='one'"); err != nil {
t.Fatal(err)
}
request("PUT", q, body("blocked", saved), sid, 409)
request("GET", q, "", sid, 200)
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
for _, e := range entries {
if strings.HasPrefix(e.Name(), ".text-") {
t.Fatal("temporary file leaked")
}
}
}
func TestUploadExtensionPolicy(t *testing.T) {
for _, tc := range []struct {
name, policy string
want bool
}{
{"file.exe", "", true}, {"file.TXT", ".txt, .md", true},
{"file.txt.exe", ".txt", false}, {"file", ".txt", false},
{"file.exe", "*,txt", false}, {"file.json", ".txt,.json", true},
} {
if got := uploadExtensionAllowed(tc.name, tc.policy); got != tc.want {
t.Errorf("%q %q: %v", tc.name, tc.policy, got)
}
}
}
func downloadTestRoot(t *testing.T) (string, *os.Root) {
t.Helper()
dir := t.TempDir()
root, err := os.OpenRoot(dir)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { root.Close() })
return dir, root
}
func TestDirectoryDownloadZip(t *testing.T) {
dir, root := downloadTestRoot(t)
if err := os.MkdirAll(filepath.Join(dir, "docs/empty"), 0700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "docs/你好.txt"), []byte("hello zip"), 0600); err != nil {
t.Fatal(err)
}
for _, p := range []string{"docs", ".", "", "docs/empty"} {
t.Run(p, func(t *testing.T) {
w := httptest.NewRecorder()
(&API{}).wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path="+url.QueryEscape(p), nil), root)
if w.Code != 200 || w.Header().Get("Content-Type") != "application/zip" || !strings.Contains(w.Header().Get("Content-Disposition"), ".zip") {
t.Fatal(w.Code, w.Header(), w.Body.String())
}
z, err := zip.NewReader(bytes.NewReader(w.Body.Bytes()), int64(w.Body.Len()))
if err != nil {
t.Fatal(err)
}
want := map[string]string{"empty/": "", "你好.txt": "hello zip"}
if p == "." || p == "" {
want = map[string]string{"docs/": "", "docs/empty/": "", "docs/你好.txt": "hello zip"}
}
if p == "docs/empty" {
want = map[string]string{}
}
if len(z.File) != len(want) {
t.Fatalf("entries %d want %d", len(z.File), len(want))
}
for _, f := range z.File {
expect, ok := want[f.Name]
if !ok {
t.Fatal("unexpected entry", f.Name)
}
r, err := f.Open()
if err != nil {
t.Fatal(err)
}
b, err := io.ReadAll(r)
r.Close()
if err != nil || string(b) != expect {
t.Fatal(f.Name, string(b), err)
}
}
})
}
r := httptest.NewRequest("GET", "/api/ws/download?path=docs/"+url.QueryEscape("你好.txt"), nil)
r.Header.Set("Range", "bytes=1-3")
w := httptest.NewRecorder()
(&API{}).wsDownload(w, r, root)
if w.Code != 206 || w.Body.String() != "ell" {
t.Fatal(w.Code, w.Body.String())
}
}
func TestDirectoryDownloadRejectsUnsafeEntries(t *testing.T) {
for _, kind := range []string{"symlink", "internal-link", "hardlink", "fifo", "backslash", "drive"} {
t.Run(kind, func(t *testing.T) {
dir, root := downloadTestRoot(t)
if err := os.Mkdir(filepath.Join(dir, "docs"), 0700); err != nil {
t.Fatal(err)
}
outside := filepath.Join(t.TempDir(), "secret")
if err := os.WriteFile(outside, []byte("secret"), 0600); err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "docs/unsafe")
var err error
switch kind {
case "symlink":
err = os.Symlink(outside, target)
case "internal-link":
err = os.Symlink(".", target)
case "hardlink":
err = os.Link(outside, target)
case "fifo":
err = syscall.Mkfifo(target, 0600)
case "backslash":
err = os.WriteFile(filepath.Join(dir, `docs/..\secret`), nil, 0600)
case "drive":
err = os.WriteFile(filepath.Join(dir, "docs/C:secret"), nil, 0600)
}
if err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
(&API{}).wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path=docs", nil), root)
if w.Code != 400 || strings.Contains(w.Body.String(), "secret") {
t.Fatal(w.Code, w.Body.String())
}
})
}
dir, root := downloadTestRoot(t)
if err := os.Mkdir(filepath.Join(dir, "real"), 0700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "real/file"), []byte("secret"), 0600); err != nil {
t.Fatal(err)
}
if err := os.Symlink("real", filepath.Join(dir, "link")); err != nil {
t.Fatal(err)
}
for _, p := range []string{"../secret", "/etc/passwd", `..\secret`, "link", "link/file"} {
w := httptest.NewRecorder()
(&API{}).wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path="+url.QueryEscape(p), nil), root)
if w.Code < 400 {
t.Fatal("unsafe path accepted", p)
}
}
}
func TestDirectoryDownloadLimits(t *testing.T) {
for _, limit := range []string{"size", "depth", "count"} {
t.Run(limit, func(t *testing.T) {
dir, root := downloadTestRoot(t)
switch limit {
case "size":
f, err := os.Create(filepath.Join(dir, "large"))
if err != nil {
t.Fatal(err)
}
err = f.Truncate(maxZipBytes + 1)
f.Close()
if err != nil {
t.Fatal(err)
}
case "depth":
if err := os.MkdirAll(filepath.Join(dir, strings.Repeat("d/", maxZipDepth+1)), 0700); err != nil {
t.Fatal(err)
}
case "count":
for i := 0; i <= maxZipEntries; i++ {
if err := os.WriteFile(filepath.Join(dir, fmt.Sprint(i)), nil, 0600); err != nil {
t.Fatal(err)
}
}
}
w := httptest.NewRecorder()
(&API{}).wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path=.", nil), root)
if w.Code != 413 {
t.Fatal(w.Code, w.Body.String())
}
})
}
}
func TestZipManifestRechecksAndCancellation(t *testing.T) {
for _, change := range []string{"symlink", "grow", "cancel"} {
t.Run(change, func(t *testing.T) {
dir, root := downloadTestRoot(t)
p := filepath.Join(dir, "file")
if err := os.WriteFile(p, []byte("data"), 0600); err != nil {
t.Fatal(err)
}
f, err := root.Open(".")
if err != nil {
t.Fatal(err)
}
defer f.Close()
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
es, err := zipManifest(ctx, f)
if err != nil {
t.Fatal(err)
}
switch change {
case "symlink":
if err := os.Remove(p); err != nil {
t.Fatal(err)
}
if err := os.Symlink("/etc/passwd", p); err != nil {
t.Fatal(err)
}
case "grow":
if err := os.WriteFile(p, []byte("more data"), 0600); err != nil {
t.Fatal(err)
}
case "cancel":
cancel()
}
if err := streamZip(ctx, io.Discard, f, es); err == nil {
t.Fatal("unsafe stream accepted")
}
if change == "cancel" {
if _, err := zipManifest(ctx, f); !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
}
})
}
}
type failingZipWriter struct{}
func (failingZipWriter) Write([]byte) (int, error) { return 0, io.ErrClosedPipe }
func TestZipWriteFailureAndSizeBound(t *testing.T) {
dir, root := downloadTestRoot(t)
p := filepath.Join(dir, "file")
if err := os.WriteFile(p, bytes.Repeat([]byte("x"), 65536), 0600); err != nil {
t.Fatal(err)
}
f, err := root.Open(".")
if err != nil {
t.Fatal(err)
}
defer f.Close()
es, err := zipManifest(context.Background(), f)
if err != nil {
t.Fatal(err)
}
if err := streamZip(context.Background(), failingZipWriter{}, f, es); !errors.Is(err, io.ErrClosedPipe) {
t.Fatal(err)
}
w := &growingZipWriter{path: p}
if err := streamZip(context.Background(), w, f, es); err == nil {
t.Fatal("concurrent growth accepted")
}
if w.Len() > 65536+1024 {
t.Fatal("stream exceeded manifest bound", w.Len())
}
}
type growingZipWriter struct {
bytes.Buffer
path string
}
func (w *growingZipWriter) Write(p []byte) (int, error) {
if w.path != "" {
f, err := os.OpenFile(w.path, os.O_WRONLY|os.O_APPEND, 0)
if err != nil {
return 0, err
}
_, err = f.Write([]byte("growth"))
f.Close()
if err != nil {
return 0, err
}
w.path = ""
}
return w.Buffer.Write(p)
}
type blockedZipWriter struct {
*httptest.ResponseRecorder
started chan struct{}
release chan struct{}
deadlines []time.Time
}
func (w *blockedZipWriter) Write(p []byte) (int, error) {
if w.started != nil {
close(w.started)
w.started = nil
<-w.release
}
return w.ResponseRecorder.Write(p)
}
func (w *blockedZipWriter) SetWriteDeadline(d time.Time) error {
w.deadlines = append(w.deadlines, d)
return nil
}
func TestZipTransferNoLocksAndConcurrencyLimit(t *testing.T) {
_, root := downloadTestRoot(t)
a := &API{}
started := make(chan struct{})
w := &blockedZipWriter{ResponseRecorder: httptest.NewRecorder(), started: started, release: make(chan struct{})}
done := make(chan struct{})
go func() {
defer close(done)
a.wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path=.", nil), root)
}()
defer func() { close(w.release); <-done }()
select {
case <-started:
case <-time.After(2 * time.Second):
t.Fatal("download not started")
}
if a.userLocks.held() {
t.Fatal("user write lock held across write")
}
if a.instanceLocks.held() {
t.Fatal("instance lock held across write")
}
if len(w.deadlines) != 1 || w.deadlines[0].IsZero() {
t.Fatal("missing deadline")
}
for i := 1; i < cap(zipSlots); i++ {
zipSlots <- struct{}{}
}
defer func() {
for i := 1; i < cap(zipSlots); i++ {
<-zipSlots
}
}()
busy := httptest.NewRecorder()
a.wsDownload(busy, httptest.NewRequest("GET", "/api/ws/download?path=.", nil), root)
if busy.Code != 429 {
t.Fatal(busy.Code)
}
}
func TestRenameNoReplace(t *testing.T) {
dir := t.TempDir()
root, err := os.OpenRoot(dir)
if err != nil {
t.Fatal(err)
}
defer root.Close()
for _, source := range []string{"source", ".upload-test"} {
if err := os.WriteFile(filepath.Join(dir, source), []byte("new"), 0600); err != nil {
t.Fatal(err)
}
if _, err := root.Lstat("target"); !errors.Is(err, os.ErrNotExist) {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "target"), []byte("runtime"), 0600); err != nil {
t.Fatal(err)
}
if err := renameNoReplace(root, source, "target"); !errors.Is(err, os.ErrExist) {
t.Fatalf("overwrite: %v", err)
}
b, err := root.ReadFile("target")
if err != nil || string(b) != "runtime" {
t.Fatalf("target changed: %q %v", b, err)
}
if err := root.Remove("target"); err != nil {
t.Fatal(err)
}
if err := renameNoReplace(root, source, "target"); err != nil {
t.Fatal(err)
}
if err := root.Remove("target"); err != nil {
t.Fatal(err)
}
}
if err := os.Mkdir(filepath.Join(dir, "real"), 0700); err != nil {
t.Fatal(err)
}
if err := os.Symlink("real", filepath.Join(dir, "link")); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "source"), []byte("new"), 0600); err != nil {
t.Fatal(err)
}
for _, target := range []string{"link/target", "../target"} {
if err := renameNoReplace(root, "source", target); err == nil {
t.Fatalf("unsafe parent accepted: %s", target)
}
}
if err := renameNoReplace(root, "link", "target"); err == nil {
t.Fatal("symlink source accepted")
}
}
type stalledUpload struct {
started chan struct{}
release chan struct{}
}
func (b *stalledUpload) Read([]byte) (int, error) {
close(b.started)
<-b.release
return 0, io.EOF
}
func (b *stalledUpload) Close() error { return nil }
type deadlineRecorder struct {
*httptest.ResponseRecorder
deadlines []time.Time
}
func (w *deadlineRecorder) SetReadDeadline(d time.Time) error {
w.deadlines = append(w.deadlines, d)
return nil
}
func TestLegacyWorkspaceRemoved(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Fatal("unexpected docker call") })
for _, method := range []string{"GET", "POST", "DELETE"} {
for _, endpoint := range []string{"tree", "download", "upload", "mkdir", "rename", "delete"} {
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest(method, "/api/ws/"+endpoint, "", sid))
if w.Code != http.StatusGone {
t.Fatalf("%s %s: %d", method, endpoint, w.Code)
}
}
}
}
func TestWorkspaceSafetyAndUpload(t *testing.T) {
s, err := store.Open(filepath.Join(t.TempDir(), "test.sqlite"))
if err != nil {
t.Fatal(err)
}
defer s.Close()
a := &API{s: s, root: t.TempDir()}
u := store.User{ID: "user", Quota: 8}
if err := s.CreateUser(u, "unused"); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(a.root, "workspaces/user/instances/one/home"), 0750); err != nil {
t.Fatal(err)
}
root, e := a.openWorkspace(u, "one")
if e != nil {
t.Fatal(e)
}
defer root.Close()
for _, p := range []string{"", ".", "a/..", "../other", "/tmp"} {
w := httptest.NewRecorder()
a.wsDelete(w, httptest.NewRequest("DELETE", "/api/ws/delete?path="+p, nil), root, u)
if w.Code < 400 {
t.Fatalf("deleted %q", p)
}
}
outside := t.TempDir()
os.WriteFile(filepath.Join(outside, "secret"), []byte("secret"), 0600)
if e = os.Symlink(outside, filepath.Join(a.root, "workspaces/user/instances/one/home/link")); e != nil {
t.Fatal(e)
}
w := httptest.NewRecorder()
a.wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path=link/secret", nil), root)
if w.Code < 400 {
t.Fatal("escaped root")
}
upload := func(name, content string) int {
var b bytes.Buffer
m := multipart.NewWriter(&b)
m.WriteField("path", ".")
f, _ := m.CreateFormFile("file", name)
f.Write([]byte(content))
m.Close()
r := httptest.NewRequest("POST", "/api/ws/upload", &b)
r.Header.Set("Content-Type", m.FormDataContentType())
w := httptest.NewRecorder()
a.wsUpload(w, r, root, u)
return w.Code
}
if code := upload("file.txt", "hello"); code != 200 {
t.Fatal(code)
}
if code := upload("big.txt", "12345"); code != 413 {
t.Fatal("quota", code)
}
if code := upload("file.txt", "x"); code != 409 {
t.Fatal("overwrite", code)
}
w = httptest.NewRecorder()
a.wsDownload(w, httptest.NewRequest("GET", "/api/ws/download?path=file.txt", nil), root)
if w.Code != 200 || w.Body.String() != "hello" {
t.Fatal(w.Code, w.Body.String())
}
for _, op := range []struct{ action, body string }{
{"mkdir", `{"path":"folder"}`},
{"rename", `{"path":"folder","new_path":"renamed"}`},
{"delete", ""},
} {
w := httptest.NewRecorder()
r := httptest.NewRequest("POST", "/api/ws/"+op.action, strings.NewReader(op.body))
switch op.action {
case "mkdir":
a.wsMkdir(w, r, root, u)
case "rename":
a.wsRename(w, r, root, u)
case "delete":
a.wsDelete(w, httptest.NewRequest("DELETE", "/api/ws/delete?path=renamed", nil), root, u)
}
if w.Code != 200 {
t.Fatal(op.action, w.Code, w.Body.String())
}
}
rows, err := s.DB.Query("SELECT uid,action,target,detail FROM audit ORDER BY id")
if err != nil {
t.Fatal(err)
}
defer rows.Close()
want := [][3]string{{"workspace.upload", "file.txt", ""}, {"workspace.mkdir", "folder", ""}, {"workspace.rename", "folder", "renamed"}, {"workspace.delete", "renamed", ""}}
i := 0
for rows.Next() {
var uid, action, target, detail string
if err := rows.Scan(&uid, &action, &target, &detail); err != nil {
t.Fatal(err)
}
if i >= len(want) || uid != u.ID || [3]string{action, target, detail} != want[i] {
t.Fatal("unexpected audit", uid, action, target, detail)
}
i++
}
if err := rows.Err(); err != nil {
t.Fatal(err)
}
if i != len(want) {
t.Fatal("missing audit entries", i)
}
}
internal/api/home_policy_test.go
-86
View File
@@ -1,86 +0,0 @@
package api
import (
"archive/zip"
"bytes"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
// Full-home access is an explicit owner capability, not a credential write-only
// boundary. Keep ownership, authentication and traversal checks independent.
func TestFullHomeOwnerPolicy(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker operation") })
for _, i := range []store.Instance{{ID: "owned", UID: "owner", Status: "running"}, {ID: "foreign", UID: "other", Status: "running"}} {
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
home := filepath.Join(a.root, "workspaces", i.UID, "instances", i.ID, "home")
if err := os.MkdirAll(filepath.Join(home, "profiles", "web"), 0700); err != nil {
t.Fatal(err)
}
for path, contents := range map[string]string{".credentials.yaml": "synthetic-test-credential", "dsh-ssh.json": "synthetic-test-ssh", "profiles/web/cordis.patch.yml": "old: value"} {
if err := os.WriteFile(filepath.Join(home, path), []byte(contents), 0600); err != nil {
t.Fatal(err)
}
}
}
request := func(method, path, body, session string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest(method, path, body, session))
return w
}
for _, name := range []string{".credentials.yaml", "dsh-ssh.json"} {
w := request("GET", "/api/ws/download?instance=owned&path="+name, "", sid)
if w.Code != 200 || w.Body.Len() == 0 {
t.Fatal(name, w.Code, w.Body.String())
}
if w = request("GET", "/api/ws/download?instance=foreign&path="+name, "", sid); w.Code != 404 {
t.Fatal("cross-owner read", w.Code)
}
if w = request("GET", "/api/ws/download?instance=owned&path="+name, "", ""); w.Code != 401 {
t.Fatal("anonymous read", w.Code)
}
}
w := request("GET", "/api/ws/download?instance=owned&path=.", "", sid)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
archive, err := zip.NewReader(bytes.NewReader(w.Body.Bytes()), int64(w.Body.Len()))
if err != nil {
t.Fatal(err)
}
found := false
for _, f := range archive.File {
if f.Name == ".credentials.yaml" {
r, e := f.Open()
if e != nil {
t.Fatal(e)
}
data, e := io.ReadAll(r)
r.Close()
if e != nil || string(data) != "synthetic-test-credential" {
t.Fatal("wrong credential archive entry", e)
}
found = true
}
}
if !found {
t.Fatal("owner archive unexpectedly filters credentials")
}
// Running state intentionally does not force the controlled profile workflow.
w = request("POST", "/api/ws/rename?instance=owned", `{"path":"profiles/web/cordis.patch.yml","new_path":"profiles/web/saved.yml"}`, sid)
if w.Code != 200 {
t.Fatal("owner profile rename blocked", w.Code, w.Body.String())
}
w = request("GET", "/api/ws/download?instance=owned&path=../../other/instances/foreign/home/.credentials.yaml", "", sid)
if w.Code != 400 && w.Code != 404 {
t.Fatal("traversal allowed", w.Code)
}
}
internal/api/images.go
-424
View File
@@ -1,424 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"net/http"
"net/url"
"regexp"
"strings"
"sync"
"time"
)
// Caller holds buildQueue; interrupted jobs cannot be reported as successful.
func (a *API) ensureBuilds() error {
if buildQueue.initialized[a.s.DB] {
return nil
}
_, err := a.s.DB.Exec(`CREATE TABLE IF NOT EXISTS image_builds(id TEXT PRIMARY KEY,version TEXT NOT NULL,status TEXT NOT NULL,logs TEXT NOT NULL DEFAULT '',error TEXT NOT NULL DEFAULT '',image_id TEXT NOT NULL DEFAULT '',created_at TEXT NOT NULL); CREATE TABLE IF NOT EXISTS image_build_requests(user_id TEXT NOT NULL,build_id TEXT PRIMARY KEY,created_at TEXT NOT NULL); CREATE TABLE IF NOT EXISTS images(image_id TEXT PRIMARY KEY,version TEXT NOT NULL,recipe_rev TEXT,platform TEXT,tag TEXT NOT NULL DEFAULT '',status TEXT NOT NULL,built_at TEXT NOT NULL,usage_count INTEGER NOT NULL DEFAULT 0);`)
if err != nil {
return err
}
for _, q := range []string{
"CREATE INDEX IF NOT EXISTS image_builds_history ON image_builds(created_at DESC,id DESC)",
"ALTER TABLE image_builds ADD COLUMN requested_by TEXT NOT NULL DEFAULT ''",
"ALTER TABLE image_builds ADD COLUMN recipe_rev TEXT",
"ALTER TABLE image_builds ADD COLUMN platform TEXT",
"ALTER TABLE image_builds ADD COLUMN finished_at TEXT",
"ALTER TABLE image_builds ADD COLUMN experimental INTEGER NOT NULL DEFAULT 0",
"ALTER TABLE image_builds ADD COLUMN release_id TEXT NOT NULL DEFAULT ''",
"ALTER TABLE image_builds ADD COLUMN source_repository TEXT NOT NULL DEFAULT ''",
"ALTER TABLE image_builds ADD COLUMN source_commit TEXT NOT NULL DEFAULT ''",
"CREATE TABLE IF NOT EXISTS image_admissions(image_id TEXT NOT NULL,version TEXT NOT NULL,recipe_rev TEXT NOT NULL,platform TEXT NOT NULL,state TEXT NOT NULL,evidence TEXT NOT NULL,admitted_by TEXT NOT NULL,admitted_at TEXT NOT NULL,PRIMARY KEY(image_id,version,recipe_rev,platform))",
"ALTER TABLE images ADD COLUMN recipe_rev TEXT",
"ALTER TABLE images ADD COLUMN platform TEXT",
"ALTER TABLE images ADD COLUMN usage_count INTEGER NOT NULL DEFAULT 0",
} {
if _, e := a.s.DB.Exec(q); e != nil && !strings.Contains(strings.ToLower(e.Error()), "duplicate column") {
return e
}
}
_, err = a.s.DB.Exec("DROP INDEX IF EXISTS image_builds_version; DROP INDEX IF EXISTS image_builds_identity; CREATE UNIQUE INDEX image_builds_identity ON image_builds(version,recipe_rev,platform) WHERE release_id='' AND recipe_rev IS NOT NULL AND platform IS NOT NULL AND status IN ('queued','running','validating','success'); CREATE UNIQUE INDEX IF NOT EXISTS image_builds_release_identity ON image_builds(source_repository,source_commit,release_id,platform) WHERE release_id<>'' AND status IN ('queued','running','validating','success')")
if err != nil {
return err
}
if !buildQueue.initialized[a.s.DB] {
if _, err = a.s.DB.Exec("UPDATE image_builds SET status='failed',error='image template identity is unknown' WHERE status='queued' AND (recipe_rev IS NULL OR recipe_rev<>? OR platform IS NULL OR platform<>?)", imageRecipeRev, imagePlatform); err != nil {
return err
}
if _, err = a.s.DB.Exec("UPDATE image_builds SET status='failed',error='build interrupted by process restart' WHERE status IN ('running','validating')"); err != nil {
return err
}
buildQueue.initialized[a.s.DB] = true
}
return nil
}
func (a *API) builds(w http.ResponseWriter, r *http.Request) {
_, _, authErr := a.user(r)
if authErr != nil {
fail(w, 401, "authentication required")
return
}
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
a.listBuilds(w, r)
}
var errBuildRateLimited = errors.New("build limit: three new tasks per user per hour")
func (a *API) startBuildWorker() {
buildQueue.Lock()
defer buildQueue.Unlock()
a.startBuildWorkerLocked()
}
// startBuildWorkerLocked requires the caller to hold buildQueue: the active map
// is only safe under that lock.
func (a *API) startBuildWorkerLocked() {
if buildQueue.active[a.s.DB] {
return
}
buildQueue.active[a.s.DB] = true
go func() {
var lastCleanupAudit time.Time
var nextCleanup time.Time
var recoveryErr error
for {
select {
case <-a.s.Done():
buildQueue.Lock()
delete(buildQueue.active, a.s.DB)
buildQueue.Unlock()
return
default:
}
if time.Now().After(nextCleanup) {
buildWorker.Lock()
recovery, cancel := context.WithTimeout(context.Background(), 20*time.Second)
recoveryErr = a.recoverBuildChecks(recovery)
cancel()
buildWorker.Unlock()
if errors.Is(recoveryErr, errBuildCheckUncertain) {
nextCleanup = time.Now().Add(time.Minute)
}
}
if recoveryErr != nil {
if time.Since(lastCleanupAudit) >= time.Minute {
a.s.AddAudit("", "build.cleanup_error", "", sanitizeBuildLog(recoveryErr.Error()))
lastCleanupAudit = time.Now()
}
}
buildQueue.Lock()
var b buildRecord
row := a.s.DB.QueryRow("SELECT " + buildSummaryColumns + " FROM image_builds WHERE status='queued' AND release_id<>'' ORDER BY created_at,id LIMIT 1")
err := scanBuildSummary(row, &b)
if err == nil {
_, err = a.s.DB.Exec("UPDATE image_builds SET status='running' WHERE id=?", b.ID)
}
if err != nil {
if !errors.Is(err, sql.ErrNoRows) || recoveryErr != nil {
buildQueue.Unlock()
select {
case <-a.s.Done():
case <-time.After(time.Second):
}
continue
}
delete(buildQueue.active, a.s.DB)
buildQueue.Unlock()
return
}
buildQueue.Unlock()
a.runImageBuild(b)
nextCleanup = time.Time{}
}
}()
}
func (a *API) runImageBuild(b buildRecord) {
buildWorker.Lock()
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
logs := ""
rawLogs := ""
var logErr error
appendLog := func(s string) {
if len(rawLogs) < buildLogLimit {
remaining := buildLogLimit - len(rawLogs)
if len(s) > remaining {
s = s[:remaining]
}
rawLogs += s
}
logs = sanitizeBuildLog(rawLogs)
if logErr == nil {
_, logErr = a.s.DB.Exec("UPDATE image_builds SET logs=? WHERE id=?", logs, b.ID)
}
}
var id string
var err error
if b.ReleaseID == "" {
err = errors.New("Git release identity required")
} else {
var tarball io.Reader
tarball, err = a.releaseBuildContext(ctx, b)
if err == nil {
err = a.checkBuildDisk(ctx)
}
if err == nil {
id, err = a.docker.BuildRelease(ctx, tarball, b.Version, "dsh-web:release-"+b.SourceCommit, b.ReleaseID, b.SourceRepository, b.SourceCommit, appendLog)
}
}
if err == nil {
var matches bool
matches, err = a.buildImageMatches(ctx, id, b)
if err == nil && !matches {
err = errors.New("image template or platform mismatch")
}
}
if err == nil {
_, err = a.s.DB.Exec("UPDATE image_builds SET status='validating',image_id=? WHERE id=?", id, b.ID)
}
if err == nil {
err = a.verifyBuiltImage(ctx, id, b.Version, b.ID)
}
if err == nil {
err = logErr
}
status, detail := "success", ""
if err != nil {
status = "failed"
detail = sanitizeBuildLog(err.Error())
appendLog(string(rune(10)) + detail)
}
finished := time.Now().UTC().Format(time.RFC3339Nano)
buildWorker.Unlock()
if err := a.persistBuildResult(b, status, logs, detail, id, finished); err != nil {
go a.finishBuild(b, status, logs, detail, id, finished)
}
}
func (a *API) verifyBuiltImage(parent context.Context, image, version string, buildIDs ...string) (result error) {
ctx, cancel := context.WithTimeout(parent, 90*time.Second)
defer cancel()
id := store.ID()
if len(buildIDs) != 0 {
id = buildIDs[0]
}
if err := a.ensureBuildChecks(); err != nil {
return err
}
if _, err := a.s.DB.Exec("INSERT INTO image_build_checks(build_id,image_id,created_at) VALUES(?,?,?)", id, image, time.Now().UTC().Format(time.RFC3339Nano)); err != nil {
return err
}
var created struct {
ID string `json:"Id"`
}
rejected := false
defer func() {
cleanup, c := context.WithTimeout(context.Background(), 10*time.Second)
defer c()
var err error
if rejected {
_, err = a.s.DB.Exec("DELETE FROM image_build_checks WHERE build_id=?", id)
} else if created.ID != "" {
err = a.deleteBuildCheck(cleanup, id, created.ID)
} else {
err = a.cleanupBuildCheck(cleanup, id, image)
}
result = errors.Join(result, err)
}()
config := map[string]any{"Image": image, "User": "1000:1000", "Labels": map[string]string{"dsh-mgr.role": "build-check", "dsh-mgr.build": id}, "HostConfig": map[string]any{"NetworkMode": "none", "ReadonlyRootfs": true, "CapDrop": []string{"ALL"}, "SecurityOpt": []string{"no-new-privileges:true"}, "Memory": int64(1 << 30), "NanoCpus": int64(1e9), "PidsLimit": 256, "Tmpfs": map[string]string{"/tmp": "rw,exec,nosuid,nodev,size=256m,mode=1777", "/workspace": "rw,nosuid,nodev,size=256m,uid=1000,gid=1000"}}}
if err := a.docker.Do(ctx, http.MethodPost, "/containers/create?name="+url.QueryEscape(buildCheckName(id)), config, &created); err != nil {
rejected = dockerCreateRejected(err)
return err
}
if created.ID == "" {
return errors.New("verification container ID missing")
}
if _, err := a.s.DB.Exec("UPDATE image_build_checks SET container_id=? WHERE build_id=?", created.ID, id); err != nil {
return err
}
path := "/containers/" + url.PathEscape(created.ID)
if err := a.docker.Do(ctx, http.MethodPost, path+"/start", nil, nil); err != nil {
return err
}
script := `const end=Date.now()+60000;async function probe(){try{const r=await fetch('http://127.0.0.1:8080/',{redirect:'manual',signal:AbortSignal.timeout(2000)});if(r.status>=200&&r.status<500)process.exit(0);}catch{}if(Date.now()>end)process.exit(1);setTimeout(probe,500);}probe();`
{
expected, _ := json.Marshal(version)
script = `const cp=require('node:child_process');const actual=cp.execFileSync('dsh',['--version'],{encoding:'utf8',timeout:20000}).trim();if(actual!==` + string(expected) + `)process.exit(2);` + script
}
script = `const fs=require('node:fs');for(const [k,v] of Object.entries({DSH_HOME:'/workspace',HOME:'/workspace',XDG_DATA_HOME:'/workspace/.local/share',XDG_CONFIG_HOME:'/workspace/.config'})){if(process.env[k]!==v)process.exit(3);}if(!fs.lstatSync('/data/dsh-home').isSymbolicLink()||fs.realpathSync('/data/dsh-home')!=='/workspace')process.exit(4);fs.writeFileSync('/workspace/.home-check','isolated');if(fs.readFileSync('/data/dsh-home/.home-check','utf8')!=='isolated')process.exit(5);fs.unlinkSync('/workspace/.home-check');` + script
var exec struct {
ID string `json:"Id"`
}
if err := a.docker.Do(ctx, http.MethodPost, path+"/exec", map[string]any{"Cmd": []string{"node", "-e", script}, "User": "1000:1000"}, &exec); err != nil {
return err
}
if exec.ID == "" {
return errors.New("verification exec ID missing")
}
ep := "/exec/" + url.PathEscape(exec.ID)
if err := a.docker.Do(ctx, http.MethodPost, ep+"/start", map[string]bool{"Detach": true, "Tty": false}, nil); err != nil {
return err
}
ticker := time.NewTicker(250 * time.Millisecond)
defer ticker.Stop()
for {
var state struct {
Running bool
ExitCode int
}
if err := a.docker.Do(ctx, http.MethodGet, ep+"/json", nil, &state); err != nil {
return err
}
if !state.Running {
if state.ExitCode != 0 {
return fmt.Errorf("image startup verification failed (exit %d)", state.ExitCode)
}
container, err := a.docker.Inspect(ctx, created.ID)
if err != nil {
return err
}
if !container.State.Running {
return errors.New("verification container exited")
}
return nil
}
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
}
}
}
const buildLogLimit = 64 << 10
var fixedVersion = regexp.MustCompile(`^(0|[1-9][0-9]*)[.](0|[1-9][0-9]*)[.](0|[1-9][0-9]*)(-[0-9A-Za-z-]+([.][0-9A-Za-z-]+)*)?$`)
var buildSecrets = regexp.MustCompile(`(?i)(authorization["']?[[:space:]]*[:=][[:space:]]*["']?(?:bearer[[:space:]]+|basic[[:space:]]+)?|(?:token|password|passwd|secret|api[_-]?key|cookie)["']?[[:space:]]*[:=][[:space:]]*["']?)[^[:space:]"'&,;]+`)
var buildURLCredentials = regexp.MustCompile(`(https?://)[^/@[:space:]]+:[^/@[:space:]]+@`)
var buildQueue = struct {
sync.Mutex
initialized map[*sql.DB]bool
active map[*sql.DB]bool
}{initialized: map[*sql.DB]bool{}, active: map[*sql.DB]bool{}}
var buildWorker sync.Mutex
type imageRecord struct {
Admission string `json:"admission"`
AdmissionReason string `json:"admission_reason"`
Selectable bool `json:"selectable"`
ID string `json:"id"`
ImageID string `json:"image_id"`
Version string `json:"version"`
Tag string `json:"tag,omitempty"`
Status string `json:"status"`
Created string `json:"created_at"`
RecipeRev string `json:"recipe_rev,omitempty"`
Platform string `json:"platform,omitempty"`
UsageCount int `json:"usage_count"`
}
type buildRecord struct {
ReleaseID string `json:"release_id,omitempty"`
SourceRepository string `json:"source_repository,omitempty"`
SourceCommit string `json:"source_commit,omitempty"`
Experimental bool `json:"experimental"`
ImageID string `json:"image_id,omitempty"`
ID string `json:"id"`
Version string `json:"version"`
Status string `json:"status"`
Logs string `json:"logs,omitempty"`
Error string `json:"error,omitempty"`
Created string `json:"created_at"`
RequestedBy string `json:"requested_by,omitempty"`
RecipeRev string `json:"recipe_rev,omitempty"`
Platform string `json:"platform,omitempty"`
Finished string `json:"finished_at,omitempty"`
}
func (a *API) images(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodDelete {
a.deleteImage(w, r)
return
}
if _, _, err := a.user(r); err != nil {
fail(w, 401, "authentication required")
return
}
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
result, err := a.imageIndex(ctx)
if err != nil {
fail(w, 502, "image index reconciliation failed")
return
}
writeJSON(w, result)
}
func (a *API) successfulImageIDs(ctx context.Context) ([]string, error) {
rows, err := a.s.DB.QueryContext(ctx, "SELECT DISTINCT image_id FROM image_builds WHERE status='success' AND image_id<>''")
if err != nil {
return nil, err
}
ids := []string{}
for rows.Next() {
var id string
if err = rows.Scan(&id); err != nil {
rows.Close()
return nil, err
}
ids = append(ids, id)
}
err = rows.Err()
rows.Close()
if err != nil {
return nil, err
}
return ids, nil
}
func (a *API) verifiedImages(ctx context.Context) (map[string]string, error) {
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
return nil, err
}
ids, err := a.successfulImageIDs(ctx)
if err != nil {
return nil, err
}
result := map[string]string{}
for _, id := range ids {
im, err := a.evaluateImage(ctx, id)
if err != nil {
return nil, err
}
if im.Selectable {
result[id] = im.Version
}
}
return result, nil
}
func validBuildVersion(v string) bool { return len(v) <= 100 && fixedVersion.MatchString(v) }
func sanitizeBuildLog(s string) string {
s = buildURLCredentials.ReplaceAllString(s, "$1[REDACTED]@")
s = buildSecrets.ReplaceAllString(s, "$1[REDACTED]")
s = strings.ToValidUTF8(s, "")
if len(s) > buildLogLimit {
s = strings.ToValidUTF8(s[:buildLogLimit], "")
}
return s
}
internal/api/images_delete.go
-145
View File
@@ -1,145 +0,0 @@
package api
import (
"context"
"database/sql"
"errors"
"net/http"
"net/url"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
)
func (a *API) deleteImage(w http.ResponseWriter, r *http.Request) {
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
id := r.URL.Query().Get("image_id")
if !runtimeImageID.MatchString(id) {
fail(w, 400, "invalid image_id")
return
}
// Hold the build queue through the checks and the Docker deletion so a
// record cannot flip to missing mid-flight. Upgrades write their journal
// entry before creating containers; the transaction below re-checks that
// journal, and an upgrade racing an administrator deletion fails safely
// with its own rollback path.
admissionMu.Lock()
defer admissionMu.Unlock()
buildQueue.Lock()
defer buildQueue.Unlock()
if buildQueue.active[a.s.DB] {
fail(w, 409, "image build worker is active")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
defer cancel()
tx, err := a.s.DB.BeginTx(ctx, nil)
if err != nil {
fail(w, 500, "database error")
return
}
defer tx.Rollback()
if _, err = tx.ExecContext(ctx, "UPDATE images SET status=status WHERE image_id=?", id); err != nil {
fail(w, 500, "image index unavailable")
return
}
var version string
err = tx.QueryRowContext(ctx, "SELECT version FROM images WHERE image_id=?", id).Scan(&version)
if errors.Is(err, sql.ErrNoRows) {
fail(w, 404, "managed image not found")
return
}
if err != nil {
fail(w, 500, "image index unavailable")
return
}
for _, check := range []struct{ table, query string }{
{"image_builds", "SELECT count(*) FROM image_builds WHERE status IN ('queued','running','validating')"},
{"upgrade_journal", "SELECT count(*) FROM upgrade_journal WHERE phase NOT IN ('complete','rolled_back')"},
} {
var exists, count int
if err = tx.QueryRowContext(ctx, "SELECT count(*) FROM sqlite_master WHERE type='table' AND name=?", check.table).Scan(&exists); err == nil && exists > 0 {
err = tx.QueryRowContext(ctx, check.query).Scan(&count)
}
if err != nil {
fail(w, 500, "active task lookup failed")
return
}
if count > 0 {
fail(w, 409, "active task prevents image deletion")
return
}
}
var references int
if err = tx.QueryRowContext(ctx, "SELECT count(*) FROM instances WHERE image_id=?", id).Scan(&references); err != nil {
fail(w, 500, "instance reference lookup failed")
return
}
if references > 0 {
fail(w, 409, "image is referenced by an instance")
return
}
var image struct {
ID string `json:"Id"`
Config struct{ Labels map[string]string }
}
path := "/images/" + url.PathEscape(id)
if err = a.docker.Do(ctx, http.MethodGet, path+"/json", nil, &image); err != nil {
fail(w, 502, "image inspection failed")
return
}
if image.ID != id || image.Config.Labels["dsh-mgr.role"] != "runtime" || version == "" || image.Config.Labels["dsh-mgr.version"] != version {
fail(w, 409, "managed runtime image required")
return
}
var containers []struct{ ImageID string }
if err = a.docker.Do(ctx, http.MethodGet, "/containers/json?all=1", nil, &containers); err != nil || containers == nil {
fail(w, 502, "container reference lookup failed")
return
}
for _, container := range containers {
if container.ImageID == "" {
fail(w, 502, "container image identity unavailable")
return
}
if container.ImageID == id {
fail(w, 409, "image is referenced by a container")
return
}
}
if _, err = tx.ExecContext(ctx, "UPDATE images SET status='missing',usage_count=0 WHERE image_id=?", id); err == nil {
_, err = tx.ExecContext(ctx, "UPDATE image_builds SET status='missing',error='image deleted by administrator' WHERE image_id=? AND status='success'", id)
}
if err == nil {
_, err = tx.ExecContext(ctx, "INSERT INTO audit(uid,action,target,detail,at) VALUES(?,'image.delete',?,'',?)", u.ID, id, time.Now().Unix())
}
if err != nil {
fail(w, 500, "image deletion recording failed")
return
}
if err = a.docker.Do(ctx, http.MethodDelete, path+"?force=false", nil, nil); err != nil {
status := http.StatusBadGateway
var apiErr *docker.APIError
if errors.As(err, &apiErr) && apiErr.StatusCode == http.StatusConflict {
status = http.StatusConflict
}
fail(w, status, "docker image deletion failed")
return
}
// Docker deletion is already authoritative, even if the DB commit fails.
// Any inventory observation taken before it must be discarded.
a.imageDeleteGeneration++
if err = tx.Commit(); err != nil {
fail(w, 500, "image deleted but recording failed")
return
}
writeJSON(w, map[string]string{"image_id": id, "status": "missing"})
}
internal/api/images_delete_test.go
-168
View File
@@ -1,168 +0,0 @@
package api
import (
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
)
func TestImageDelete(t *testing.T) {
id := "sha256:" + strings.Repeat("a", 64)
for _, mode := range []string{"success", "anonymous", "non-admin", "invalid", "unmanaged", "role", "instance", "container-running", "container-exited", "containers-failure", "containers-null", "containers-unknown", "inspect-failure", "delete-failure", "delete-conflict", "queued", "running", "validating", "worker", "journal", "complete", "rolled_back", "db-failure", "task-query-failure", "audit-failure"} {
t.Run(mode, func(t *testing.T) {
var deletes atomic.Int32
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodDelete:
deletes.Add(1)
if r.URL.Path != "/v1.47/images/"+id || r.URL.Query().Get("force") != "false" {
t.Errorf("unsafe delete: %s", r.URL)
}
if mode == "delete-failure" {
w.WriteHeader(500)
return
}
if mode == "delete-conflict" {
w.WriteHeader(409)
return
}
writeJSON(w, []any{})
case r.URL.Path == "/v1.47/images/"+id+"/json":
if mode == "inspect-failure" {
w.WriteHeader(500)
return
}
if mode == "role" {
writeJSON(w, map[string]any{"Id": id})
return
}
writeImageInspect(w, id, "1.2.3")
case r.URL.Path == "/v1.47/containers/json":
if r.URL.RawQuery != "all=1" {
t.Errorf("must inspect all containers: %s", r.URL)
}
if mode == "containers-failure" {
w.WriteHeader(500)
return
}
if mode == "containers-null" {
writeJSON(w, nil)
return
}
if mode == "containers-unknown" {
writeJSON(w, []any{map[string]string{"Image": "tag"}})
return
}
if strings.HasPrefix(mode, "container-") {
writeJSON(w, []any{map[string]string{"ImageID": id, "State": strings.TrimPrefix(mode, "container-")}})
return
}
writeJSON(w, []any{})
default:
t.Errorf("unexpected Docker request: %s %s", r.Method, r.URL)
w.WriteHeader(500)
}
})
exec := func(q string, args ...any) {
t.Helper()
if _, err := a.s.DB.Exec(q, args...); err != nil {
t.Fatal(err)
}
}
seedVerifiedImage(t, a, id, "1.2.3")
exec("INSERT INTO images(image_id,version,status,built_at) VALUES(?,'1.2.3','available','now')", id)
if mode != "non-admin" {
exec("UPDATE users SET admin=1 WHERE id='u'")
}
want := 409
switch mode {
case "success", "complete", "rolled_back":
want = 200
case "anonymous":
sid = ""
want = 401
case "non-admin":
want = 403
case "invalid":
want = 400
case "unmanaged":
exec("DELETE FROM images")
want = 404
case "inspect-failure", "containers-failure", "containers-null", "containers-unknown", "delete-failure":
want = 502
case "instance":
exec("INSERT INTO instances(id,uid,name,image_id,created) VALUES('i','other','i',?,0)", id)
case "queued", "running", "validating":
exec("INSERT INTO image_builds(id,version,status,created_at) VALUES('active','2.0.0',?,'now')", mode)
case "worker":
buildQueue.Lock()
buildQueue.active[a.s.DB] = true
buildQueue.Unlock()
defer func() { buildQueue.Lock(); delete(buildQueue.active, a.s.DB); buildQueue.Unlock() }()
case "db-failure":
exec("ALTER TABLE instances RENAME COLUMN image_id TO broken")
want = 500
case "task-query-failure":
exec("CREATE TABLE upgrade_journal(broken TEXT)")
want = 500
case "audit-failure":
exec("CREATE TRIGGER deny_audit BEFORE INSERT ON audit BEGIN SELECT RAISE(FAIL,'unavailable'); END")
want = 500
}
if mode == "journal" || mode == "complete" || mode == "rolled_back" {
exec("CREATE TABLE upgrade_journal(phase TEXT)")
exec("INSERT INTO upgrade_journal VALUES(?)", mode)
}
queryID := id
if mode == "invalid" {
queryID = "runtime:latest"
}
r := httptest.NewRequest(http.MethodDelete, "/api/images?image_id="+queryID, nil)
if sid != "" {
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: sid})
addTestAccess(r, sid)
}
w := httptest.NewRecorder()
a.images(w, r)
if w.Code != want {
t.Fatalf("status=%d want=%d body=%s", w.Code, want, w.Body.String())
}
wantDeletes := int32(0)
if want == 200 || mode == "delete-failure" || mode == "delete-conflict" {
wantDeletes = 1
}
if deletes.Load() != wantDeletes {
t.Fatalf("deletes=%d want=%d", deletes.Load(), wantDeletes)
}
if mode != "unmanaged" {
var status string
if err := a.s.DB.QueryRow("SELECT status FROM images WHERE image_id=?", id).Scan(&status); err != nil {
t.Fatal(err)
}
wantStatus := "available"
if want == 200 {
wantStatus = "missing"
}
if status != wantStatus {
t.Fatalf("image status=%s want=%s", status, wantStatus)
}
}
var audits int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='image.delete' AND uid='u' AND target=?", id).Scan(&audits); err != nil {
t.Fatal(err)
}
if (audits == 1) != (want == 200) {
t.Fatalf("audits=%d", audits)
}
var status string
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE image_id=?", id).Scan(&status); err != nil {
t.Fatal(err)
}
if (status == "missing") != (want == 200) {
t.Fatalf("build status=%s", status)
}
})
}
}
internal/api/images_enqueue_concurrency_test.go
-129
View File
@@ -1,129 +0,0 @@
package api
import (
"net/http"
"sync"
"testing"
"time"
)
func TestEnqueueReleaseConcurrentMissingImageReturnsReplacement(t *testing.T) {
inspections := make(chan chan struct{}, 2)
stop := make(chan struct{})
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
t.Errorf("unexpected Docker method: %s", r.Method)
w.WriteHeader(http.StatusInternalServerError)
return
}
switch r.URL.Path {
case "/v1.47/images/json":
writeJSON(w, []map[string]any{{"Id": "old-image", "Labels": map[string]string{
"dsh-mgr.role": "runtime", "dsh-mgr.version": "0.2.0-rc.1", "dsh-mgr.recipe": imageRecipeRev,
}}})
case "/v1.47/images/old-image/json":
release := make(chan struct{})
select {
case inspections <- release:
case <-stop:
return
}
select {
case <-release:
w.WriteHeader(http.StatusNotFound)
case <-stop:
case <-r.Context().Done():
}
default:
t.Errorf("unexpected Docker path: %s", r.URL.Path)
w.WriteHeader(http.StatusInternalServerError)
}
})
seedVerifiedImage(t, a, "old-image", "0.2.0-rc.1")
info := releaseFixture()
if _, err := a.s.DB.Exec("UPDATE image_builds SET release_id=?,source_repository=?,source_commit=?,version=? WHERE image_id='old-image'", info.ReleaseID, info.Repository, info.Commit, info.Version); err != nil {
t.Fatal(err)
}
var oldID string
if err := a.s.DB.QueryRow("SELECT id FROM image_builds WHERE image_id='old-image'").Scan(&oldID); err != nil {
t.Fatal(err)
}
// Keep the replacement queued without starting a background worker.
buildQueue.Lock()
buildQueue.active[a.s.DB] = true
buildQueue.Unlock()
var submissions sync.WaitGroup
t.Cleanup(func() {
close(stop)
submissions.Wait()
buildQueue.Lock()
delete(buildQueue.active, a.s.DB)
delete(buildQueue.initialized, a.s.DB)
buildQueue.Unlock()
})
type result struct {
build buildRecord
err error
}
submit := func() <-chan result {
done := make(chan result, 1)
submissions.Add(1)
go func() {
defer submissions.Done()
b, err := a.enqueueRelease("u", info)
done <- result{b, err}
}()
return done
}
waitInspection := func() chan struct{} {
t.Helper()
select {
case release := <-inspections:
return release
case <-time.After(3 * time.Second):
t.Fatal("submission did not inspect the old image")
return nil
}
}
waitResult := func(done <-chan result) buildRecord {
t.Helper()
select {
case got := <-done:
if got.err != nil {
t.Fatal(got.err)
}
return got.build
case <-time.After(3 * time.Second):
t.Fatal("submission did not finish")
return buildRecord{}
}
}
firstDone := submit()
firstRelease := waitInspection()
secondDone := submit()
secondRelease := waitInspection()
close(firstRelease)
first := waitResult(firstDone)
if first.ID == "" || first.ID == oldID || first.Status != "queued" {
t.Fatalf("first submission did not queue a replacement: %+v", first)
}
close(secondRelease)
second := waitResult(secondDone)
if second.ID != first.ID || second.Status != "queued" {
t.Errorf("second submission returned id=%q status=%q; want replacement id=%q status=queued", second.ID, second.Status, first.ID)
}
var status string
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE id=?", oldID).Scan(&status); err != nil || status != "missing" {
t.Errorf("old build status=%q, err=%v; want missing", status, err)
}
for query, want := range map[string]int{
"SELECT count(*) FROM image_builds": 2,
"SELECT count(*) FROM image_builds WHERE status='queued'": 1,
"SELECT count(*) FROM image_build_requests": 1,
} {
var count int
if err := a.s.DB.QueryRow(query).Scan(&count); err != nil || count != want {
t.Errorf("%s: count=%d, err=%v; want %d", query, count, err, want)
}
}
}
internal/api/images_index_concurrency_test.go
-137
View File
@@ -1,137 +0,0 @@
package api
import (
"context"
"errors"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func TestImageIndexDoesNotUndoConcurrentDelete(t *testing.T) {
for _, initial := range []string{"available", "missing"} {
t.Run(initial, func(t *testing.T) {
id := "sha256:" + strings.Repeat("a", 64)
blocker := "sha256:" + strings.Repeat("b", 64)
reached, release := make(chan struct{}), make(chan struct{})
var once sync.Once
var reachedOnce sync.Once
var deleted atomic.Bool
unblock := func() { once.Do(func() { close(release) }) }
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == http.MethodDelete:
deleted.Store(true)
writeJSON(w, []any{})
case r.URL.Path == "/v1.47/images/"+id+"/json":
if deleted.Load() {
w.WriteHeader(http.StatusNotFound)
return
}
writeImageInspect(w, id, "1.2.3")
case r.URL.Path == "/v1.47/images/"+blocker+"/json":
reachedOnce.Do(func() { close(reached) })
select {
case <-release:
case <-r.Context().Done():
return
}
w.WriteHeader(http.StatusNotFound)
case r.URL.Path == "/v1.47/images/json", r.URL.Path == "/v1.47/containers/json":
writeJSON(w, []any{})
default:
t.Errorf("unexpected Docker request: %s %s", r.Method, r.URL)
w.WriteHeader(500)
}
})
defer unblock()
seedVerifiedImage(t, a, id, "1.2.3")
for _, q := range []struct {
sql string
args []any
}{
{"UPDATE users SET admin=1 WHERE id='u'", nil},
{"INSERT INTO images(image_id,version,status,built_at) VALUES(?,'1.2.3',?,'2'),(?,'2.0.0','missing','1')", []any{id, initial, blocker}},
} {
if _, err := a.s.DB.Exec(q.sql, q.args...); err != nil {
t.Fatal(err)
}
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
type outcome struct {
images []imageRecord
err error
}
done := make(chan outcome, 1)
go func() { images, err := a.imageIndex(ctx); done <- outcome{images, err} }()
select {
case <-reached:
case <-ctx.Done():
t.Fatal("index did not reach second image")
}
r := httptest.NewRequest(http.MethodDelete, "/api/images?image_id="+id, nil).WithContext(ctx)
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: sid})
addTestAccess(r, sid)
w := httptest.NewRecorder()
a.images(w, r)
if w.Code != 200 {
t.Fatalf("delete: %d %s", w.Code, w.Body.String())
}
unblock()
select {
case out := <-done:
if out.err != nil {
t.Fatal(out.err)
}
matches := 0
for _, im := range out.images {
if im.ImageID == id {
matches++
}
}
if matches != 0 {
t.Errorf("returned %d records for deleted image, want 0", matches)
}
case <-ctx.Done():
t.Fatal("index did not finish")
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM images WHERE image_id=?", id).Scan(&count); err != nil {
t.Fatal(err)
}
if count != 0 {
t.Errorf("persisted %d rows for deleted image", count)
}
})
}
}
func TestImageIndexBoundsInvalidatedSnapshots(t *testing.T) {
var a *API
var calls atomic.Int32
a, _ = imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1.47/images/json" {
t.Errorf("unexpected request: %s", r.URL)
w.WriteHeader(500)
return
}
// Model a completed deletion during every unlocked observation phase.
buildQueue.Lock()
a.imageDeleteGeneration++
buildQueue.Unlock()
calls.Add(1)
writeJSON(w, []any{})
})
images, err := a.imageIndex(context.Background())
if !errors.Is(err, errImageIndexChanged) || images != nil {
t.Fatalf("images=%v err=%v; want no inventory and invalidation error", images, err)
}
if calls.Load() != 3 {
t.Fatalf("attempts=%d, want 3", calls.Load())
}
}
internal/api/images_policy.go
-285
View File
@@ -1,285 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"net/http"
"net/url"
"os"
"path/filepath"
"runtime"
"syscall"
"time"
)
const imageRecipeRev = "7"
var imagePlatform = runtime.GOOS + "/" + runtime.GOARCH
const minimumBuildDiskBytes = uint64(2 << 30)
var buildDiskAvailable = func(path string) (uint64, error) {
var stat syscall.Statfs_t
if err := syscall.Statfs(path, &stat); err != nil {
return 0, err
}
return uint64(stat.Bavail) * uint64(stat.Bsize), nil
}
type inspectedImage struct {
ID string `json:"Id"`
Os, Architecture, Variant, Created string
RepoTags []string
Config struct{ Labels map[string]string }
}
func (im inspectedImage) platform() string {
if im.Os == "" || im.Architecture == "" {
return ""
}
p := im.Os + "/" + im.Architecture
// Docker can report the default ARM64 variant explicitly. Keep its
// identity consistent with build records and admission keys; preserve
// all other variants so incompatible images remain rejected.
if im.Architecture == "arm64" && im.Variant == "v8" {
return p
}
if im.Variant != "" {
p += "/" + im.Variant
}
return p
}
func (a *API) inspectLocalImage(ctx context.Context, id string) (*inspectedImage, error) {
if id == "" {
return nil, nil
}
var im inspectedImage
if err := a.docker.Do(ctx, http.MethodGet, "/images/"+url.PathEscape(id)+"/json", nil, &im); err != nil {
var apiErr *docker.APIError
if errors.As(err, &apiErr) && apiErr.StatusCode == http.StatusNotFound {
return nil, nil
}
return nil, err
}
return &im, nil
}
func (a *API) imageMatches(ctx context.Context, id, version string) (bool, error) {
im, err := a.inspectLocalImage(ctx, id)
if err != nil || im == nil {
return false, err
}
return im.ID == id && im.platform() == imagePlatform && im.Config.Labels["dsh-mgr.role"] == "runtime" && im.Config.Labels["dsh-mgr.version"] == version && im.Config.Labels["dsh-mgr.recipe"] == imageRecipeRev, nil
}
var errImageIndexChanged = errors.New("image inventory changed during reconciliation")
func (a *API) imageIndex(ctx context.Context) ([]imageRecord, error) {
// Retry stale observations without holding the queue across Docker I/O.
// Bound retries so sustained deletions cannot keep a request alive forever.
for attempt := 0; attempt < 3; attempt++ {
result, err := a.imageIndexSnapshot(ctx)
if !errors.Is(err, errImageIndexChanged) {
return result, err
}
}
return nil, errImageIndexChanged
}
func (a *API) imageIndexSnapshot(ctx context.Context) ([]imageRecord, error) {
// Phase 1: read the managed index under the queue lock (DB only).
buildQueue.Lock()
generation := a.imageDeleteGeneration
err := a.ensureBuilds()
if err == nil {
_, err = a.s.DB.ExecContext(ctx, `INSERT OR IGNORE INTO images(image_id,version,recipe_rev,platform,tag,status,built_at)
SELECT image_id,version,recipe_rev,platform,'','missing',COALESCE(finished_at,created_at) FROM image_builds WHERE status='success' AND image_id<>'' ORDER BY created_at DESC,id DESC`)
}
var rows *sql.Rows
if err == nil {
rows, err = a.s.DB.QueryContext(ctx, `SELECT image_id,version,COALESCE(recipe_rev,''),COALESCE(platform,''),tag,status,built_at,
(SELECT count(*) FROM instances WHERE instances.image_id=images.image_id) FROM images ORDER BY built_at DESC,image_id`)
}
result := []imageRecord{}
if err == nil {
for rows.Next() {
var im imageRecord
if err = rows.Scan(&im.ImageID, &im.Version, &im.RecipeRev, &im.Platform, &im.Tag, &im.Status, &im.Created, &im.UsageCount); err != nil {
break
}
im.ID = im.ImageID
result = append(result, im)
}
scanErr := rows.Err()
rows.Close()
if err == nil {
err = scanErr
}
}
buildQueue.Unlock()
if err != nil {
return nil, err
}
// Phase 2: inspect Docker without the queue lock; per-image state comes
// from real daemon observations, and instance creation must not wait here.
known := make(map[string]bool, len(result))
for _, im := range result {
known[im.ImageID] = true
}
var local []struct {
ID string
Labels map[string]string
}
{
filters, _ := json.Marshal(map[string][]string{"label": {"dsh-mgr.role=runtime"}})
if err := a.docker.Do(ctx, http.MethodGet, "/images/json?filters="+url.QueryEscape(string(filters)), nil, &local); err != nil {
return nil, err
}
}
for _, im := range local {
if im.ID != "" && !known[im.ID] && im.Labels["dsh-mgr.role"] == "runtime" {
known[im.ID] = true
result = append(result, imageRecord{ID: im.ID, ImageID: im.ID})
}
}
for i := range result {
im := &result[i]
im.Status = "missing"
im.Admission = "existing_only"
im.AdmissionReason = "local image missing"
actual, err := a.inspectLocalImage(ctx, im.ImageID)
if err != nil {
return nil, err
}
if actual == nil {
continue
}
im.Status = "unverified"
im.Platform = actual.platform()
im.Version = actual.Config.Labels["dsh-mgr.version"]
im.RecipeRev = actual.Config.Labels["dsh-mgr.recipe"]
im.Tag = ""
if len(actual.RepoTags) > 0 {
im.Tag = actual.RepoTags[0]
}
if actual.Created != "" {
im.Created = actual.Created
}
if err := a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM instances WHERE image_id=?", im.ImageID).Scan(&im.UsageCount); err != nil {
return nil, err
}
policy, err := a.evaluateImage(ctx, im.ImageID)
if err != nil {
return nil, err
}
im.Status, im.Admission, im.AdmissionReason, im.Selectable = policy.Status, policy.Admission, policy.AdmissionReason, policy.Selectable
}
// Phase 3: write observed state back under the queue lock (DB only).
buildQueue.Lock()
defer buildQueue.Unlock()
if generation != a.imageDeleteGeneration {
return nil, errImageIndexChanged
}
tx, err := a.s.DB.BeginTx(ctx, nil)
if err != nil {
return nil, err
}
defer tx.Rollback()
present := make([]imageRecord, 0, len(result))
for _, im := range result {
if im.Status == "missing" {
if _, err = tx.ExecContext(ctx, "UPDATE image_builds SET status='missing',error='image no longer available or template identity is unknown' WHERE image_id=? AND status='success'", im.ImageID); err != nil {
return nil, err
}
if _, err = tx.ExecContext(ctx, "DELETE FROM images WHERE image_id=?", im.ImageID); err != nil {
return nil, err
}
continue
}
if _, err = tx.ExecContext(ctx, `INSERT INTO images(image_id,version,recipe_rev,platform,tag,status,built_at,usage_count) VALUES(?,?,?,?,?,?,?,?)
ON CONFLICT(image_id) DO UPDATE SET version=excluded.version,recipe_rev=excluded.recipe_rev,platform=excluded.platform,tag=excluded.tag,status=excluded.status,built_at=excluded.built_at,usage_count=excluded.usage_count`, im.ImageID, im.Version, im.RecipeRev, im.Platform, im.Tag, im.Status, im.Created, im.UsageCount); err != nil {
return nil, err
}
present = append(present, im)
}
if err = tx.Commit(); err != nil {
return nil, err
}
return present, nil
}
func (a *API) StartImageBuilds() error {
buildQueue.Lock()
defer buildQueue.Unlock()
if err := a.ensureBuilds(); err != nil {
return err
}
a.startBuildWorkerLocked()
return nil
}
func (a *API) checkBuildDisk(ctx context.Context) error {
var info struct{ DockerRootDir string }
if err := a.docker.Do(ctx, http.MethodGet, "/info", nil, &info); err != nil {
return fmt.Errorf("build disk check: %w", err)
}
// A matching pathname alone cannot establish daemon filesystem identity.
confirmed := os.Getenv("DSH_DOCKER_ROOT_VISIBLE")
if !filepath.IsAbs(info.DockerRootDir) || confirmed != info.DockerRootDir {
return errors.New("Docker disk capacity unknown: mount the daemon DockerRootDir at the identical path and set DSH_DOCKER_ROOT_VISIBLE to that path; new builds are blocked")
}
available, err := buildDiskAvailable(confirmed)
if err != nil {
return fmt.Errorf("Docker disk capacity unavailable: %w", err)
}
return requireBuildDisk(available)
}
func requireBuildDisk(available uint64) error {
if available < minimumBuildDiskBytes {
return fmt.Errorf("insufficient Docker disk space: %d available bytes, require %d", available, minimumBuildDiskBytes)
}
return nil
}
func (a *API) insertBuildRequest(b buildRecord, user string) error {
tx, err := a.s.DB.Begin()
if err != nil {
return err
}
defer tx.Rollback()
if _, err = tx.Exec("INSERT INTO image_builds(id,version,status,created_at,requested_by,recipe_rev,platform,experimental,release_id,source_repository,source_commit) VALUES(?,?,?,?,?,?,?,?,?,?,?)", b.ID, b.Version, b.Status, b.Created, b.RequestedBy, b.RecipeRev, b.Platform, b.Experimental, b.ReleaseID, b.SourceRepository, b.SourceCommit); err != nil {
return err
}
if _, err = tx.Exec("INSERT INTO image_build_requests(user_id,build_id,created_at) VALUES(?,?,?)", user, b.ID, b.Created); err != nil {
return err
}
return tx.Commit()
}
func (a *API) reconcileBuilds(parent context.Context, records []buildRecord) error {
ctx, cancel := context.WithTimeout(parent, 15*time.Second)
defer cancel()
for i := range records {
b := &records[i]
if b.Status != "success" {
continue
}
actual, err := a.inspectLocalImage(ctx, b.ImageID)
if err != nil {
return err
}
if actual == nil || actual.ID != b.ImageID || actual.Config.Labels["dsh-mgr.role"] != "runtime" || actual.Config.Labels["dsh-mgr.version"] != b.Version || actual.Config.Labels["dsh-mgr.recipe"] != b.RecipeRev || actual.platform() != b.Platform || !releaseLabelsMatch(actual, *b) {
b.Status, b.Error = "missing", "image no longer available"
if _, err := a.s.DB.Exec("UPDATE image_builds SET status=?,error=? WHERE id=? AND status='success'", b.Status, b.Error, b.ID); err != nil {
return err
}
}
}
return nil
}
internal/api/images_policy_test.go
-410
View File
@@ -1,410 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"io"
"net/http"
"os"
"strings"
"sync/atomic"
"testing"
)
func TestInspectedImagePlatform(t *testing.T) {
for _, tc := range []struct{ os, arch, variant, want string }{
{"linux", "arm64", "", "linux/arm64"},
{"linux", "arm64", "v8", "linux/arm64"},
{"linux", "arm64", "v9", "linux/arm64/v9"},
{"linux", "arm64", "v8.1", "linux/arm64/v8.1"},
{"linux", "arm", "v7", "linux/arm/v7"},
{"linux", "arm", "v8", "linux/arm/v8"},
{"linux", "amd64", "", "linux/amd64"},
{"linux", "amd64", "v8", "linux/amd64/v8"},
{"", "arm64", "v8", ""},
{"linux", "", "v8", ""},
} {
t.Run(tc.os+"/"+tc.arch+"/"+tc.variant, func(t *testing.T) {
im := inspectedImage{Os: tc.os, Architecture: tc.arch, Variant: tc.variant}
if got := im.platform(); got != tc.want {
t.Fatalf("platform() = %q, want %q", got, tc.want)
}
})
}
}
func TestImageLegacyMigrationNeverCertifies(t *testing.T) {
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/json" {
io.WriteString(w, "[]")
return
}
writeImageInspect(w, strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/v1.47/images/"), "/json"), "0.2.0-rc.1")
})
_, err := a.s.DB.Exec(`CREATE TABLE image_builds(id TEXT PRIMARY KEY,version TEXT NOT NULL,status TEXT NOT NULL,logs TEXT NOT NULL DEFAULT '',error TEXT NOT NULL DEFAULT '',image_id TEXT NOT NULL DEFAULT '',created_at TEXT NOT NULL);
CREATE UNIQUE INDEX image_builds_version ON image_builds(version) WHERE status IN ('queued','running','validating','success');
INSERT INTO image_builds(id,version,status,image_id,created_at) VALUES('legacy','0.2.0-rc.1','success','legacy-image','now'),('queued','0.2.0-rc.2','queued','','now');
CREATE TABLE images(image_id TEXT PRIMARY KEY,version TEXT NOT NULL,tag TEXT NOT NULL DEFAULT '',status TEXT NOT NULL,built_at TEXT NOT NULL);
INSERT INTO images VALUES('legacy-image','0.2.0-rc.1','','available','now');`)
if err != nil {
t.Fatal(err)
}
verified, err := a.verifiedImages(context.Background())
if err != nil || len(verified) != 0 {
t.Fatalf("%v %v", verified, err)
}
var recipe, platform sql.NullString
if err = a.s.DB.QueryRow("SELECT recipe_rev,platform FROM image_builds WHERE id='legacy'").Scan(&recipe, &platform); err != nil || recipe.Valid || platform.Valid {
t.Fatalf("legacy metadata inferred: %v %v %v", recipe, platform, err)
}
w := imageRequest(a.images, sid, "GET", "")
var images []imageRecord
if err = json.Unmarshal(w.Body.Bytes(), &images); err != nil || len(images) != 1 || images[0].Status != "unverified" || images[0].RecipeRev != imageRecipeRev || images[0].Platform != imagePlatform {
t.Fatal(w.Body.String(), err)
}
if err = a.s.DB.QueryRow("SELECT recipe_rev,platform FROM image_builds WHERE id='legacy'").Scan(&recipe, &platform); err != nil || recipe.Valid || platform.Valid {
t.Fatalf("GET certified legacy metadata: %v %v %v", recipe, platform, err)
}
var status string
if err = a.s.DB.QueryRow("SELECT status FROM image_builds WHERE id='queued'").Scan(&status); err != nil || status != "failed" {
t.Fatal(status, err)
}
seedVerifiedImage(t, a, "new-image", "0.2.0-rc.1")
verified, err = a.verifiedImages(context.Background())
if err != nil || len(verified) != 1 || verified["new-image"] != "0.2.0-rc.1" {
t.Fatal(verified, err)
}
}
func TestImageInspectIdentity(t *testing.T) {
for _, mode := range []string{"valid", "recipe", "wrong-key", "platform", "variant", "unknown-platform", "role", "version", "id", "missing", "unavailable"} {
t.Run(mode, func(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
t.Fatal("unexpected mutation")
}
if mode == "missing" {
w.WriteHeader(404)
return
}
if mode == "unavailable" {
w.WriteHeader(500)
return
}
parts := strings.Split(imagePlatform, "/")
labels := map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "0.2.0-rc.1", "dsh-mgr.recipe": imageRecipeRev}
im := map[string]any{"Id": "image", "Os": parts[0], "Architecture": parts[1], "Config": map[string]any{"Labels": labels}}
switch mode {
case "recipe":
labels["dsh-mgr.recipe"] = "old"
case "wrong-key":
delete(labels, "dsh-mgr.recipe")
labels["dsh-mgr.recipe_rev"] = imageRecipeRev
case "platform":
im["Architecture"] = "other"
case "variant":
im["Variant"] = "v7"
case "unknown-platform":
delete(im, "Architecture")
case "role":
labels["dsh-mgr.role"] = "other"
case "version":
labels["dsh-mgr.version"] = "2.0.0"
case "id":
im["Id"] = "other"
}
writeJSON(w, im)
})
ok, err := a.imageMatches(context.Background(), "image", "0.2.0-rc.1")
if ok != (mode == "valid") || (err != nil) != (mode == "unavailable") {
t.Fatal(ok, err)
}
})
}
}
func TestImageIndexLocalInventory(t *testing.T) {
for _, mode := range []string{"external", "old-recipe", "other-platform", "inspect-failure"} {
t.Run(mode, func(t *testing.T) {
var deleted atomic.Bool
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
t.Errorf("unexpected Docker mutation: %s", r.Method)
}
if r.URL.Path == "/v1.47/images/json" {
if deleted.Load() {
io.WriteString(w, "[]")
} else {
io.WriteString(w, `[{"Id":"image","Labels":{"dsh-mgr.role":"runtime","dsh-mgr.recipe":"2","dsh-mgr.version":"9.9.9"}}]`)
}
return
}
if mode == "inspect-failure" {
w.WriteHeader(500)
return
}
if deleted.Load() {
w.WriteHeader(404)
return
}
parts := strings.Split(imagePlatform, "/")
im := inspectedImage{ID: "image", Os: parts[0], Architecture: parts[1], Created: "2026-01-01T00:00:00Z", RepoTags: []string{"dsh-web:0.2.0-rc.1"}}
im.Config.Labels = map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "0.2.0-rc.1", "dsh-mgr.recipe": imageRecipeRev}
if mode == "old-recipe" {
im.Config.Labels["dsh-mgr.recipe"] = "1"
}
if mode == "other-platform" {
im.Architecture, im.Variant = "arm", "v7"
}
writeJSON(w, im)
})
if mode != "external" {
seedVerifiedImage(t, a, "image", "0.2.0-rc.1")
}
if mode == "old-recipe" {
if _, err := a.s.DB.Exec("UPDATE image_builds SET recipe_rev='1'"); err != nil {
t.Fatal(err)
}
}
w := imageRequest(a.images, sid, "GET", "")
if mode == "inspect-failure" {
var status string
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE image_id='image'").Scan(&status); err != nil || status != "success" || w.Code != 502 {
t.Fatal(status, w.Code, err)
}
return
}
wantStatus := "unverified"
if mode == "old-recipe" {
wantStatus = "existing_only"
}
var images []imageRecord
if err := json.Unmarshal(w.Body.Bytes(), &images); err != nil || w.Code != 200 || len(images) != 1 || images[0].Status != wantStatus || images[0].Version != "0.2.0-rc.1" || images[0].Created != "2026-01-01T00:00:00Z" || images[0].Tag != "dsh-web:0.2.0-rc.1" {
t.Fatal(w.Body.String(), err)
}
if mode == "other-platform" && images[0].Platform != strings.Split(imagePlatform, "/")[0]+"/arm/v7" {
t.Fatal(images[0].Platform)
}
if mode == "old-recipe" && images[0].RecipeRev != "1" {
t.Fatal(images[0].RecipeRev)
}
if w := imageRequest(a.instances, sid, "POST", `{"name":"rejected","image_id":"image"}`); w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
deleted.Store(true)
w = imageRequest(a.images, sid, "GET", "")
if err := json.Unmarshal(w.Body.Bytes(), &images); err != nil || w.Code != 200 || len(images) != 0 {
t.Fatal(w.Body.String(), err)
}
})
}
}
func TestImageIndexUsageMissingAndDockerFailure(t *testing.T) {
var mode atomic.Int32
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
t.Error("unexpected mutation")
}
if r.URL.Path == "/v1.47/images/json" && mode.Load() != 1 {
io.WriteString(w, "[]")
return
}
switch mode.Load() {
case 1:
w.WriteHeader(500)
case 2:
w.WriteHeader(404)
default:
writeImageInspect(w, "image", "0.2.0-rc.1")
}
})
seedVerifiedImage(t, a, "image", "0.2.0-rc.1")
if _, err := a.s.DB.Exec(`INSERT INTO instances(id,uid,name,image_id,status,created) VALUES('one','u','one','image','running',0),('two','other-user','two','image','stopped',0),('three','u','three','other','running',0)`); err != nil {
t.Fatal(err)
}
check := func(status string, count int) {
t.Helper()
w := imageRequest(a.images, sid, "GET", "")
var images []imageRecord
if err := json.Unmarshal(w.Body.Bytes(), &images); err != nil || w.Code != 200 || len(images) != 1 || images[0].Status != status || images[0].UsageCount != count {
t.Fatal(w.Body.String(), err)
}
var storedStatus string
var storedCount int
if err := a.s.DB.QueryRow("SELECT status,usage_count FROM images WHERE image_id='image'").Scan(&storedStatus, &storedCount); err != nil || storedStatus != status || storedCount != count {
t.Fatal(storedStatus, storedCount, err)
}
}
check("available", 2)
mode.Store(1)
if w := imageRequest(a.images, sid, "GET", ""); w.Code != 502 {
t.Fatal(w.Code, w.Body.String())
}
var status string
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE image_id='image'").Scan(&status); err != nil || status != "success" {
t.Fatal(status, err)
}
mode.Store(2)
for refresh := 0; refresh < 3; refresh++ {
w := imageRequest(a.images, sid, "GET", "")
if w.Code != 200 || strings.TrimSpace(w.Body.String()) != "[]" {
t.Fatal(w.Code, w.Body.String())
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM images").Scan(&count); err != nil || count != 0 {
t.Fatal(count, err)
}
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM instances WHERE image_id='image'").Scan(&count); err != nil || count != 2 {
t.Fatal(count, err)
}
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE image_id='image'").Scan(&status); err != nil || status != "missing" {
t.Fatal(status, err)
}
verified, err := a.verifiedImages(context.Background())
if err != nil || len(verified) != 0 {
t.Fatal(verified, err)
}
}
func TestImageIndexFailurePreservesRows(t *testing.T) {
for _, mode := range []string{"list", "inspect", "rollback"} {
t.Run(mode, func(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/json" {
if mode == "list" {
w.WriteHeader(500)
} else {
writeJSON(w, []any{})
}
return
}
if mode == "inspect" && strings.Contains(r.URL.Path, "second") {
w.WriteHeader(500)
return
}
w.WriteHeader(404)
})
seedVerifiedImage(t, a, "first", "1.0.0")
seedVerifiedImage(t, a, "second", "2.0.0")
if _, err := a.s.DB.Exec("INSERT INTO images(image_id,version,status,built_at) VALUES('first','1.0.0','available','2'),('second','2.0.0','available','1')"); err != nil {
t.Fatal(err)
}
if mode == "rollback" {
if _, err := a.s.DB.Exec("CREATE TRIGGER deny_second_delete BEFORE DELETE ON images WHEN OLD.image_id='second' BEGIN SELECT RAISE(FAIL,'unavailable'); END"); err != nil {
t.Fatal(err)
}
}
if images, err := a.imageIndex(context.Background()); err == nil || images != nil {
t.Fatalf("images=%v err=%v", images, err)
}
for _, query := range []string{"SELECT count(*) FROM images WHERE status='available'", "SELECT count(*) FROM image_builds WHERE status='success'"} {
var count int
if err := a.s.DB.QueryRow(query).Scan(&count); err != nil || count != 2 {
t.Fatal(query, count, err)
}
}
})
}
}
func TestImageIndexMissingHistoryAndBlockedReimport(t *testing.T) {
var present atomic.Bool
var inspections atomic.Int32
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/json" {
if present.Load() {
writeJSON(w, []any{map[string]any{"Id": "image", "Labels": map[string]string{"dsh-mgr.role": "runtime"}}})
} else {
writeJSON(w, []any{})
}
return
}
inspections.Add(1)
if !present.Load() {
w.WriteHeader(404)
return
}
writeImageInspect(w, "image", "0.2.0-rc.1")
})
seedVerifiedImage(t, a, "image", "0.2.0-rc.1")
if _, err := a.s.DB.Exec("UPDATE image_admissions SET state='blocked',evidence='unsafe'"); err != nil {
t.Fatal(err)
}
for refresh := 0; refresh < 3; refresh++ {
if images, err := a.imageIndex(context.Background()); err != nil || len(images) != 0 {
t.Fatal(images, err)
}
}
if inspections.Load() != 1 {
t.Fatalf("missing history caused %d inspections", inspections.Load())
}
present.Store(true)
images, err := a.imageIndex(context.Background())
if err != nil || len(images) != 1 || images[0].Status != "blocked" || images[0].Selectable || images[0].AdmissionReason != "unsafe" {
t.Fatal(images, err)
}
var status string
if err := a.s.DB.QueryRow("SELECT status FROM image_builds WHERE image_id='image'").Scan(&status); err != nil || status != "missing" {
t.Fatal(status, err)
}
}
func TestBuildDiskPolicy(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected build") })
if err := a.checkBuildDisk(context.Background()); err != nil {
t.Fatal(err)
}
probeErr := errors.New("capacity probe failed")
for _, tc := range []struct {
name string
available uint64
err error
want string
}{
{"insufficient", minimumBuildDiskBytes - 1, nil, "insufficient"},
{"exact", minimumBuildDiskBytes, nil, ""},
{"probe-error", 0, probeErr, "unavailable"},
} {
t.Run(tc.name, func(t *testing.T) {
buildDiskAvailable = func(path string) (uint64, error) {
if path != os.Getenv("DSH_DOCKER_ROOT_VISIBLE") {
t.Fatalf("unexpected probe path %q", path)
}
return tc.available, tc.err
}
err := a.checkBuildDisk(context.Background())
if tc.want == "" && err != nil || tc.want != "" && (err == nil || !strings.Contains(err.Error(), tc.want)) {
t.Fatal(err)
}
if tc.err != nil && !errors.Is(err, tc.err) {
t.Fatalf("probe error not preserved: %v", err)
}
})
}
buildDiskAvailable = func(string) (uint64, error) {
t.Fatal("unconfirmed Docker root must not be probed")
return 0, nil
}
t.Setenv("DSH_DOCKER_ROOT_VISIBLE", "")
if err := a.checkBuildDisk(context.Background()); err == nil || !strings.Contains(err.Error(), "unknown") {
t.Fatal(err)
}
if requireBuildDisk(minimumBuildDiskBytes-1) == nil || requireBuildDisk(minimumBuildDiskBytes) != nil {
t.Fatal("disk threshold")
}
}
func TestBuildListReconcilesMissingAndLatestTimestamp(t *testing.T) {
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(404) })
seedVerifiedImage(t, a, "deleted", "0.2.0-rc.1")
w := imageRequest(a.builds, sid, "GET", "")
var records []buildRecord
if err := json.Unmarshal(w.Body.Bytes(), &records); err != nil || len(records) != 1 || records[0].Status != "missing" {
t.Fatal(w.Body.String())
}
}
internal/api/images_test.go
-253
View File
@@ -1,253 +0,0 @@
package api
import (
"context"
"encoding/json"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"sync/atomic"
"testing"
"time"
)
func imageTestAPI(t *testing.T, h http.HandlerFunc) (*API, string) {
t.Helper()
oldDiskAvailable := buildDiskAvailable
buildDiskAvailable = func(string) (uint64, error) { return minimumBuildDiskBytes, nil }
t.Cleanup(func() { buildDiskAvailable = oldDiskAvailable })
root := t.TempDir()
t.Setenv("DSH_DOCKER_ROOT_VISIBLE", root)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/info" {
writeJSON(w, map[string]string{"DockerRootDir": root})
return
}
h(w, r)
}))
t.Cleanup(srv.Close)
s, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatal(err)
}
s.DB.SetMaxOpenConns(1)
a := New(s, t.TempDir())
a.docker = docker.New(srv.URL)
t.Cleanup(func() {
deadline := time.Now().Add(5 * time.Second)
for {
buildQueue.Lock()
active := buildQueue.active[s.DB]
buildQueue.Unlock()
if !active {
break
}
if time.Now().After(deadline) {
t.Error("worker did not finish")
break
}
time.Sleep(time.Millisecond)
}
s.Close()
})
if err = s.CreateUser(store.User{ID: "u", Username: "user"}, "unused"); err != nil {
t.Fatal(err)
}
sid, _, err := s.Session("u")
if err != nil {
t.Fatal(err)
}
registerTestAccess(t, a, sid)
return a, sid
}
func imageRequest(h http.HandlerFunc, sid, method, body string) *httptest.ResponseRecorder {
r := httptest.NewRequest(method, "/", strings.NewReader(body))
if sid != "" {
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: sid})
addTestAccess(r, sid)
}
w := httptest.NewRecorder()
h(w, r)
return w
}
func TestImageListContract(t *testing.T) {
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/unverified/json" {
writeImageInspect(w, "unverified", "0.2.0-rc.1")
return
}
if r.URL.Path == "/v1.47/images/sha256:abc/json" {
writeImageInspect(w, "sha256:abc", "0.2.0-rc.1")
return
}
if !strings.Contains(r.URL.Query().Get("filters"), "dsh-mgr.role=runtime") {
t.Error("missing label filter")
}
io.WriteString(w, `[{"Id":"sha256:abc","Labels":{"dsh-mgr.role":"runtime","dsh-mgr.version":"0.2.0-rc.1","dsh-mgr.recipe":"`+imageRecipeRev+`"},"RepoTags":["runtime:0.2.0-rc.1"]},{"Id":"other","Labels":{}},{"Id":"unverified","Labels":{"dsh-mgr.role":"runtime","dsh-mgr.version":"0.2.0-rc.1","dsh-mgr.recipe":"`+imageRecipeRev+`"}},{"Id":"sha256:abc","Labels":{"dsh-mgr.role":"runtime","dsh-mgr.version":"9.9.9"}}]`)
})
seedVerifiedImage(t, a, "sha256:abc", "0.2.0-rc.1")
if w := imageRequest(a.images, "", "GET", ""); w.Code != 401 {
t.Fatal(w.Code)
}
w := imageRequest(a.images, sid, "GET", "")
var images []imageRecord
if err := json.Unmarshal(w.Body.Bytes(), &images); err != nil {
t.Fatal(err)
}
if len(images) != 2 || images[0].ID != "sha256:abc" || images[0].ImageID != images[0].ID || images[0].Status != "available" || images[1].Status != "unverified" {
t.Fatal(w.Body.String())
}
if w = imageRequest(a.images, sid, "POST", ""); w.Code != 405 {
t.Fatal(w.Code)
}
}
func seedVerifiedImage(t *testing.T, a *API, id, version string) {
t.Helper()
buildQueue.Lock()
defer buildQueue.Unlock()
if err := a.ensureBuilds(); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO image_builds(id,version,status,image_id,created_at,recipe_rev,platform) VALUES(?,?,'success',?,'now',?,?)", store.ID(), version, id, imageRecipeRev, imagePlatform); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT OR REPLACE INTO image_admissions(image_id,version,recipe_rev,platform,state,evidence,admitted_by,admitted_at) VALUES(?,?,?,?,'compatible','test fixture explicit acceptance','test','now')", id, version, imageRecipeRev, imagePlatform); err != nil {
t.Fatal(err)
}
}
func TestImageSanitizing(t *testing.T) {
s := sanitizeBuildLog("token=topsecret Authorization: Bearer hidden password=private https://user:[email protected]/")
for _, secret := range []string{"topsecret", "hidden", "private", "user:pass"} {
if strings.Contains(s, secret) {
t.Fatal(s)
}
}
if len(sanitizeBuildLog(strings.Repeat("x", 2*buildLogLimit))) > buildLogLimit {
t.Fatal("unbounded")
}
for _, v := range []string{"latest", "*", "^0.2.0-rc.1", "1.2", "0.2.0-rc.1;id", "https://example.com", ""} {
if validBuildVersion(v) {
t.Fatal(v)
}
}
if !validBuildVersion("0.2.0-rc.1-beta.1") {
t.Fatal("fixed prerelease rejected")
}
}
func TestVerificationFailureCleansUp(t *testing.T) {
var deleted atomic.Bool
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/create"):
io.WriteString(w, `{"Id":"bad"}`)
case r.Method == "DELETE":
deleted.Store(true)
w.WriteHeader(204)
default:
w.WriteHeader(500)
io.WriteString(w, `{"message":"cannot start"}`)
}
})
if err := a.verifyBuiltImage(context.Background(), "bad", "0.2.0-rc.1"); err == nil {
t.Fatal("accepted failed start")
}
if !deleted.Load() {
t.Fatal("container leaked")
}
}
func TestBuildRestartRecovery(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(500) })
buildQueue.Lock()
defer buildQueue.Unlock()
if err := a.ensureBuilds(); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO image_builds(id,version,status,created_at) VALUES('old','0.2.0-rc.1','running','now')"); err != nil {
t.Fatal(err)
}
delete(buildQueue.initialized, a.s.DB)
if err := a.ensureBuilds(); err != nil {
t.Fatal(err)
}
var status, detail string
if err := a.s.DB.QueryRow("SELECT status,error FROM image_builds WHERE id='old'").Scan(&status, &detail); err != nil {
t.Fatal(err)
}
if status != "failed" || !strings.Contains(detail, "restart") {
t.Fatalf("%s %s", status, detail)
}
}
func TestFailedVerificationNeverAvailable(t *testing.T) {
var deleted atomic.Bool
imageID := "sha256:" + strings.Repeat("c", 64)
info := releaseFixture()
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1.47/images/sha256:abc/json", "/v1.47/images/sha256:reuse/json":
writeImageInspect(w, strings.TrimSuffix(strings.TrimPrefix(r.URL.Path, "/v1.47/images/"), "/json"), "0.2.0-rc.1")
case "/v1.47/images/json":
writeJSON(w, []any{map[string]any{"Id": imageID, "Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "0.2.0-rc.1", "dsh-mgr.recipe": imageRecipeRev}}})
case "/v1.47/containers/create":
io.WriteString(w, `{"Id":"check"}`)
case "/v1.47/containers/check/start", "/v1.47/exec/probe/start":
w.WriteHeader(204)
case "/v1.47/containers/check/exec":
io.WriteString(w, `{"Id":"probe"}`)
case "/v1.47/exec/probe/json":
io.WriteString(w, `{"Running":false,"ExitCode":1}`)
case "/v1.47/containers/check":
deleted.Store(true)
w.WriteHeader(204)
case "/v1.47/build":
writeJSON(w, map[string]any{"aux": map[string]string{"ID": imageID}})
case "/v1.47/images/" + imageID + "/json":
parts := strings.Split(imagePlatform, "/")
writeJSON(w, map[string]any{"Id": imageID, "Os": parts[0], "Architecture": parts[1], "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": info.Version, "dsh-mgr.recipe": imageRecipeRev, "dsh-mgr.release": info.ReleaseID, "dsh-mgr.source": info.Repository, "dsh-mgr.commit": info.Commit}}})
default:
t.Errorf("unexpected Docker call %s", r.URL)
w.WriteHeader(500)
}
})
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='u'"); err != nil {
t.Fatal(err)
}
old := fetchRelease
t.Cleanup(func() { fetchRelease = old })
fetchRelease = func(context.Context, string, string) (releaseInfo, io.Reader, error) {
return info, strings.NewReader("context"), nil
}
b, err := a.enqueueRelease("u", info)
if err != nil {
t.Fatal(err)
}
a.runImageBuild(b)
if err := scanBuildSummary(a.s.DB.QueryRow("SELECT "+buildSummaryColumns+" FROM image_builds WHERE id=?", b.ID), &b); err != nil {
t.Fatal(err)
}
if b.Status != "failed" || b.ImageID != imageID || !strings.Contains(b.Error, "verification failed") || !deleted.Load() {
t.Fatalf("%+v cleanup=%v", b, deleted.Load())
}
w := imageRequest(a.images, sid, "GET", "")
var images []imageRecord
if err := json.Unmarshal(w.Body.Bytes(), &images); err != nil || w.Code != 200 || len(images) != 1 || images[0].Status != "unverified" {
t.Fatal("failed image advertised", w.Body.String())
}
w = imageRequest(a.instances, sid, "POST", fmt.Sprintf(`{"name":"rejected","image_id":%q}`, imageID))
if w.Code != 400 {
t.Fatal("unverified image accepted", w.Code, w.Body.String())
}
}
func writeImageInspect(w http.ResponseWriter, id, version string) {
parts := strings.Split(imagePlatform, "/")
writeJSON(w, map[string]any{"Id": id, "Os": parts[0], "Architecture": parts[1], "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": version, "dsh-mgr.recipe": imageRecipeRev}}})
}
internal/api/instance_data.go
-57
View File
@@ -1,57 +0,0 @@
package api
import (
"errors"
"os"
"strings"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
// Call only with the instance lock held, after Docker resources are removed.
func (a *API) removeInstanceData(i store.Instance) error {
for _, id := range []string{i.UID, i.ID} {
if id == "" || id == "." || id == ".." || strings.ContainsAny(id, "/\\") {
return errors.New("invalid instance storage path")
}
}
for _, parts := range [][]string{{"workspaces", i.UID, "instances", i.ID}, {"backups", i.UID, i.ID}} {
if err := a.removeDataDirectory(parts); err != nil {
return err
}
}
return nil
}
// Walk parents without following symlinks; RemoveAll unlinks child symlinks.
func (a *API) removeDataDirectory(parts []string) error {
root, err := os.OpenRoot(a.root)
if err != nil {
return err
}
defer func() { root.Close() }()
for _, part := range parts[:len(parts)-1] {
st, err := root.Lstat(part)
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return err
}
if !st.IsDir() {
return errors.New("unsafe storage path")
}
next, err := root.OpenRoot(part)
if err != nil {
return err
}
actual, err := next.Stat(".")
if err != nil || !os.SameFile(st, actual) {
next.Close()
return errors.New("storage path changed")
}
root.Close()
root = next
}
return root.RemoveAll(parts[len(parts)-1])
}
internal/api/instance_data_test.go
-110
View File
@@ -1,110 +0,0 @@
package api
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func TestInstanceDeleteDataAndRetry(t *testing.T) {
for _, blocked := range []bool{false, true} {
t.Run(map[bool]string{false: "success", true: "retry"}[blocked], func(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == http.MethodDelete {
w.WriteHeader(204)
} else {
w.WriteHeader(404)
}
})
i := store.Instance{ID: "one", UID: "owner", ContainerID: "container"}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
write := func(rel string) {
p := filepath.Join(a.root, rel)
if err := os.MkdirAll(filepath.Dir(p), 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, []byte("keep"), 0600); err != nil {
t.Fatal(err)
}
}
write("workspaces/owner/instances/one/home/data")
write("workspaces/owner/instances/one/pre-restore-test/data")
write("workspaces/owner/instances/two/home/data")
write("backups/owner/one/archive.dshbak")
write("backups/owner/two/archive.dshbak")
outside := t.TempDir()
sentinel := filepath.Join(outside, "sentinel")
if err := os.WriteFile(sentinel, []byte("safe"), 0600); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Join(a.root, "workspaces/owner/instances/one/home/link")); err != nil {
t.Fatal(err)
}
backups := filepath.Join(a.root, "backups/owner")
if blocked {
if err := os.Rename(backups, backups+"-saved"); err != nil {
t.Fatal(err)
}
if err := os.Symlink(backups+"-saved", backups); err != nil {
t.Fatal(err)
}
}
remove := func() *httptest.ResponseRecorder {
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("DELETE", "/api/instances/one", "", sid))
return w
}
w := remove()
if blocked {
if w.Code != 500 {
t.Fatal(w.Code, w.Body.String())
}
current, err := a.s.GetInstance(i.ID)
if err != nil || current.Status != "error" {
t.Fatal(current, err)
}
if err := os.Remove(backups); err != nil {
t.Fatal(err)
}
if err := os.Rename(backups+"-saved", backups); err != nil {
t.Fatal(err)
}
w = remove()
}
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
for _, rel := range []string{"workspaces/owner/instances/one", "backups/owner/one"} {
if _, err := os.Lstat(filepath.Join(a.root, rel)); !os.IsNotExist(err) {
t.Fatal("data retained", rel, err)
}
}
for _, p := range []string{sentinel, filepath.Join(a.root, "workspaces/owner/instances/two/home/data"), filepath.Join(a.root, "backups/owner/two/archive.dshbak")} {
if _, err := os.Stat(p); err != nil {
t.Fatal("unrelated data removed", err)
}
}
})
}
}
func TestUsageDoesNotCreateLegacyFiles(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Fatal("unexpected Docker request") })
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest("GET", "/api/ws/usage", "", sid))
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
if _, err := os.Stat(filepath.Join(a.root, "workspaces/owner/files")); !os.IsNotExist(err) {
t.Fatal("legacy directory created", err)
}
if _, err := a.openWorkspace(store.User{ID: "owner"}, ""); err == nil {
t.Fatal("shared workspace accepted")
}
}
internal/api/locks.go
-56
View File
@@ -1,56 +0,0 @@
package api
import "sync"
// keyedLocks hands out one mutex per string key so slow work for one instance
// or user never blocks unrelated work behind a shared global lock. Registry
// entries are removed once the last holder releases, keeping the map bounded.
type keyedLocks struct {
mu sync.Mutex
locks map[string]*keyedLock
}
type keyedLock struct {
owner *keyedLocks
key string
mu sync.Mutex
refs int
}
// Acquire registers interest in the key's lock (blocking callers keep the
// registry entry alive) and then blocks until the mutex is held.
func (k *keyedLocks) Acquire(key string) *keyedLock {
k.mu.Lock()
if k.locks == nil {
k.locks = make(map[string]*keyedLock)
}
l, ok := k.locks[key]
if !ok {
l = &keyedLock{owner: k, key: key}
k.locks[key] = l
}
l.refs++
k.mu.Unlock()
l.mu.Lock()
return l
}
// Release frees the mutex and drops the registry entry when no holder or
// waiter remains.
func (l *keyedLock) Release() {
l.mu.Unlock()
k := l.owner
k.mu.Lock()
l.refs--
if l.refs == 0 {
delete(k.locks, l.key)
}
k.mu.Unlock()
}
// held reports whether any keyed lock is currently held or being waited for.
func (k *keyedLocks) held() bool {
k.mu.Lock()
defer k.mu.Unlock()
return len(k.locks) > 0
}
internal/api/orphans.go
-133
View File
@@ -1,133 +0,0 @@
package api
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/url"
"os"
"path/filepath"
"regexp"
"strings"
"time"
)
var orphanID = regexp.MustCompile(`^[a-f0-9]{64}$`)
var orphanLabelID = regexp.MustCompile(`^[a-f0-9]{32}$`)
func (a *API) checkOrphan(ctx context.Context, id string) error {
var c struct {
ID, Name string
Config struct{ Labels map[string]string }
State struct {
Status string
Running, Paused, Restarting bool
}
Mounts []struct{ Type, Source, Destination string }
}
if !orphanID.MatchString(id) {
return errors.New("invalid container ID")
}
if err := a.docker.Do(ctx, "GET", "/containers/"+id+"/json", nil, &c); err != nil {
return errors.New("container unavailable")
}
inst, owner := c.Config.Labels["dsh-mgr.instance"], c.Config.Labels["dsh-mgr.owner"]
if c.ID != id || c.Config.Labels["dsh-mgr.role"] != "dsh" || !orphanLabelID.MatchString(inst) || !orphanLabelID.MatchString(owner) || strings.TrimPrefix(c.Name, "/") != runtimeNetwork(inst) {
return errors.New("container identity not eligible")
}
if c.State.Running || c.State.Paused || c.State.Restarting || (c.State.Status != "exited" && c.State.Status != "created") {
return errors.New("only stopped containers can be reclaimed")
}
var n int
if err := a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM instances WHERE id=? OR container_id=?", inst, id).Scan(&n); err != nil || n != 0 {
return errors.New("instance reference or database unavailable")
}
if err := a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM users WHERE id=?", owner).Scan(&n); err != nil || n != 1 {
return errors.New("unknown owner")
}
if err := a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM sqlite_master WHERE type='table' AND name='upgrade_journal'").Scan(&n); err != nil {
return err
}
if n != 0 {
if err := a.s.DB.QueryRowContext(ctx, "SELECT count(*) FROM upgrade_journal WHERE instance_id=? OR old_container=? OR new_container=?", inst, id, id).Scan(&n); err != nil || n != 0 {
return errors.New("upgrade reference or database unavailable")
}
}
host := os.Getenv("HOST_DATA_ROOT")
if host == "" {
host = a.root
}
if !filepath.IsAbs(host) {
return errors.New("host root unavailable")
}
expected := map[string]string{"/workspace": filepath.Join(host, "workspaces", owner, "instances", inst, "home")}
for _, m := range c.Mounts {
if m.Type == "tmpfs" && m.Destination == "/tmp" {
continue
}
want, ok := expected[m.Destination]
if !ok || m.Type != "bind" || filepath.Clean(m.Source) != want {
return errors.New("mount identity not eligible")
}
delete(expected, m.Destination)
}
if len(expected) != 0 {
return errors.New("required mounts missing")
}
return nil
}
func (a *API) orphans(w http.ResponseWriter, r *http.Request) {
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 20*time.Second)
defer cancel()
// checkOrphan re-verifies instance rows, upgrade journal entries and mount
// identities before removal, so no lifecycle lock is needed here: a
// container is only reclaimed when its instance row is already gone.
switch r.Method {
case "GET":
filters, _ := json.Marshal(map[string][]string{"label": {"dsh-mgr.role=dsh"}})
var list []struct {
ID string `json:"Id"`
}
if a.docker.Do(ctx, "GET", "/containers/json?all=1&filters="+url.QueryEscape(string(filters)), nil, &list) != nil {
fail(w, 502, "container inventory unavailable")
return
}
eligible := []string{}
for _, c := range list {
if a.checkOrphan(ctx, c.ID) == nil {
eligible = append(eligible, c.ID)
}
}
writeJSON(w, eligible)
case "DELETE":
id := r.URL.Query().Get("id")
if r.URL.Query().Get("confirm") != id || id == "" {
fail(w, 400, "exact container ID confirmation required")
return
}
if err := a.checkOrphan(ctx, id); err != nil {
fail(w, 409, err.Error())
return
}
// Never force removal or remove volumes. Docker rejects a concurrent start.
if a.docker.Do(ctx, "DELETE", "/containers/"+id+"?force=false&v=false", nil, nil) != nil {
fail(w, 409, "container changed or removal rejected")
return
}
a.s.AddAudit(u.ID, "reconcile.reclaim", id, "stopped orphan removed; bind data retained")
writeJSON(w, map[string]bool{"ok": true})
default:
fail(w, 405, "method not allowed")
}
}
internal/api/orphans_test.go
-73
View File
@@ -1,73 +0,0 @@
package api
import (
"context"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestOrphanReclaimGuards(t *testing.T) {
for _, kind := range []string{"safe", "legacy", "shared", "running", "foreign-mount", "reference", "upgrade", "confirmation", "nonadmin"} {
t.Run(kind, func(t *testing.T) {
t.Setenv("HOST_DATA_ROOT", "/host/data")
id, inst, owner := strings.Repeat("a", 64), strings.Repeat("b", 32), strings.Repeat("c", 32)
deleted := false
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == "DELETE" {
if r.URL.Query().Get("force") != "false" || r.URL.Query().Get("v") != "false" {
t.Fatal("unsafe deletion")
}
deleted = true
w.WriteHeader(204)
return
}
source := "/host/data/workspaces/" + owner + "/instances/" + inst + "/home"
if kind == "foreign-mount" {
source = "/unrelated"
}
mounts := []map[string]string{{"Type": "bind", "Source": source, "Destination": "/workspace"}}
if kind == "legacy" {
mounts[0]["Destination"] = "/data/dsh-home"
mounts = append(mounts, map[string]string{"Type": "bind", "Source": "/host/data/workspaces/" + owner + "/files", "Destination": "/workspace"})
}
if kind == "shared" {
mounts = append(mounts, map[string]string{"Type": "bind", "Source": "/host/data/workspaces/" + owner + "/files", "Destination": "/shared"})
}
writeJSON(w, map[string]any{"Id": id, "Name": "/dsh-inst-" + inst, "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": "dsh", "dsh-mgr.instance": inst, "dsh-mgr.owner": owner}}, "State": map[string]any{"Status": "exited", "Running": kind == "running"}, "Mounts": mounts})
})
if err := a.s.CreateUser(store.User{ID: owner, Username: "orphan-owner"}, "unused"); err != nil {
t.Fatal(err)
}
if kind != "nonadmin" {
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
}
if kind == "reference" {
if err := a.s.AddInstance(store.Instance{ID: inst, UID: owner, ContainerID: id}); err != nil {
t.Fatal(err)
}
}
if kind == "upgrade" {
if _, err := a.s.DB.Exec("CREATE TABLE upgrade_journal(instance_id TEXT,old_container TEXT,new_container TEXT); INSERT INTO upgrade_journal VALUES(?,?,'')", inst, id); err != nil {
t.Fatal(err)
}
}
if kind == "safe" && a.checkOrphan(context.Background(), id) != nil {
t.Fatal("safe orphan rejected")
}
confirm := id
if kind == "confirmation" {
confirm = "wrong"
}
w := httptest.NewRecorder()
a.orphans(w, runtimeRequest("DELETE", "/api/orphans?id="+id+"&confirm="+confirm, "", sid))
if (w.Code == 200) != (kind == "safe") || deleted != (kind == "safe") {
t.Fatalf("status=%d deleted=%v body=%s", w.Code, deleted, w.Body.String())
}
})
}
}
internal/api/password.go
-75
View File
@@ -1,75 +0,0 @@
package api
import (
"encoding/json"
"io"
"net/http"
"time"
"golang.org/x/crypto/bcrypt"
)
func (a *API) changePassword(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "no-store")
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
if _, _, err := a.user(r); err != nil {
fail(w, 401, "authentication required")
return
}
var in struct {
CurrentPassword string `json:"current_password"`
Password string `json:"password"`
}
decoder := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096))
decoder.DisallowUnknownFields()
if decoder.Decode(&in) != nil || decoder.Decode(new(any)) != io.EOF || in.CurrentPassword == "" || len(in.Password) < 12 || len(in.Password) > 72 {
fail(w, 400, "current password and new password 12–72 bytes required")
return
}
// Recheck credentials after waiting for concurrent logins or password resets.
a.authLoginMu.Lock()
defer a.authLoginMu.Unlock()
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !a.reauthenticateLocked(w, u.ID, in.CurrentPassword, http.StatusBadRequest) {
return
}
hash, err := bcrypt.GenerateFromPassword([]byte(in.Password), bcrypt.DefaultCost)
if err != nil {
fail(w, 500, "password error")
return
}
tx, err := a.s.DB.BeginTx(r.Context(), nil)
if err != nil {
fail(w, 500, "password change failed")
return
}
defer tx.Rollback()
_, err = tx.ExecContext(r.Context(), "UPDATE users SET pass_hash=? WHERE id=?", string(hash), u.ID)
if err == nil {
_, err = tx.ExecContext(r.Context(), "DELETE FROM auth_refresh WHERE sid IN (SELECT id FROM sessions WHERE uid=?)", u.ID)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "DELETE FROM sessions WHERE uid=?", u.ID)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "DELETE FROM auth_failures WHERE username=?", u.Username)
}
if err == nil {
_, err = tx.ExecContext(r.Context(), "INSERT INTO audit(uid,action,target,detail,at) VALUES(?,?,?,?,?)", u.ID, "user.change_password", u.ID, "", time.Now().Unix())
}
if err == nil {
err = tx.Commit()
}
if err != nil {
fail(w, 500, "password change failed")
return
}
writeJSON(w, map[string]bool{"ok": true, "reauthenticate": true})
}
internal/api/password_test.go
-365
View File
@@ -1,365 +0,0 @@
package api
import (
"encoding/json"
"fmt"
"golang.org/x/crypto/bcrypt"
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
const changePath = "/api/me/password"
func TestChangePasswordCurrentLength(t *testing.T) {
a, req := authFixture(t)
password := strings.Repeat("a", 72)
hash, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
if _, err = a.s.DB.Exec("UPDATE users SET pass_hash=? WHERE id='alice'", string(hash)); err != nil {
t.Fatal(err)
}
cs := req("POST", "/api/login", resetLogin("alice", password), "", nil).Result().Cookies()
if w := req("POST", changePath, changeBody(password+"x", resetSecret), cookieValue(cs, "dsh_csrf"), cs); w.Code != 400 {
t.Fatal("accepted prefix", w.Code)
}
if w := req("POST", changePath, changeBody(password, resetSecret), cookieValue(cs, "dsh_csrf"), cs); w.Code != 200 {
t.Fatal("72-byte current rejected", w.Code)
}
}
func TestChangePasswordConcurrentChanges(t *testing.T) {
a, req := authFixture(t)
cs := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
start := make(chan struct{})
results := make(chan int, 2)
for i := 0; i < 2; i++ {
go func() {
<-start
results <- req("POST", changePath, changeBody("password-123", resetSecret), cookieValue(cs, "dsh_csrf"), cs).Code
}()
}
close(start)
successes := 0
for i := 0; i < 2; i++ {
code := <-results
if code == 200 {
successes++
} else if code != 401 && code != 403 {
t.Fatal(code)
}
}
if successes != 1 {
t.Fatal("multiple changes", successes)
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='user.change_password'").Scan(&n); err != nil || n != 1 {
t.Fatal(n, err)
}
}
func TestChangePasswordRechecksWaitingCredentials(t *testing.T) {
for _, revoke := range []string{"session", "password"} {
t.Run(revoke, func(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
admin := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
reader := &resetSignalReader{reader: strings.NewReader(changeBody("password-123", resetSecret)), started: make(chan struct{})}
r := httptest.NewRequest("POST", changePath, reader)
r.Header.Set("Origin", a.origin)
r.Header.Set("X-CSRF-Token", cookieValue(admin, "dsh_csrf"))
for _, c := range admin {
r.AddCookie(c)
}
newHash, err := bcrypt.GenerateFromPassword([]byte(resetSecret), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
if _, err = a.s.DB.Exec("UPDATE users SET pass_hash=? WHERE id='bob'", string(newHash)); err != nil {
t.Fatal(err)
}
a.authLoginMu.Lock()
locked := true
defer func() {
if locked {
a.authLoginMu.Unlock()
}
}()
done := make(chan *httptest.ResponseRecorder, 1)
go func() { w := httptest.NewRecorder(); a.Handler().ServeHTTP(w, r); done <- w }()
select {
case <-reader.started:
case <-time.After(5 * time.Second):
t.Fatal("request did not reach decoder")
}
select {
case w := <-done:
t.Fatal("reset bypassed login lock", w.Code)
case <-time.After(150 * time.Millisecond):
}
query := "UPDATE users SET pass_hash=(SELECT pass_hash FROM users WHERE id='bob') WHERE id='alice'"
want := 400
if revoke == "session" {
query = "DELETE FROM sessions WHERE uid='alice'"
want = 401
}
if _, err := a.s.DB.Exec(query); err != nil {
t.Fatal(err)
}
a.authLoginMu.Unlock()
locked = false
select {
case w := <-done:
if w.Code != want {
t.Fatal(w.Code, w.Body.String())
}
case <-time.After(5 * time.Second):
t.Fatal("reset did not finish")
}
expected := resetSecret
if revoke == "session" {
expected = "password-123"
}
hash, err := a.s.Password("alice")
if err != nil || bcrypt.CompareHashAndPassword([]byte(hash), []byte(expected)) != nil {
t.Fatal("unauthorized reset", err)
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='user.change_password'").Scan(&n); err != nil || n != 0 {
t.Fatal(n, err)
}
})
}
}
func changeBody(current, password string) string {
b, _ := json.Marshal(map[string]string{"current_password": current, "password": password})
return string(b)
}
func TestChangePasswordValidation(t *testing.T) {
a, req := authFixture(t)
cs := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
csrf := cookieValue(cs, "dsh_csrf")
valid := changeBody("password-123", resetSecret)
for _, tc := range []struct {
name, method, body, csrf string
cookies []*http.Cookie
code int
}{
{"anonymous", "POST", valid, "", nil, 401},
{"csrf", "POST", valid, "bad", cs, 403},
{"method", "GET", "", csrf, cs, 405},
{"short", "POST", changeBody("password-123", strings.Repeat("a", 11)), csrf, cs, 400},
{"long", "POST", changeBody("password-123", strings.Repeat("a", 73)), csrf, cs, 400},
{"unicode long", "POST", changeBody("password-123", strings.Repeat("密", 25)), csrf, cs, 400},
{"missing current", "POST", changeBody("", resetSecret), csrf, cs, 400},
{"malformed", "POST", "{", csrf, cs, 400},
{"trailing", "POST", valid + "{}", csrf, cs, 400},
{"oversized", "POST", changeBody("password-123", strings.Repeat("a", 4096)), csrf, cs, 400},
{"target id", "POST", strings.TrimSuffix(valid, "}") + `,"id":"bob"}`, csrf, cs, 400},
{"wrong current", "POST", changeBody("wrong", resetSecret), csrf, cs, 400},
{"missing access", "POST", valid, csrf, []*http.Cookie{{Name: "dsh_session", Value: cookieValue(cs, "dsh_session")}}, 401},
} {
t.Run(tc.name, func(t *testing.T) {
w := req(tc.method, changePath, tc.body, tc.csrf, tc.cookies)
if w.Code != tc.code {
t.Fatal(w.Code, w.Body.String())
}
})
}
for _, origin := range []string{"", "https://evil.test"} {
r := httptest.NewRequest("POST", changePath, strings.NewReader(valid))
r.Header.Set("Origin", origin)
r.Header.Set("X-CSRF-Token", csrf)
for _, c := range cs {
r.AddCookie(c)
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
if w.Code != 403 {
t.Fatal(w.Code)
}
}
if w := req("GET", "/api/me", "", "", cs); w.Code != 200 {
t.Fatal("validation revoked session", w.Code)
}
for _, user := range []string{"alice", "bob"} {
if w := req("POST", "/api/login", resetLogin(user, "password-123"), "", nil); w.Code != 200 {
t.Fatal("validation changed password", user, w.Code)
}
}
}
func TestChangePasswordLifecycle(t *testing.T) {
for _, password := range []string{strings.Repeat("a", 12), strings.Repeat("密", 24)} {
t.Run(fmt.Sprint(len(password)), func(t *testing.T) {
a, req := authFixture(t)
login := func(user string) []*http.Cookie {
w := req("POST", "/api/login", resetLogin(user, "password-123"), "", nil)
if w.Code != 200 {
t.Fatal(w.Code)
}
return w.Result().Cookies()
}
cs, second, other := login("alice"), login("alice"), login("bob")
if _, err := a.s.DB.Exec("INSERT INTO auth_failures VALUES('alice',2,0)"); err != nil {
t.Fatal(err)
}
w := req("POST", changePath, changeBody("password-123", password), cookieValue(cs, "dsh_csrf"), cs)
var result map[string]bool
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &result) != nil || !result["ok"] || !result["reauthenticate"] || w.Header().Get("Cache-Control") != "no-store" {
t.Fatal(w.Code, w.Body.String())
}
for _, old := range [][]*http.Cookie{cs, second} {
if w := req("GET", "/api/me", "", "", old); w.Code != 401 {
t.Fatal("old access", w.Code)
}
if w := req("POST", "/api/refresh", "", cookieValue(old, "dsh_csrf"), old); w.Code != 403 {
t.Fatal("old refresh", w.Code)
}
}
for query, want := range map[string]int{"SELECT count(*) FROM sessions WHERE uid='alice'": 0, "SELECT count(*) FROM auth_refresh": 1, "SELECT count(*) FROM auth_failures WHERE username='alice'": 0, "SELECT count(*) FROM audit WHERE action='user.change_password' AND uid='alice' AND target='alice' AND detail=''": 1} {
var n int
if err := a.s.DB.QueryRow(query).Scan(&n); err != nil || n != want {
t.Fatal(query, n, err)
}
}
if w := req("GET", "/api/me", "", "", other); w.Code != 200 {
t.Fatal("other session", w.Code)
}
if w := req("POST", "/api/refresh", "", cookieValue(other, "dsh_csrf"), other); w.Code != 200 {
t.Fatal("other refresh", w.Code)
}
if w := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil); w.Code != 401 {
t.Fatal("old password", w.Code)
}
if w := req("POST", "/api/login", resetLogin("alice", password), "", nil); w.Code != 200 {
t.Fatal("new password", w.Code)
}
if w := req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil); w.Code != 200 {
t.Fatal("other password", w.Code)
}
})
}
}
func TestChangePasswordConcurrentFailures(t *testing.T) {
a, req := authFixture(t)
cs := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
results := make(chan int, 12)
start := make(chan struct{})
for i := 0; i < 12; i++ {
go func() {
<-start
results <- req("POST", changePath, changeBody("wrong", resetSecret), cookieValue(cs, "dsh_csrf"), cs).Code
}()
}
close(start)
counts := map[int]int{}
for i := 0; i < 12; i++ {
counts[<-results]++
}
if counts[400] != 5 || counts[429] != 7 {
t.Fatal(counts)
}
var n int
var until int64
if err := a.s.DB.QueryRow("SELECT failures,locked_until FROM auth_failures WHERE username='alice'").Scan(&n, &until); err != nil || n != 5 || until <= time.Now().Unix() {
t.Fatal(n, until, err)
}
if w := req("POST", changePath, changeBody("password-123", resetSecret), cookieValue(cs, "dsh_csrf"), cs); w.Code != 429 {
t.Fatal("logged-in bypass", w.Code)
}
if w := req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil); w.Code != 429 {
t.Fatal("login bypass", w.Code)
}
if w := req("GET", "/api/me", "", "", cs); w.Code != 200 {
t.Fatal("wrong password logged out", w.Code)
}
if _, err := a.s.DB.Exec("UPDATE auth_failures SET locked_until=? WHERE username='alice'", time.Now().Add(-time.Second).Unix()); err != nil {
t.Fatal(err)
}
if w := req("POST", changePath, changeBody("password-123", resetSecret), cookieValue(cs, "dsh_csrf"), cs); w.Code != 200 {
t.Fatal("expired lock", w.Code)
}
}
func TestChangePasswordRollback(t *testing.T) {
for _, step := range []struct{ table, event string }{{"users", "UPDATE"}, {"auth_refresh", "DELETE"}, {"sessions", "DELETE"}, {"auth_failures", "DELETE"}, {"audit", "INSERT"}} {
t.Run(step.table, func(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
_ = req("POST", "/api/login", resetLogin("alice", "password-123"), "", nil).Result().Cookies()
target := req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil).Result().Cookies()
before, err := a.s.Password("bob")
if err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO auth_failures VALUES('bob',2,0)"); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec(fmt.Sprintf("CREATE TRIGGER reject_reset BEFORE %s ON %s BEGIN SELECT RAISE(ABORT,'forced failure'); END", step.event, step.table)); err != nil {
t.Fatal(err)
}
w := req("POST", changePath, changeBody("password-123", resetSecret), cookieValue(target, "dsh_csrf"), target)
if w.Code != 500 || strings.Contains(w.Body.String(), resetSecret) {
t.Fatal(w.Code, w.Body.String())
}
after, err := a.s.Password("bob")
if err != nil || before != after {
t.Fatal("hash changed", err)
}
for query, want := range map[string]int{"SELECT count(*) FROM sessions WHERE uid='bob'": 1, "SELECT count(*) FROM auth_refresh": 2, "SELECT count(*) FROM auth_failures WHERE username='bob' AND failures=2 AND locked_until=0": 1, "SELECT count(*) FROM audit WHERE action='user.change_password'": 0} {
var n int
if err := a.s.DB.QueryRow(query).Scan(&n); err != nil || n != want {
t.Fatal(query, n, err)
}
}
if w := req("GET", "/api/me", "", "", target); w.Code != 200 {
t.Fatal("session changed", w.Code)
}
if w := req("POST", "/api/refresh", "", cookieValue(target, "dsh_csrf"), target); w.Code != 200 {
t.Fatal("refresh changed", w.Code)
}
})
}
}
func TestChangePasswordConcurrentLogin(t *testing.T) {
a, req := authFixture(t)
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='alice'"); err != nil {
t.Fatal(err)
}
admin := req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil).Result().Cookies()
start := make(chan struct{})
results := make(chan *httptest.ResponseRecorder, 8)
for i := 0; i < 8; i++ {
go func() { <-start; results <- req("POST", "/api/login", resetLogin("bob", "password-123"), "", nil) }()
}
close(start)
w := req("POST", changePath, changeBody("password-123", resetSecret), cookieValue(admin, "dsh_csrf"), admin)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
for i := 0; i < 8; i++ {
w := <-results
if w.Code != 200 && w.Code != 401 && w.Code != 429 {
t.Fatal(w.Code, w.Body.String())
}
if w.Code == 200 {
if check := req("GET", "/api/me", "", "", w.Result().Cookies()); check.Code != 401 {
t.Fatal("old-password login retained session", check.Code)
}
}
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM sessions WHERE uid='bob'").Scan(&n); err != nil || n != 0 {
t.Fatal("old login sessions", n, err)
}
}
internal/api/paths.go
-18
View File
@@ -1,18 +0,0 @@
package api
import (
"errors"
"path/filepath"
"strings"
)
func toHostPath(containerRoot, hostRoot, containerPath string) (string, error) {
if !filepath.IsAbs(containerRoot) || !filepath.IsAbs(hostRoot) || !filepath.IsAbs(containerPath) {
return "", errors.New("absolute storage paths required")
}
rel, err := filepath.Rel(containerRoot, containerPath)
if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) {
return "", errors.New("storage path escapes root")
}
return filepath.Join(hostRoot, rel), nil
}
internal/api/paths_test.go
-15
View File
@@ -1,15 +0,0 @@
package api
import "testing"
func TestToHostPath(t *testing.T) {
for _, tc := range []struct {
input, want string
bad bool
}{{"/data/workspaces/u/files", "/host/workspaces/u/files", false}, {"/data", "/host", false}, {"/data-other/x", "", true}, {"/data/../secret", "", true}, {"relative", "", true}} {
got, err := toHostPath("/data", "/host", tc.input)
if (err != nil) != tc.bad || got != tc.want {
t.Fatalf("%s: %q %v", tc.input, got, err)
}
}
}
internal/api/profile_preview.go
-226
View File
@@ -1,226 +0,0 @@
package api
import (
"bytes"
"context"
"encoding/binary"
"encoding/json"
"errors"
"io"
"net/http"
"net/url"
"os"
"regexp"
"strings"
"syscall"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"gopkg.in/yaml.v3"
)
const previewLimit = 1 << 20
var bundleReference = regexp.MustCompile("^(@[a-z0-9._-]+/)?[a-z0-9._-]+$")
// The caller must authenticate and check instance ownership before dispatching.
func (a *API) profilePreview(w http.ResponseWriter, r *http.Request, i store.Instance) {
w.Header().Set("Cache-Control", "no-store")
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
bundles, err := a.profileBundles(i)
if err != nil {
fail(w, 400, "bundle manifest unavailable")
return
}
result := map[string]any{"bundles": bundles, "config": "", "redacted": true}
if i.ContainerID == "" || i.Status != "running" {
result["preview_error"] = "启动实例后可预览已保存的配置。"
writeJSON(w, result)
return
}
ctx, cancel := context.WithTimeout(r.Context(), 15*time.Second)
defer cancel()
content, err := a.dumpProfile(ctx, i.ContainerID)
if err != nil {
result["preview_error"] = "配置预览不可用；命令输出和错误详情已隐藏。"
} else {
result["config"] = content
}
writeJSON(w, result)
}
func (a *API) profileBundles(i store.Instance) (any, error) {
parent, err := upgradeRoot(a.root, i)
if err != nil {
return nil, err
}
defer parent.Close()
if err = configCheckPath(parent, "home/profiles/web/package.json"); err != nil {
return nil, err
}
root, err := configProfileRoot(parent, false)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
defer root.Close()
f, err := root.OpenFile("package.json", os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
if err != nil {
return nil, err
}
defer f.Close()
st, err := f.Stat()
if err != nil || !st.Mode().IsRegular() || st.Size() > previewLimit {
return nil, errors.New("invalid manifest")
}
data, err := io.ReadAll(io.LimitReader(f, previewLimit+1))
if err != nil || len(data) > previewLimit {
return nil, errors.New("invalid manifest")
}
var pkg map[string]json.RawMessage
if json.Unmarshal(data, &pkg) != nil {
return nil, errors.New("invalid manifest")
}
var profile any
raw := pkg["dsh.profile"]
if raw == nil {
var dsh map[string]json.RawMessage
if pkg["dsh"] != nil && json.Unmarshal(pkg["dsh"], &dsh) != nil {
return nil, errors.New("invalid manifest")
}
raw = dsh["profile"]
}
if raw == nil {
return nil, nil
}
if json.Unmarshal(raw, &profile) != nil {
return nil, errors.New("invalid manifest")
}
return redactPreview(profile, false), nil
}
func (a *API) dumpProfile(ctx context.Context, container string) (string, error) {
var exec struct{ ID string }
err := a.docker.Do(ctx, "POST", "/containers/"+url.PathEscape(container)+"/exec", map[string]any{
"Cmd": []string{"dsh", "--profile", "web", "--dump-config"}, "AttachStdout": true, "AttachStderr": true, "Tty": false,
}, &exec)
if err != nil {
return "", err
}
if exec.ID == "" {
return "", errors.New("missing exec ID")
}
path := "/exec/" + url.PathEscape(exec.ID)
body, _ := json.Marshal(map[string]bool{"Detach": false, "Tty": false})
resp, err := a.docker.Stream(ctx, "POST", path+"/start", bytes.NewReader(body), "application/json")
if err != nil {
return "", err
}
defer resp.Body.Close()
data, err := previewStdout(resp.Body)
if err != nil {
return "", err
}
var state struct {
Running bool
ExitCode int
}
if err = a.docker.Do(ctx, "GET", path+"/json", nil, &state); err != nil {
return "", err
}
if state.Running || state.ExitCode != 0 {
return "", errors.New("preview command failed")
}
var document any
decoder := yaml.NewDecoder(bytes.NewReader(data))
if err = decoder.Decode(&document); err != nil {
return "", errors.New("invalid preview")
}
var extra any
if decoder.Decode(&extra) != io.EOF {
return "", errors.New("multiple preview documents")
}
switch document.(type) {
case map[string]any, []any:
// DSH exports its composed plugin tree as a YAML sequence.
default:
return "", errors.New("invalid preview structure")
}
safe, err := yaml.Marshal(redactPreview(document, true))
return string(safe), err
}
func previewStdout(r io.Reader) ([]byte, error) {
var output bytes.Buffer
total := 0
for {
var header [8]byte
_, err := io.ReadFull(r, header[:])
if err == io.EOF {
return output.Bytes(), nil
}
if err != nil {
return nil, err
}
if (header[0] != 1 && header[0] != 2) || header[1] != 0 || header[2] != 0 || header[3] != 0 {
return nil, errors.New("invalid exec stream")
}
size := int64(binary.BigEndian.Uint32(header[4:]))
total += 8
if size > int64(previewLimit-total) {
return nil, errors.New("preview too large")
}
total += int(size)
var dst io.Writer = io.Discard
if header[0] == 1 {
dst = &output
}
if _, err = io.CopyN(dst, r, size); err != nil {
return nil, err
}
}
}
func redactPreview(value any, strict bool) any {
switch v := value.(type) {
case map[string]any:
out := make(map[string]any, len(v))
for key, item := range v {
normalized := strings.ToLower(strings.NewReplacer("_", "", "-", "", ".", "").Replace(key))
sensitive := false
for _, word := range []string{"key", "secret", "token", "password", "passwd", "credential", "authorization", "cookie", "env", "header"} {
sensitive = sensitive || strings.Contains(normalized, word)
}
if sensitive {
out[key] = "[REDACTED]"
} else {
out[key] = redactPreview(item, strict)
}
}
return out
case []any:
out := make([]any, len(v))
for j, item := range v {
out[j] = redactPreview(item, strict)
}
return out
case string:
// Plugins can resolve credentials into arbitrary string fields, not just apiKey.
if strict || !bundleReference.MatchString(v) {
return "[REDACTED]"
}
return v
case map[any]any:
return "[REDACTED]"
default:
return v
}
}
internal/api/profile_preview_test.go
-188
View File
@@ -1,188 +0,0 @@
package api
import (
"bytes"
"context"
"encoding/binary"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func previewFrame(stream byte, text string) []byte {
header := make([]byte, 8)
header[0] = stream
binary.BigEndian.PutUint32(header[4:], uint32(len(text)))
return append(header, []byte(text)...)
}
func TestProfileDumpFixedCommandAndRedaction(t *testing.T) {
daemon := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1.47/containers/container/exec":
var request struct {
Cmd []string
AttachStdout, AttachStderr, Tty bool
}
if json.NewDecoder(r.Body).Decode(&request) != nil {
t.Error("invalid exec request")
}
if r.Method != "POST" || !reflect.DeepEqual(request.Cmd, []string{"dsh", "--profile", "web", "--dump-config"}) || !request.AttachStdout || !request.AttachStderr || request.Tty {
t.Errorf("unsafe command: %+v", request)
}
writeJSON(w, map[string]string{"Id": "exec"})
case "/v1.47/exec/exec/start":
w.Write(previewFrame(2, "stderr-secret"))
w.Write(previewFrame(1, "provider:\n apiKey: direct-secret\n API_KEY: underscore-secret\n password: numeric-secret\n nested:\n - token: nested-secret\n arbitrary: resolved-credential\n endpoint: https://name:[email protected]/?token=secret\n enabled: true\n"))
case "/v1.47/exec/exec/json":
writeJSON(w, map[string]any{"Running": false, "ExitCode": 0})
default:
t.Errorf("unexpected Docker call %s", r.URL.Path)
w.WriteHeader(404)
}
}))
defer daemon.Close()
a := &API{docker: docker.New(daemon.URL)}
out, err := a.dumpProfile(context.Background(), "container")
if err != nil {
t.Fatal(err)
}
for _, secret := range []string{"secret", "resolved-credential", "https://"} {
if strings.Contains(out, secret) {
t.Fatalf("secret leaked: %s", out)
}
}
if !strings.Contains(out, "[REDACTED]") || !strings.Contains(out, "enabled: true") {
t.Fatalf("missing structure: %s", out)
}
}
func TestProfileDumpPluginSequence(t *testing.T) {
daemon := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/exec"):
writeJSON(w, map[string]string{"Id": "exec"})
case strings.HasSuffix(r.URL.Path, "/start"):
w.Write(previewFrame(1, "# bundle\n- id: plugin\n disabled: !!js '!ctx.get(\"profileContext\")'\n config:\n apiKey: private-secret\n enabled: true\n"))
default:
writeJSON(w, map[string]any{"Running": false, "ExitCode": 0})
}
}))
defer daemon.Close()
a := &API{docker: docker.New(daemon.URL)}
out, err := a.dumpProfile(context.Background(), "container")
if err != nil {
t.Fatal(err)
}
if strings.Contains(out, "private-secret") || strings.Contains(out, "ctx.get") || !strings.Contains(out, "enabled: true") || !strings.Contains(out, "[REDACTED]") {
t.Fatalf("invalid sequence redaction: %s", out)
}
}
func TestProfileDumpRejectsUnsafeOutput(t *testing.T) {
for _, tc := range []struct {
name, output string
exit int
}{
{"invalid", "apiKey: [secret", 0}, {"plain", "secret", 0},
{"documents", "a: b\n---\napiKey: secret", 0}, {"failure", "a: b", 1},
{"empty", "", 0},
} {
t.Run(tc.name, func(t *testing.T) {
daemon := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch {
case strings.HasSuffix(r.URL.Path, "/exec"):
writeJSON(w, map[string]string{"Id": "exec"})
case strings.HasSuffix(r.URL.Path, "/start"):
w.Write(previewFrame(1, tc.output))
default:
writeJSON(w, map[string]any{"Running": false, "ExitCode": tc.exit})
}
}))
defer daemon.Close()
a := &API{docker: docker.New(daemon.URL)}
if out, err := a.dumpProfile(context.Background(), "container"); err == nil || out != "" {
t.Fatalf("unsafe output returned: %q, %v", out, err)
}
})
}
}
func TestPreviewStreamBounds(t *testing.T) {
for _, data := range [][]byte{
[]byte("raw secret"), {1, 0}, previewFrame(3, "secret"),
previewFrame(1, strings.Repeat("x", previewLimit)),
previewFrame(1, "secret")[:10],
} {
if _, err := previewStdout(bytes.NewReader(data)); err == nil {
t.Fatal("accepted invalid stream")
}
}
data := append(previewFrame(2, "secret"), previewFrame(1, "safe")...)
got, err := previewStdout(bytes.NewReader(data))
if err != nil || string(got) != "safe" {
t.Fatalf("stdout: %q %v", got, err)
}
}
func previewFixture(t *testing.T) (*API, store.Instance, string) {
t.Helper()
root := t.TempDir()
i := store.Instance{ID: "instance", UID: "owner", ContainerID: "container", Status: "stopped"}
path := filepath.Join(root, "workspaces", i.UID, "instances", i.ID, "home", "profiles", "web")
if err := os.MkdirAll(path, 0700); err != nil {
t.Fatal(err)
}
return &API{root: root}, i, path
}
func TestProfilePreviewManifestAndErrors(t *testing.T) {
a, i, path := previewFixture(t)
if err := os.WriteFile(filepath.Join(path, "package.json"), []byte(`{"dsh":{"profile":{"bundles":["@deepseek/bundle"],"apiKey":"private-value"}},"scripts":{"secret":"ignored-value"}}`), 0600); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.profilePreview(w, httptest.NewRequest("GET", "/preview?cmd=cat&path=.credentials.yaml", nil), i)
if w.Code != 200 || !strings.Contains(w.Body.String(), "@deepseek/bundle") || strings.Contains(w.Body.String(), "private-value") || strings.Contains(w.Body.String(), "ignored-value") {
t.Fatalf("manifest response: %d %s", w.Code, w.Body.String())
}
if w.Header().Get("Cache-Control") != "no-store" {
t.Fatal("preview must not be cached")
}
daemon := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { http.Error(w, "daemon-private-value", 500) }))
defer daemon.Close()
a.docker = docker.New(daemon.URL)
i.Status = "running"
w = httptest.NewRecorder()
a.profilePreview(w, httptest.NewRequest("GET", "/preview", nil), i)
if strings.Contains(w.Body.String(), "daemon-private-value") || !strings.Contains(w.Body.String(), "preview_error") {
t.Fatalf("unsafe error: %s", w.Body.String())
}
w = httptest.NewRecorder()
a.profilePreview(w, httptest.NewRequest("POST", "/preview", nil), i)
if w.Code != 405 {
t.Fatalf("method: %d", w.Code)
}
}
func TestProfileBundlesRejectsCredentialSymlink(t *testing.T) {
a, i, path := previewFixture(t)
credentials := filepath.Join(path, "..", "..", ".credentials.yaml")
if err := os.WriteFile(credentials, []byte("private-value"), 0600); err != nil {
t.Fatal(err)
}
if err := os.Symlink(credentials, filepath.Join(path, "package.json")); err != nil {
t.Fatal(err)
}
if _, err := a.profileBundles(i); err == nil {
t.Fatal("accepted credential symlink")
}
}
internal/api/quota.go
-63
View File
@@ -1,63 +0,0 @@
package api
import (
"context"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"log"
"time"
)
// RunQuotaMonitor reports soft quota crossings; it never claims to constrain
// runtime writes and never deletes user data to recover capacity.
func (a *API) RunQuotaMonitor(ctx context.Context) {
ticker := time.NewTicker(5 * time.Minute)
defer ticker.Stop()
exceeded := map[string]bool{}
for {
a.checkQuotas(ctx, exceeded)
select {
case <-ctx.Done():
return
case <-ticker.C:
}
}
}
func (a *API) checkQuotas(ctx context.Context, exceeded map[string]bool) {
rows, err := a.s.DB.QueryContext(ctx, "SELECT id,quota FROM users")
if err != nil {
return
}
var users []store.User
for rows.Next() {
var u store.User
if rows.Scan(&u.ID, &u.Quota) != nil {
rows.Close()
return
}
users = append(users, u)
}
err = rows.Err()
rows.Close()
if err != nil {
return
}
for _, u := range users {
if ctx.Err() != nil {
return
}
used, err := a.userUsage(u)
if err != nil {
log.Printf("quota usage unavailable for user %s", u.ID)
continue
}
over := used > u.Quota
if over && !exceeded[u.ID] {
a.s.AddAudit(u.ID, "quota.exceeded", u.ID, "soft quota exceeded; runtime writes remain unrestricted")
}
if !over && exceeded[u.ID] {
a.s.AddAudit(u.ID, "quota.recovered", u.ID, "")
}
exceeded[u.ID] = over
}
}
internal/api/quota_test.go
-31
View File
@@ -1,31 +0,0 @@
package api
import (
"context"
"net/http"
"os"
"path/filepath"
"testing"
)
func TestQuotaMonitorReportsTransitions(t *testing.T) {
a, _ := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) {})
dir := filepath.Join(a.root, "workspaces", "owner", "files")
if err := os.MkdirAll(dir, 0700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "test"), []byte("over"), 0600); err != nil {
t.Fatal(err)
}
state := map[string]bool{}
a.checkQuotas(context.Background(), state)
a.checkQuotas(context.Background(), state)
if _, err := a.s.DB.Exec("UPDATE users SET quota=100 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
a.checkQuotas(context.Background(), state)
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action IN ('quota.exceeded','quota.recovered')").Scan(&count); err != nil || count != 2 {
t.Fatalf("transitions=%d error=%v", count, err)
}
}
internal/api/quota_write.go
-63
View File
@@ -1,63 +0,0 @@
package api
import (
"errors"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io/fs"
"os"
"strings"
)
// Call with the owner's user lock held through the write. Runtime writes remain soft-limited.
func (a *API) quotaGrowth(uid string, growth int64) error {
var u store.User
u.ID = uid
if err := a.s.DB.QueryRow("SELECT quota FROM users WHERE id=?", uid).Scan(&u.Quota); err != nil {
return err
}
used, err := a.userUsage(u)
if err != nil {
return err
}
if growth > 0 && (used > u.Quota || growth > u.Quota-used) {
return errors.New("quota exceeded")
}
return nil
}
func quotaTreeBytes(root *os.Root, name string) (int64, error) {
var total int64
entries := 0
err := fs.WalkDir(root.FS(), name, func(p string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
entries++
if entries > 100000 || strings.Count(p, "/") > 128 {
return errors.New("tree limit")
}
info, err := d.Info()
if err != nil {
return err
}
if info.Mode().IsRegular() {
if info.Size() > upgradeSnapshotLimit-total {
return errors.New("snapshot size limit")
}
total += info.Size()
}
return nil
})
return total, err
}
func (a *API) quotaSnapshot(uid string, root *os.Root, extra int64) (int64, error) {
size, err := quotaTreeBytes(root, "home")
if err != nil {
return 0, err
}
if err = a.quotaGrowth(uid, size+extra); err != nil {
return 0, err
}
return size, nil
}
internal/api/quota_write_test.go
-94
View File
@@ -1,94 +0,0 @@
package api
import (
"os"
"path/filepath"
"sync"
"testing"
)
func TestProfileQuotaReplacement(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
profile := filepath.Join(home, "profiles/web/cordis.patch.yml")
if err := os.MkdirAll(filepath.Dir(profile), 0700); err != nil {
t.Fatal(err)
}
configWrite(t, profile, "foo: old")
if _, err := a.s.DB.Exec("UPDATE users SET quota=8 WHERE id=?", i.UID); err != nil {
t.Fatal(err)
}
for _, tc := range []struct {
body string
code int
}{
{`{"content":"foo: biggg"}`, 413},
{`{"content":"foo: new"}`, 200},
{`{"content":"foo: x"}`, 200},
} {
w := configCall(a, i, "PUT", "profile", tc.body)
if w.Code != tc.code {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
}
data, err := os.ReadFile(profile)
if err != nil || string(data) != "foo: x" {
t.Fatalf("%q %v", data, err)
}
}
func TestSnapshotQuotaIncludesResourcesAndExistingSnapshots(t *testing.T) {
a, i, home := configTestAPI(t, false)
configWrite(t, filepath.Join(home, "state"), "1234")
if err := os.MkdirAll(filepath.Join(filepath.Dir(home), "pre-restore-old"), 0700); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(filepath.Dir(home), "pre-restore-old/state"), "1234")
root, err := os.OpenRoot(filepath.Dir(home))
if err != nil {
t.Fatal(err)
}
defer root.Close()
if _, err = a.s.DB.Exec("UPDATE users SET quota=15 WHERE id=?", i.UID); err != nil {
t.Fatal(err)
}
if _, err = a.quotaSnapshot(i.UID, root, 4); err == nil {
t.Fatal("resource bytes omitted")
}
if size, err := a.quotaSnapshot(i.UID, root, 3); err != nil || size != 4 {
t.Fatalf("size=%d err=%v", size, err)
}
if _, err := root.Lstat("pre-restore-old/state"); err != nil {
t.Fatal("existing snapshot changed")
}
}
func TestQuotaSerializedWriters(t *testing.T) {
a, i, home := configTestAPI(t, false)
if _, err := a.s.DB.Exec("UPDATE users SET quota=4 WHERE id=?", i.UID); err != nil {
t.Fatal(err)
}
var wg sync.WaitGroup
accepted := 0
for _, name := range []string{"one", "two"} {
wg.Add(1)
go func() {
defer wg.Done()
user := a.userLocks.Acquire(i.UID)
defer user.Release()
if a.quotaGrowth(i.UID, 4) == nil {
if err := os.WriteFile(filepath.Join(home, name), []byte("1234"), 0600); err != nil {
t.Error(err)
return
}
accepted++
}
}()
}
wg.Wait()
if accepted != 1 {
t.Fatalf("accepted %d writes", accepted)
}
}
internal/api/reauth.go
-45
View File
@@ -1,45 +0,0 @@
package api
import (
"net/http"
"golang.org/x/crypto/bcrypt"
)
func (a *API) reauthenticate(w http.ResponseWriter, uid, password string) bool {
a.authLoginMu.Lock()
defer a.authLoginMu.Unlock()
return a.reauthenticateLocked(w, uid, password, http.StatusForbidden)
}
// The caller holds authLoginMu; never acquire instance or user locks while holding it.
func (a *API) reauthenticateLocked(w http.ResponseWriter, uid, password string, invalidStatus int) bool {
u, err := a.s.UserByID(uid)
if err != nil {
fail(w, 503, "authentication unavailable")
return false
}
locked, err := a.accountLocked(u.Username)
if err != nil {
fail(w, 503, "authentication unavailable")
return false
}
if locked {
fail(w, 429, "account temporarily locked; retry later")
return false
}
hash, err := a.s.Password(u.Username)
if err != nil {
fail(w, 503, "authentication unavailable")
return false
}
if len(password) > 72 || bcrypt.CompareHashAndPassword([]byte(hash), []byte(password)) != nil {
if err = a.failedLogin(u.Username); err != nil {
fail(w, 503, "authentication unavailable")
return false
}
fail(w, invalidStatus, "invalid current password")
return false
}
return true
}
internal/api/reauth_test.go
-335
View File
@@ -1,335 +0,0 @@
package api
import (
"bytes"
"encoding/json"
"io"
"mime/multipart"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
)
func TestReauthSlowUpgradeDoesNotBlockOtherLogin(t *testing.T) {
inspecting, release := make(chan struct{}), make(chan struct{})
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet || !strings.HasSuffix(r.URL.Path, "/containers/old/json") {
t.Errorf("unexpected Docker request: %s %s", r.Method, r.URL.Path)
w.WriteHeader(500)
return
}
close(inspecting)
<-release
w.WriteHeader(503)
})
hash, err := bcrypt.GenerateFromPassword([]byte("original-password"), bcrypt.MinCost)
if err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("UPDATE users SET pass_hash=? WHERE id='owner'", string(hash)); err != nil {
t.Fatal(err)
}
if err := a.s.CreateUser(store.User{ID: "other", Username: "other"}, string(hash)); err != nil {
t.Fatal(err)
}
i := store.Instance{ID: "one", UID: "owner", ContainerID: "old", Status: "exited", Created: time.Now()}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
upgradeDone := make(chan struct{})
go func() {
defer close(upgradeDone)
a.upgrade(httptest.NewRecorder(), runtimeRequest("POST", "/api/instances/one/upgrade", `{"image_id":"sha256:`+strings.Repeat("a", 64)+`","password":"original-password"}`, sid), i)
}()
defer func() {
close(release)
select {
case <-upgradeDone:
case <-time.After(5 * time.Second):
t.Error("upgrade did not finish after releasing Docker")
}
}()
select {
case <-inspecting:
case <-upgradeDone:
t.Fatal("upgrade did not reach Docker after reauthentication")
case <-time.After(5 * time.Second):
t.Fatal("upgrade did not reach inspection")
}
loginDone := make(chan *httptest.ResponseRecorder, 1)
go func() {
w := httptest.NewRecorder()
r := runtimeRequest("POST", "/api/login", resetLogin("other", "original-password"), "")
r.Header.Set("Origin", a.origin)
a.login(w, r)
loginDone <- w
}()
select {
case w := <-loginDone:
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
case <-time.After(5 * time.Second):
t.Fatal("slow upgrade held authentication lock and blocked another user's login")
}
select {
case <-upgradeDone:
t.Fatal("upgrade finished before Docker was released")
default:
}
}
func TestReauthConcurrentFailures(t *testing.T) {
a, i, _ := configTestAPI(t, false)
start := make(chan struct{})
done := make(chan int, 10)
for n := 0; n < 10; n++ {
go func() {
<-start
w := httptest.NewRecorder()
a.reauthenticate(w, i.UID, "wrong")
done <- w.Code
}()
}
close(start)
counts := map[int]int{}
for n := 0; n < 10; n++ {
select {
case code := <-done:
counts[code]++
case <-time.After(5 * time.Second):
t.Fatal("reauthentication deadlocked")
}
}
if counts[403] != 5 || counts[429] != 5 {
t.Fatal(counts)
}
}
func TestReauthOldBackupPasswordIndependent(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
configWrite(t, filepath.Join(home, ".credentials.yaml"), "original")
w := configCall(a, i, "POST", "backups", `{"password":"original-password"}`)
var result struct {
ID string `json:"id"`
}
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &result) != nil || result.ID == "" {
t.Fatal(w.Code, w.Body.String())
}
if _, err := a.s.DB.Exec("UPDATE users SET pass_hash='changed' WHERE id=?", i.UID); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO auth_failures VALUES('owner',5,?)", time.Now().Add(time.Hour).Unix()); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(home, ".credentials.yaml"), "changed")
w = configCall(a, i, "POST", "restore", `{"backup_id":"`+result.ID+`","password":"original-password"}`)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
content, err := os.ReadFile(filepath.Join(home, ".credentials.yaml"))
if err != nil || string(content) != "original" {
t.Fatal(string(content), err)
}
var failures int
if err := a.s.DB.QueryRow("SELECT failures FROM auth_failures WHERE username='owner'").Scan(&failures); err != nil || failures != 5 {
t.Fatal(failures, err)
}
}
func TestReauthSharedFailureLock(t *testing.T) {
a, i, _ := configTestAPI(t, false)
sid, _, err := a.s.Session(i.UID)
if err != nil {
t.Fatal(err)
}
registerTestAccess(t, a, sid)
backup := func(password string) *httptest.ResponseRecorder {
return configCall(a, i, "POST", "backups", `{"password":"`+password+`"}`)
}
upgrade := func(password string) *httptest.ResponseRecorder {
w := httptest.NewRecorder()
a.upgrade(w, runtimeRequest("POST", "/api/instances/one/upgrade", `{"image_id":"sha256:`+strings.Repeat("a", 64)+`","password":"`+password+`"}`, sid), i)
return w
}
for _, call := range []func(string) *httptest.ResponseRecorder{backup, upgrade, backup, upgrade} {
if w := call("wrong"); w.Code != 403 {
t.Fatal(w.Code, w.Body.String())
}
}
w := httptest.NewRecorder()
a.changePassword(w, runtimeRequest("POST", changePath, changeBody("wrong", resetSecret), sid))
if w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
for _, call := range []func(string) *httptest.ResponseRecorder{backup, upgrade} {
if w := call("original-password"); w.Code != 429 {
t.Fatal(w.Code, w.Body.String())
}
}
w = httptest.NewRecorder()
r := runtimeRequest("POST", "/api/login", resetLogin("owner", "original-password"), "")
r.Header.Set("Origin", a.origin)
a.login(w, r)
if w.Code != 429 {
t.Fatal(w.Code, w.Body.String())
}
var failures int
if err := a.s.DB.QueryRow("SELECT failures FROM auth_failures WHERE username='owner'").Scan(&failures); err != nil || failures != 5 {
t.Fatal(failures, err)
}
if _, err := a.s.DB.Exec("UPDATE auth_failures SET locked_until=?", time.Now().Add(-time.Minute).Unix()); err != nil {
t.Fatal(err)
}
if w := upgrade("wrong"); w.Code != 403 {
t.Fatal(w.Code, w.Body.String())
}
if err := a.s.DB.QueryRow("SELECT failures FROM auth_failures WHERE username='owner'").Scan(&failures); err != nil || failures != 1 {
t.Fatal(failures, err)
}
}
func TestReauthFailureStorageUnavailable(t *testing.T) {
a, i, _ := configTestAPI(t, false)
if _, err := a.s.DB.Exec("DROP TABLE auth_failures"); err != nil {
t.Fatal(err)
}
w := configCall(a, i, "POST", "backups", `{"password":"original-password"}`)
if w.Code != 503 {
t.Fatal(w.Code, w.Body.String())
}
}
type quotaPausedBody struct {
io.Reader
started chan struct{}
resume chan struct{}
}
func (b *quotaPausedBody) Read(p []byte) (int, error) {
if b.started != nil {
close(b.started)
b.started = nil
<-b.resume
}
return b.Reader.Read(p)
}
func TestReauthQuotaInFlightUpload(t *testing.T) {
a, sid := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) {})
if _, err := a.s.DB.Exec("UPDATE users SET admin=1,quota=1048576 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
u, err := a.s.UserByID("owner")
if err != nil {
t.Fatal(err)
}
dir := filepath.Join(a.root, "workspaces", u.ID)
if err := os.MkdirAll(dir, 0700); err != nil {
t.Fatal(err)
}
root, err := os.OpenRoot(dir)
if err != nil {
t.Fatal(err)
}
defer root.Close()
var body bytes.Buffer
form := multipart.NewWriter(&body)
part, err := form.CreateFormFile("file", "large.txt")
if err != nil {
t.Fatal(err)
}
part.Write(bytes.Repeat([]byte("x"), 4096))
form.Close()
started, resume := make(chan struct{}), make(chan struct{})
r := httptest.NewRequest("POST", "/api/ws/upload", &quotaPausedBody{Reader: &body, started: started, resume: resume})
r.Header.Set("Content-Type", form.FormDataContentType())
done := make(chan *httptest.ResponseRecorder, 1)
go func() {
w := httptest.NewRecorder()
a.wsUpload(w, r, root, u)
done <- w
}()
select {
case <-started:
case <-time.After(5 * time.Second):
close(resume)
t.Fatal("upload did not start reading")
}
w := httptest.NewRecorder()
a.users(w, runtimeRequest("PATCH", "/api/users", `{"id":"owner","quota_bytes":1024}`, sid))
close(resume)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
select {
case w := <-done:
if w.Code != 413 {
t.Fatal(w.Code, w.Body.String())
}
case <-time.After(5 * time.Second):
t.Fatal("upload deadlocked")
}
entries, err := os.ReadDir(dir)
if err != nil || len(entries) != 0 {
t.Fatal("rejected upload left files", entries, err)
}
}
func TestReauthQuotaUpdateUsesUserLock(t *testing.T) {
a, sid := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) {})
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='owner'"); err != nil {
t.Fatal(err)
}
lock := a.userLocks.Acquire("owner")
defer func() {
if lock != nil {
lock.Release()
}
}()
done := make(chan *httptest.ResponseRecorder, 1)
go func() {
w := httptest.NewRecorder()
a.users(w, runtimeRequest("PATCH", "/api/users", `{"id":"owner","quota_bytes":1024}`, sid))
done <- w
}()
deadline := time.Now().Add(5 * time.Second)
for {
a.userLocks.mu.Lock()
refs := lock.refs
a.userLocks.mu.Unlock()
if refs == 2 {
break
}
select {
case w := <-done:
t.Fatal("quota update bypassed user lock", w.Code)
default:
}
if time.Now().After(deadline) {
t.Fatal("quota update did not wait on user lock")
}
runtime.Gosched()
}
lock.Release()
lock = nil
select {
case w := <-done:
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
case <-time.After(5 * time.Second):
t.Fatal("quota update deadlocked")
}
}
internal/api/reconcile.go
-141
View File
@@ -1,141 +0,0 @@
package api
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
"os"
)
// Reconcile checks persisted identities against Docker without deleting unknown
// containers. Orphans require explicit administrator review, never blind adoption.
// Runs before the HTTP server starts, so no request can race this pass.
func (a *API) Reconcile(ctx context.Context) error {
filters, _ := json.Marshal(map[string][]string{"label": {"dsh-mgr.instance"}})
var containers []struct {
ID string
State string
Labels map[string]string
}
if err := a.docker.Do(ctx, "GET", "/containers/json?all=1&filters="+url.QueryEscape(string(filters)), nil, &containers); err != nil {
return err
}
rows, err := a.s.DB.QueryContext(ctx, "SELECT id,uid,container_id,status FROM instances")
if err != nil {
return err
}
type record struct{ id, uid, container, status string }
var records []record
for rows.Next() {
var r record
if err = rows.Scan(&r.id, &r.uid, &r.container, &r.status); err != nil {
break
}
records = append(records, r)
}
if err == nil {
err = rows.Err()
}
rows.Close()
if err != nil {
return err
}
known := make(map[string]record)
for _, r := range records {
if r.container == "" && (r.status == "creating" || r.status == "error") {
i, loadErr := a.s.GetInstance(r.id)
if loadErr != nil {
return loadErr
}
if recoveryErr := a.recoverRuntimeContainer(ctx, &i); recoveryErr != nil {
a.s.AddAudit(r.uid, "reconcile.recovery_error", r.id, recoveryErr.Error())
}
r.container = i.ContainerID
r.status = "error"
if _, err = a.s.DB.ExecContext(ctx, "UPDATE instances SET status='error' WHERE id=?", r.id); err != nil {
return err
}
}
known[r.id] = r
}
matched := make(map[string]bool)
panel := os.Getenv("PANEL_CONTAINER_ID")
var networkErrors []error
for _, c := range containers {
id := c.Labels["dsh-mgr.instance"]
r, ok := known[id]
if !ok || r.container != c.ID || r.uid != c.Labels["dsh-mgr.owner"] {
a.s.AddAudit("", "reconcile.orphan", c.ID, "container identity requires administrator review")
continue
}
matched[id] = true
if panel != "" {
if networkErr := a.reconcilePanelNetwork(ctx, id, panel); networkErr != nil {
if _, err = a.s.DB.ExecContext(ctx, "UPDATE instances SET status='error' WHERE id=?", id); err != nil {
return err
}
a.s.AddAudit(r.uid, "reconcile.network_error", id, networkErr.Error())
networkErrors = append(networkErrors, fmt.Errorf("instance %s: %w", id, networkErr))
continue
}
}
if r.status != "error" {
if _, err = a.s.DB.ExecContext(ctx, "UPDATE instances SET status=? WHERE id=?", c.State, id); err != nil {
return err
}
}
}
for _, r := range records {
if r.container == "" || matched[r.id] || r.status == "error" {
continue
}
if _, err = a.s.DB.ExecContext(ctx, "UPDATE instances SET status='error' WHERE id=?", r.id); err != nil {
return err
}
a.s.AddAudit(r.uid, "reconcile.missing", r.id, "recorded container is absent or identity mismatch")
}
if ctx.Err() != nil {
return fmt.Errorf("reconcile: %w", ctx.Err())
}
return errors.Join(networkErrors...)
}
func (a *API) reconcilePanelNetwork(ctx context.Context, id, panel string) error {
name := runtimeNetwork(id)
var network struct {
ID, Name, Driver string
Labels map[string]string
}
if err := a.docker.Do(ctx, "GET", "/networks/"+url.PathEscape(name), nil, &network); err != nil {
return fmt.Errorf("inspect instance network: %w", err)
}
if network.ID == "" || network.Name != name || network.Driver != "bridge" || network.Labels["dsh-mgr.instance"] != id {
return fmt.Errorf("instance network identity mismatch")
}
container, err := a.docker.Inspect(ctx, panel)
if err != nil {
return fmt.Errorf("inspect panel: %w", err)
}
if container.ID == "" {
return fmt.Errorf("panel container identity missing")
}
for _, endpoint := range container.NetworkSettings.Networks {
if endpoint.NetworkID == network.ID {
return nil
}
}
if err := a.docker.Do(ctx, "POST", "/networks/"+url.PathEscape(network.ID)+"/connect", map[string]string{"Container": container.ID}, nil); err != nil {
// Another reconciliation may have connected the panel after inspection.
if current, inspectErr := a.docker.Inspect(ctx, container.ID); inspectErr == nil && current.ID == container.ID {
for _, endpoint := range current.NetworkSettings.Networks {
if endpoint.NetworkID == network.ID {
return nil
}
}
}
return fmt.Errorf("connect panel network: %w", err)
}
return nil
}
internal/api/reconcile_test.go
-141
View File
@@ -1,141 +0,0 @@
package api
import (
"context"
"encoding/json"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"net/http"
"testing"
"time"
)
func TestReconcileIdentityAndMissing(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" {
t.Fatalf("unexpected destructive request %s", r.Method)
}
w.Write([]byte(`[{"Id":"c1","State":"running","Labels":{"dsh-mgr.instance":"one","dsh-mgr.owner":"owner"}},{"Id":"c3","State":"running","Labels":{"dsh-mgr.instance":"orphan","dsh-mgr.owner":"owner"}}]`))
})
for _, i := range []store.Instance{{ID: "one", ContainerID: "c1", Status: "exited"}, {ID: "missing", ContainerID: "c2", Status: "running"}, {ID: "locked", ContainerID: "c4", Status: "error"}} {
i.UID = "owner"
i.Created = time.Now()
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
}
if err := a.Reconcile(context.Background()); err != nil {
t.Fatal(err)
}
for id, want := range map[string]string{"one": "running", "missing": "error", "locked": "error"} {
i, err := a.s.GetInstance(id)
if err != nil || i.Status != want {
t.Fatalf("%s: %+v %v", id, i, err)
}
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='reconcile.orphan'").Scan(&count); err != nil || count != 1 {
t.Fatalf("orphan audit: %d %v", count, err)
}
}
func TestReconcilePanelNetwork(t *testing.T) {
for _, tc := range []struct {
name string
connected, race, connectFail, missing, foreign, wrongDriver, panelFail bool
wantConnect int
wantError bool
}{
{name: "already connected", connected: true},
{name: "reconnect", wantConnect: 1},
{name: "concurrent connection", race: true, wantConnect: 1},
{name: "connect failure", connectFail: true, wantConnect: 1, wantError: true},
{name: "missing network", missing: true, wantError: true},
{name: "foreign network", foreign: true, wantError: true},
{name: "not bridge", wrongDriver: true, wantError: true},
{name: "panel inspect failure", panelFail: true, wantError: true},
} {
t.Run(tc.name, func(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "configured-panel")
connects := 0
connected := tc.connected
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch r.Method + " " + r.URL.Path {
case "GET /v1.47/containers/json":
fmt.Fprint(w, `[{"Id":"c1","State":"running","Labels":{"dsh-mgr.instance":"one","dsh-mgr.owner":"owner"}},{"Id":"foreign","State":"running","Labels":{"dsh-mgr.instance":"foreign","dsh-mgr.owner":"other"}},{"Id":"unknown","State":"running","Labels":{"dsh-mgr.instance":"unknown","dsh-mgr.owner":"owner"}}]`)
case "GET /v1.47/networks/" + runtimeNetwork("one"):
if tc.missing {
http.Error(w, "network not found", http.StatusNotFound)
return
}
owner, driver := "one", "bridge"
if tc.foreign {
owner = "other"
}
if tc.wrongDriver {
driver = "host"
}
json.NewEncoder(w).Encode(map[string]any{"Id": "network-id", "Name": runtimeNetwork("one"), "Driver": driver, "Labels": map[string]string{"dsh-mgr.instance": owner}})
case "GET /v1.47/containers/configured-panel/json", "GET /v1.47/containers/panel-full-id/json":
if tc.panelFail {
http.Error(w, "panel missing", http.StatusNotFound)
return
}
networkID := "unrelated-network"
if connected {
networkID = "network-id"
}
fmt.Fprintf(w, `{"Id":"panel-full-id","NetworkSettings":{"Networks":{"bridge":{"NetworkID":%q}}}}`, networkID)
case "POST /v1.47/networks/network-id/connect":
connects++
var body map[string]string
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || len(body) != 1 || body["Container"] != "panel-full-id" {
t.Errorf("unsafe connection body: %v, %v", body, err)
}
if tc.connectFail || tc.race {
connected = tc.race
http.Error(w, "connection failed", http.StatusForbidden)
return
}
connected = true
w.WriteHeader(http.StatusOK)
default:
t.Errorf("unexpected request: %s %s", r.Method, r.URL.Path)
http.Error(w, "unexpected request", http.StatusBadRequest)
}
})
for _, i := range []store.Instance{
{ID: "one", UID: "owner", ContainerID: "c1", Status: "exited", Created: time.Now()},
{ID: "foreign", UID: "owner", ContainerID: "foreign", Status: "running", Created: time.Now()},
} {
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
}
if err := a.Reconcile(context.Background()); (err != nil) != tc.wantError {
t.Fatalf("Reconcile error = %v, want error %v", err, tc.wantError)
}
if !tc.wantError {
if err := a.Reconcile(context.Background()); err != nil {
t.Fatal(err)
}
}
if connects != tc.wantConnect {
t.Fatalf("connections = %d, want %d", connects, tc.wantConnect)
}
wantStatus, wantAudits := "running", 0
if tc.wantError {
wantStatus, wantAudits = "error", 1
}
i, err := a.s.GetInstance("one")
if err != nil || i.Status != wantStatus {
t.Fatalf("instance: %+v %v", i, err)
}
var audits int
if err := a.s.DB.QueryRow("SELECT count(*) FROM audit WHERE action='reconcile.network_error'").Scan(&audits); err != nil || audits != wantAudits {
t.Fatalf("network audits: %d %v", audits, err)
}
})
}
}
internal/api/refresh_recovery_test.go
-59
View File
@@ -1,59 +0,0 @@
package api
import (
"net/http"
"net/http/httptest"
"strings"
"testing"
"time"
)
func TestRefreshLostResponseRecovery(t *testing.T) {
a, req := authFixture(t)
w := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil)
old := w.Result().Cookies()
csrf, sid := cookieValue(old, "dsh_csrf"), cookieValue(old, "dsh_session")
w = req("POST", "/api/refresh", "", csrf, old)
if w.Code != 200 {
t.Fatal(w.Code)
}
successor := cookieValue(w.Result().Cookies(), "dsh_refresh")
// Lose the response, expire access and restart the server: recovery is durable
// and does not depend on an active access cookie or a short grace timer.
for _, c := range old {
if c.Name == "dsh_access" {
c.Value = a.accessJWT(sid, "alice", time.Now().Add(-time.Hour))
}
}
b := New(a.s, a.root)
b.origin = a.origin
r := httptest.NewRequest("POST", "/api/refresh", strings.NewReader(""))
r.Header.Set("Origin", a.origin)
r.Header.Set("X-CSRF-Token", csrf)
for _, c := range old {
r.AddCookie(c)
}
w = httptest.NewRecorder()
b.Handler().ServeHTTP(w, r)
if w.Code != 200 || cookieValue(w.Result().Cookies(), "dsh_refresh") != successor {
t.Fatal("lost response recovery failed", w.Code)
}
current := w.Result().Cookies()
w = req("POST", "/api/refresh", "", csrf, current)
if w.Code != 200 {
t.Fatal(w.Code)
}
if w = req("POST", "/api/refresh", "", csrf, old); w.Code != 401 || len(w.Result().Cookies()) != 0 {
t.Fatal("two-generation-old token accepted", w.Code)
}
bad := []*http.Cookie{{Name: "dsh_session", Value: sid}, {Name: "dsh_refresh", Value: "invalid"}}
if w = req("POST", "/api/refresh", "", csrf, bad); w.Code != 401 {
t.Fatal(w.Code)
}
if _, err := a.s.DB.Exec("UPDATE sessions SET expires=? WHERE id=?", time.Now().Unix(), sid); err != nil {
t.Fatal(err)
}
if w = req("POST", "/api/refresh", "", csrf, current); w.Code == 200 || len(w.Result().Cookies()) != 0 {
t.Fatal("expired session recovered")
}
}
internal/api/releases.go
-198
View File
@@ -1,198 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"net/http"
"strconv"
"time"
)
var fetchRelease = readRelease
func (a *API) latestRelease(w http.ResponseWriter, r *http.Request) {
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
repository, branch, err := releaseSource()
if err != nil {
fail(w, 500, err.Error())
return
}
info, _, err := fetchRelease(r.Context(), repository, "refs/heads/"+branch)
if err != nil && !errors.Is(err, errReleaseIncompatible) {
fail(w, 502, sanitizeBuildLog(err.Error()))
return
}
compatible := err == nil
reason := "trusted release supports current layout and platform"
if !compatible {
reason = errReleaseIncompatible.Error()
}
buildQueue.Lock()
err = a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
fail(w, 500, "database error")
return
}
var b buildRecord
err = scanBuildSummary(a.s.DB.QueryRow("SELECT "+buildSummaryColumns+" FROM image_builds WHERE source_repository=? AND source_commit=? AND release_id=? AND platform=? ORDER BY created_at DESC,id DESC LIMIT 1", repository, info.Commit, info.ReleaseID, imagePlatform), &b)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
fail(w, 500, "database error")
return
}
result := map[string]any{"release_id": info.ReleaseID, "version": info.Version, "commit": info.Commit, "repository": repository, "branch": branch, "recipe_rev": strconv.Itoa(info.DeploymentSchema), "platforms": info.Platforms, "compatible": compatible, "reason": reason}
if err == nil {
result["build"] = b
}
writeJSON(w, result)
}
func (a *API) releaseBuild(w http.ResponseWriter, r *http.Request) {
u, _, err := a.user(r)
if err != nil {
fail(w, 401, "authentication required")
return
}
if !u.Admin {
fail(w, 403, "administrator required")
return
}
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
var in struct {
Commit string `json:"commit"`
}
dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 4096))
dec.DisallowUnknownFields()
if dec.Decode(&in) != nil || dec.Decode(new(any)) != io.EOF || !releaseCommit.MatchString(in.Commit) {
fail(w, 400, "fixed commit required")
return
}
repository, branch, err := releaseSource()
if err != nil {
fail(w, 500, err.Error())
return
}
info, _, err := fetchRelease(r.Context(), repository, "refs/heads/"+branch)
if err != nil {
fail(w, 502, sanitizeBuildLog(err.Error()))
return
}
if info.Commit != in.Commit {
fail(w, 409, "release changed; check the latest release again")
return
}
b, err := a.enqueueRelease(u.ID, info)
if err != nil {
status := 500
if errors.Is(err, errBuildRateLimited) {
status = 429
}
fail(w, status, sanitizeBuildLog(err.Error()))
return
}
a.startBuildWorker()
w.WriteHeader(http.StatusAccepted)
writeJSON(w, b)
}
func (a *API) enqueueRelease(uid string, info releaseInfo) (buildRecord, error) {
buildQueue.Lock()
defer buildQueue.Unlock()
var b buildRecord
if err := a.ensureBuilds(); err != nil {
return b, err
}
err := scanBuildSummary(a.s.DB.QueryRow("SELECT "+buildSummaryColumns+" FROM image_builds WHERE source_repository=? AND source_commit=? AND release_id=? AND platform=? AND status IN ('queued','running','validating','success')", info.Repository, info.Commit, info.ReleaseID, imagePlatform), &b)
if err == nil {
if b.Status == "success" {
buildQueue.Unlock()
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
matches, verifyErr := a.buildImageMatches(ctx, b.ImageID, b)
cancel()
buildQueue.Lock()
if verifyErr != nil {
return b, verifyErr
}
if !matches {
if _, err = a.s.DB.Exec("UPDATE image_builds SET status='missing',error='image no longer available' WHERE id=? AND status='success'", b.ID); err != nil {
return b, err
}
buildQueue.Unlock()
b, err = a.enqueueRelease(uid, info)
buildQueue.Lock()
return b, err
}
}
return b, nil
}
if !errors.Is(err, sql.ErrNoRows) {
return b, err
}
var count int
if err = a.s.DB.QueryRow("SELECT count(*) FROM image_build_requests WHERE user_id=? AND created_at>?", uid, time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano)).Scan(&count); err != nil {
return b, err
}
if count >= 3 {
return b, errBuildRateLimited
}
b = buildRecord{ID: store.ID(), Version: info.Version, Status: "queued", Created: time.Now().UTC().Format(time.RFC3339Nano), RequestedBy: uid, RecipeRev: imageRecipeRev, Platform: imagePlatform, ReleaseID: info.ReleaseID, SourceRepository: info.Repository, SourceCommit: info.Commit}
return b, a.insertBuildRequest(b, uid)
}
func (a *API) releaseBuildContext(ctx context.Context, b buildRecord) (io.Reader, error) {
repository, _, err := releaseSource()
if err != nil {
return nil, err
}
if repository != b.SourceRepository || !releaseCommit.MatchString(b.SourceCommit) {
return nil, errors.New("queued release source is no longer trusted")
}
u, err := a.s.UserByID(b.RequestedBy)
if err != nil || !u.Admin {
return nil, errors.New("release requester is not an administrator")
}
info, archive, err := fetchRelease(ctx, repository, b.SourceCommit)
if err != nil {
return nil, err
}
if info.Commit != b.SourceCommit || info.ReleaseID != b.ReleaseID || info.Version != b.Version {
return nil, errors.New("pinned release manifest mismatch")
}
return archive, nil
}
func releaseLabelsMatch(im *inspectedImage, b buildRecord) bool {
if b.ReleaseID == "" {
return im.Config.Labels["dsh-mgr.release"] == ""
}
return im.Config.Labels["dsh-mgr.release"] == b.ReleaseID && im.Config.Labels["dsh-mgr.source"] == b.SourceRepository && im.Config.Labels["dsh-mgr.commit"] == b.SourceCommit
}
func (a *API) buildImageMatches(ctx context.Context, id string, b buildRecord) (bool, error) {
matches, err := a.imageMatches(ctx, id, b.Version)
if err != nil || !matches {
return matches, err
}
im, err := a.inspectLocalImage(ctx, id)
if err != nil || im == nil {
return false, err
}
return releaseLabelsMatch(im, b), nil
}
internal/api/releases_git.go
-217
View File
@@ -1,217 +0,0 @@
package api
import (
"archive/tar"
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/url"
"os"
"os/exec"
"path"
"regexp"
"strings"
"syscall"
"time"
)
type releaseManifest struct {
SchemaVersion int `json:"schema_version"`
ReleaseID string `json:"release_id"`
Version string `json:"version"`
DeploymentSchema int `json:"deployment_schema"`
Platforms []string `json:"platforms"`
}
type releaseInfo struct {
releaseManifest
Repository string `json:"source_repository"`
Commit string `json:"source_commit"`
}
var errReleaseIncompatible = errors.New("release layout or platform is incompatible")
var releaseCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
var releaseID = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`)
const releaseLimit = 32 << 20
func releaseSource() (string, string, error) {
repository := os.Getenv("DSH_RELEASE_REPOSITORY")
if repository == "" {
repository = "https://git.misaka.ren/m1saka/dsh-web.git"
}
branch := os.Getenv("DSH_RELEASE_BRANCH")
if branch == "" {
branch = "dockerfile"
}
u, err := url.Parse(repository)
if err != nil || u.Scheme != "https" || u.Hostname() == "" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || strings.ContainsAny(repository, "\r\n") || !regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._/-]*$`).MatchString(branch) || strings.Contains(branch, "..") {
return "", "", errors.New("invalid trusted release source configuration")
}
return repository, branch, nil
}
type boundedReleaseBuffer struct{ bytes.Buffer }
func (b *boundedReleaseBuffer) Write(p []byte) (int, error) {
if b.Len()+len(p) > releaseLimit {
return 0, errors.New("release exceeds size limit")
}
return b.Buffer.Write(p)
}
func gitReleaseCommand(ctx context.Context, dir string, args ...string) ([]byte, error) {
gitArgs := append([]string{"-c", "credential.helper=", "-c", "http.followRedirects=false", "-c", "core.hooksPath=/dev/null", "-c", "protocol.allow=never", "-c", "protocol.https.allow=always", "-c", "fetch.unpackLimit=1"}, args...)
// The inherited file-size limit also bounds Git's on-disk pack, not just stdout.
cmd := exec.CommandContext(ctx, "/bin/sh", append([]string{"-c", "ulimit -f 32768; exec git \"$@\"", "git"}, gitArgs...)...)
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error { return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL) }
cmd.Dir = dir
cmd.Env = []string{"PATH=" + os.Getenv("PATH"), "GIT_CONFIG_NOSYSTEM=1", "GIT_CONFIG_GLOBAL=/dev/null", "GIT_TERMINAL_PROMPT=0", "GIT_LFS_SKIP_SMUDGE=1"}
cmd.WaitDelay = time.Second
var out boundedReleaseBuffer
cmd.Stdout = &out
if err := cmd.Run(); err != nil {
return nil, errors.New("trusted release Git operation failed")
}
return out.Bytes(), nil
}
var releaseFetchSlot = make(chan struct{}, 1)
func readRelease(ctx context.Context, repository, ref string) (releaseInfo, io.Reader, error) {
var info releaseInfo
ctx, cancel := context.WithTimeout(ctx, 45*time.Second)
defer cancel()
select {
case releaseFetchSlot <- struct{}{}:
defer func() { <-releaseFetchSlot }()
case <-ctx.Done():
return info, nil, ctx.Err()
}
dir, err := os.MkdirTemp("", "dsh-release-")
if err != nil {
return info, nil, err
}
defer os.RemoveAll(dir)
if _, err = gitReleaseCommand(ctx, dir, "init", "--quiet"); err != nil {
return info, nil, err
}
if _, err = gitReleaseCommand(ctx, dir, "fetch", "--quiet", "--depth=1", "--no-tags", repository, ref); err != nil {
return info, nil, err
}
commit, err := gitReleaseCommand(ctx, dir, "rev-parse", "FETCH_HEAD^{commit}")
if err != nil {
return info, nil, err
}
info.Commit = strings.TrimSpace(string(commit))
info.Repository = repository
if !releaseCommit.MatchString(info.Commit) || (releaseCommit.MatchString(ref) && info.Commit != ref) {
return info, nil, errors.New("release commit mismatch")
}
archive, err := gitReleaseCommand(ctx, dir, "archive", "--format=tar", info.Commit, "--", "release.json", "image")
if err != nil {
return info, nil, err
}
return parseReleaseArchive(info, archive)
}
func parseReleaseArchive(info releaseInfo, archive []byte) (releaseInfo, io.Reader, error) {
var manifest []byte
tr := tar.NewReader(bytes.NewReader(archive))
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return info, nil, err
}
if h.Name == "release.json" && h.Typeflag == tar.TypeReg {
if h.Size > 16384 {
return info, nil, errors.New("release manifest too large")
}
manifest, err = io.ReadAll(tr)
if err != nil {
return info, nil, err
}
}
}
dec := json.NewDecoder(bytes.NewReader(manifest))
dec.DisallowUnknownFields()
if dec.Decode(&info.releaseManifest) != nil || dec.Decode(new(any)) != io.EOF {
return info, nil, errors.New("invalid release manifest")
}
m := info.releaseManifest
compatible := false
for _, p := range m.Platforms {
if p == imagePlatform {
compatible = true
}
}
if m.SchemaVersion != 1 || !releaseID.MatchString(m.ReleaseID) || !validBuildVersion(m.Version) || m.DeploymentSchema < 1 || len(m.Platforms) == 0 {
return info, nil, errors.New("unsupported release manifest or platform")
}
if m.DeploymentSchema != 7 || !compatible {
return info, nil, errReleaseIncompatible
}
var out bytes.Buffer
tw := tar.NewWriter(&out)
tr = tar.NewReader(bytes.NewReader(archive))
dockerfile := false
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return info, nil, err
}
name := h.Name
if !strings.HasPrefix(name, "image/") {
continue
}
name = strings.TrimPrefix(name, "image/")
if name == "" {
continue
}
if path.IsAbs(name) || path.Clean(strings.TrimSuffix(name, "/")) != strings.TrimSuffix(name, "/") || name == ".." || strings.HasPrefix(name, "../") {
return info, nil, errors.New("unsafe release context path")
}
excluded := false
for _, part := range strings.Split(name, "/") {
if part == ".git" || part == ".env" || strings.HasPrefix(part, ".env.") || part == ".netrc" || part == ".npmrc" || part == ".ssh" || part == ".git-credentials" {
excluded = true
}
}
if excluded {
continue
}
if h.Typeflag != tar.TypeReg && h.Typeflag != tar.TypeDir {
return info, nil, errors.New("release context links and special files are not supported")
}
h.Name = name
h.PAXRecords = nil
h.Uname = ""
h.Gname = ""
h.Uid = 0
h.Gid = 0
if name == "Dockerfile" && h.Typeflag == tar.TypeReg {
dockerfile = true
}
if err = tw.WriteHeader(h); err != nil {
return info, nil, err
}
if _, err = io.Copy(tw, tr); err != nil {
return info, nil, err
}
}
if !dockerfile {
return info, nil, errors.New("release context Dockerfile missing")
}
if err := tw.Close(); err != nil {
return info, nil, err
}
return info, bytes.NewReader(out.Bytes()), nil
}
internal/api/releases_test.go
-318
View File
@@ -1,318 +0,0 @@
package api
import (
"archive/tar"
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/http"
"strings"
"testing"
)
func testReleaseArchive(t *testing.T, m string, files map[string]string) []byte {
t.Helper()
var b bytes.Buffer
tw := tar.NewWriter(&b)
files["release.json"] = m
for name, data := range files {
if err := tw.WriteHeader(&tar.Header{Name: name, Size: int64(len(data)), Mode: 0644}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write([]byte(data)); err != nil {
t.Fatal(err)
}
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
func releaseFixture() releaseInfo {
return releaseInfo{releaseManifest: releaseManifest{SchemaVersion: 1, ReleaseID: "runtime-7.1", Version: "0.2.0-rc.3", DeploymentSchema: 7, Platforms: []string{imagePlatform}}, Repository: "https://git.misaka.ren/m1saka/dsh-web.git", Commit: strings.Repeat("a", 40)}
}
func TestReleaseManifestAndContext(t *testing.T) {
info := releaseFixture()
data, _ := json.Marshal(info.releaseManifest)
for _, tc := range []struct {
name, manifest string
files map[string]string
wantErr bool
}{
{"valid", string(data), map[string]string{"image/Dockerfile": "FROM scratch", "image/nested/helper": "complete", "image/.git/config": "secret", "image/.env": "secret"}, false},
{"malformed", "{", map[string]string{"image/Dockerfile": "FROM scratch"}, true},
{"unknown", strings.Replace(string(data), "schema_version", "unknown", 1), map[string]string{"image/Dockerfile": "FROM scratch"}, true},
{"incompatible", strings.Replace(string(data), "deployment_schema\":7", "deployment_schema\":8", 1), map[string]string{"image/Dockerfile": "FROM scratch"}, true},
{"missing Dockerfile", string(data), map[string]string{"image/helper": "x"}, true},
{"unsafe path", string(data), map[string]string{"image/Dockerfile": "FROM scratch", "image/../escape": "x"}, true},
} {
t.Run(tc.name, func(t *testing.T) {
_, r, err := parseReleaseArchive(info, testReleaseArchive(t, tc.manifest, tc.files))
if (err != nil) != tc.wantErr {
t.Fatalf("err=%v", err)
}
if err != nil {
return
}
tr := tar.NewReader(r)
seen := map[string]bool{}
for {
h, e := tr.Next()
if e == io.EOF {
break
}
if e != nil {
t.Fatal(e)
}
seen[h.Name] = true
}
if !seen["nested/helper"] || seen[".env"] || seen[".git/config"] {
t.Fatal(seen)
}
})
}
}
func TestReleaseQueueIdentityRecoveryAndFailure(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(404) })
info := releaseFixture()
b, err := a.enqueueRelease("u", info)
if err != nil {
t.Fatal(err)
}
again, err := a.enqueueRelease("u", info)
if err != nil || again.ID != b.ID {
t.Fatalf("repeat %v %v", again, err)
}
info.Commit = strings.Repeat("b", 40)
info.ReleaseID = "runtime-7.2"
next, err := a.enqueueRelease("u", info)
if err != nil || next.ID == b.ID || next.Version != b.Version {
t.Fatalf("next %v %v", next, err)
}
buildQueue.Lock()
delete(buildQueue.initialized, a.s.DB)
err = a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
t.Fatal(err)
}
var recovered buildRecord
if err = scanBuildSummary(a.s.DB.QueryRow("SELECT "+buildSummaryColumns+" FROM image_builds WHERE id=?", next.ID), &recovered); err != nil {
t.Fatal(err)
}
if recovered.Status != "queued" || recovered.SourceCommit != info.Commit {
t.Fatal(recovered)
}
if _, err = a.s.DB.Exec("UPDATE image_builds SET status='failed' WHERE id=?", next.ID); err != nil {
t.Fatal(err)
}
retry, err := a.enqueueRelease("u", info)
if err != nil || retry.ID == next.ID {
t.Fatalf("retry %v %v", retry, err)
}
}
func TestReleaseEndpointsAuthorizationFailureAndMismatch(t *testing.T) {
a, sid := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(404) })
if w := imageRequest(a.latestRelease, sid, "GET", ""); w.Code != 403 {
t.Fatal(w.Code)
}
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='u'"); err != nil {
t.Fatal(err)
}
old := fetchRelease
t.Cleanup(func() { fetchRelease = old })
fetchRelease = func(context.Context, string, string) (releaseInfo, io.Reader, error) {
return releaseInfo{}, nil, errors.New("network failure")
}
if w := imageRequest(a.latestRelease, sid, "GET", ""); w.Code != 502 {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
fetchRelease = func(context.Context, string, string) (releaseInfo, io.Reader, error) {
return releaseFixture(), nil, nil
}
if w := imageRequest(a.releaseBuild, sid, "POST", `{"commit":"bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"}`); w.Code != 409 {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
fetchRelease = func(context.Context, string, string) (releaseInfo, io.Reader, error) {
return releaseFixture(), nil, errReleaseIncompatible
}
if w := imageRequest(a.latestRelease, sid, "GET", ""); w.Code != 200 || !strings.Contains(w.Body.String(), `"compatible":false`) {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
}
func TestReleaseGitEnvironmentAndFailure(t *testing.T) {
out, err := gitReleaseCommand(context.Background(), t.TempDir(), "--version")
if err != nil || !strings.HasPrefix(string(out), "git version") {
t.Fatalf("%q %v", out, err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if _, err = gitReleaseCommand(ctx, t.TempDir(), "--version"); err == nil {
t.Fatal("cancel ignored")
}
for _, repository := range []string{"http://example.test/repo", "https://user:[email protected]/repo", "https://example.test/repo?token=secret"} {
t.Setenv("DSH_RELEASE_REPOSITORY", repository)
if _, _, err = releaseSource(); err == nil {
t.Fatal("accepted unsafe source")
}
}
}
func TestPinnedReleaseWorkerAdmissionAndBlock(t *testing.T) {
testPinnedReleaseWorkerAdmissionAndBlock(t, "")
}
func TestPinnedReleaseWorkerARM64V8AdmissionAndBlock(t *testing.T) {
previous := imagePlatform
imagePlatform = "linux/arm64"
t.Cleanup(func() { imagePlatform = previous })
testPinnedReleaseWorkerAdmissionAndBlock(t, "v8")
}
func testPinnedReleaseWorkerAdmissionAndBlock(t *testing.T, variant string) {
t.Helper()
info := releaseFixture()
labels := map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": info.Version, "dsh-mgr.recipe": "7", "dsh-mgr.release": info.ReleaseID, "dsh-mgr.source": info.Repository, "dsh-mgr.commit": info.Commit}
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/v1.47/build":
var got map[string]string
if err := json.Unmarshal([]byte(r.URL.Query().Get("labels")), &got); err != nil {
t.Error(err)
}
if got["dsh-mgr.commit"] != info.Commit || r.URL.Query().Get("t") != "dsh-web:release-"+info.Commit {
t.Error("missing provenance", r.URL)
}
data, _ := io.ReadAll(r.Body)
if !bytes.Contains(data, []byte("pinned context")) {
t.Error("wrong build context")
}
io.WriteString(w, `{"aux":{"ID":"sha256:release"}}`)
case "/v1.47/images/sha256:release/json":
parts := strings.Split(imagePlatform, "/")
writeJSON(w, map[string]any{"Id": "sha256:release", "Os": parts[0], "Architecture": parts[1], "Variant": variant, "Config": map[string]any{"Labels": labels}})
case "/v1.47/containers/create":
var cfg struct {
Image, User string
HostConfig struct {
NetworkMode string
ReadonlyRootfs bool
Tmpfs map[string]string
}
}
if err := json.NewDecoder(r.Body).Decode(&cfg); err != nil || cfg.Image != "sha256:release" || cfg.User != "1000:1000" || cfg.HostConfig.NetworkMode != "none" || !cfg.HostConfig.ReadonlyRootfs || !strings.Contains(cfg.HostConfig.Tmpfs["/tmp"], "exec") {
t.Errorf("unsafe verification config %+v err=%v", cfg, err)
}
io.WriteString(w, `{"Id":"check"}`)
case "/v1.47/containers/check/start", "/v1.47/exec/probe/start", "/v1.47/containers/check":
w.WriteHeader(204)
case "/v1.47/containers/check/exec":
var cfg struct{ Cmd []string }
expected, _ := json.Marshal(info.Version)
if err := json.NewDecoder(r.Body).Decode(&cfg); err != nil || len(cfg.Cmd) != 3 || !strings.Contains(cfg.Cmd[2], "execFileSync('dsh',['--version']") || !strings.Contains(cfg.Cmd[2], "actual!=="+string(expected)) {
t.Errorf("missing exact version check %+v err=%v", cfg, err)
}
io.WriteString(w, `{"Id":"probe"}`)
case "/v1.47/exec/probe/json":
io.WriteString(w, `{"Running":false,"ExitCode":0}`)
case "/v1.47/containers/check/json":
io.WriteString(w, `{"State":{"Running":true}}`)
default:
t.Errorf("unexpected %s", r.URL)
w.WriteHeader(500)
}
})
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='u'"); err != nil {
t.Fatal(err)
}
old := fetchRelease
t.Cleanup(func() { fetchRelease = old })
fetchRelease = func(ctx context.Context, repository, ref string) (releaseInfo, io.Reader, error) {
if repository != info.Repository || ref != info.Commit {
t.Error("not pinned", repository, ref)
}
return info, strings.NewReader("pinned context"), nil
}
b, err := a.enqueueRelease("u", info)
if err != nil {
t.Fatal(err)
}
a.runImageBuild(b)
if err = scanBuildSummary(a.s.DB.QueryRow("SELECT "+buildSummaryColumns+" FROM image_builds WHERE id=?", b.ID), &b); err != nil {
t.Fatal(err)
}
if b.Status != "success" {
t.Fatalf("%+v", b)
}
records := []buildRecord{b}
if err = a.reconcileBuilds(context.Background(), records); err != nil || records[0].Status != "success" {
t.Fatalf("reconciliation: %+v %v", records, err)
}
im, err := a.evaluateImage(context.Background(), b.ImageID)
if err != nil || !im.Selectable {
t.Fatalf("%+v %v", im, err)
}
if _, err = a.s.DB.Exec("INSERT INTO image_admissions(image_id,version,recipe_rev,platform,state,evidence,admitted_by,admitted_at) VALUES(?,?,?,?,?,?,?,?)", b.ImageID, b.Version, "7", imagePlatform, "blocked", "explicit block", "u", "now"); err != nil {
t.Fatal(err)
}
im, err = a.evaluateImage(context.Background(), b.ImageID)
if err != nil || im.Selectable || im.Admission != "blocked" {
t.Fatalf("%+v %v", im, err)
}
}
func TestReleaseRateLimitDedup(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("queued release contacted Docker") })
info := releaseFixture()
first, err := a.enqueueRelease("u", info)
if err != nil {
t.Fatal(err)
}
for _, commit := range []string{strings.Repeat("b", 40), strings.Repeat("c", 40)} {
next := info
next.Commit = commit
if _, err := a.enqueueRelease("u", next); err != nil {
t.Fatal(err)
}
}
again, err := a.enqueueRelease("u", info)
if err != nil || again.ID != first.ID {
t.Fatal(again, err)
}
info.Commit = strings.Repeat("d", 40)
if _, err := a.enqueueRelease("u", info); err != errBuildRateLimited {
t.Fatal(err)
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_requests").Scan(&count); err != nil || count != 3 {
t.Fatal(count, err)
}
}
func TestReleaseUnknownDiskBlocksBuild(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Errorf("unexpected Docker request %s", r.URL) })
t.Setenv("DSH_DOCKER_ROOT_VISIBLE", "")
if _, err := a.s.DB.Exec("UPDATE users SET admin=1 WHERE id='u'"); err != nil {
t.Fatal(err)
}
info := releaseFixture()
old := fetchRelease
t.Cleanup(func() { fetchRelease = old })
fetchRelease = func(context.Context, string, string) (releaseInfo, io.Reader, error) {
return info, strings.NewReader("context"), nil
}
b, err := a.enqueueRelease("u", info)
if err != nil {
t.Fatal(err)
}
a.runImageBuild(b)
if err := scanBuildSummary(a.s.DB.QueryRow("SELECT "+buildSummaryColumns+" FROM image_builds WHERE id=?", b.ID), &b); err != nil {
t.Fatal(err)
}
if b.Status != "failed" || !strings.Contains(b.Error, "capacity unknown") {
t.Fatalf("%+v", b)
}
}
internal/api/resource_recovery.go
-309
View File
@@ -1,309 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"log"
"net/http"
"net/url"
"os"
"strings"
"time"
)
// pendingColumns whitelists the runtime_creates flag columns so no caller can
// concatenate an unvalidated column name into SQL.
type pendingColumn struct{ flag, stamp string }
var pendingColumns = map[string]pendingColumn{
"container_pending": {flag: "container_pending", stamp: "container_pending_at"},
"network_pending": {flag: "network_pending", stamp: "network_pending_at"},
}
// ensureColumn adds a TEXT column, tolerating databases that already have it.
func (a *API) ensureColumn(table, column string) error {
_, err := a.s.DB.Exec("ALTER TABLE " + table + " ADD COLUMN " + column + " TEXT NOT NULL DEFAULT ''")
if err != nil && strings.Contains(strings.ToLower(err.Error()), "duplicate column") {
return nil
}
return err
}
func (a *API) recoverRuntimeContainer(ctx context.Context, i *store.Instance) error {
c, err := a.docker.Inspect(ctx, runtimeNetwork(i.ID))
if dockerNotFound(err) {
_, pending, checkErr := a.pendingSince(i.ID, "container_pending")
if checkErr != nil {
return checkErr
}
if pending {
return errors.New("container creation outcome remains uncertain")
}
return nil
}
if err != nil {
return err
}
if c.ID == "" || strings.TrimPrefix(c.Name, "/") != runtimeNetwork(i.ID) || c.Config.Labels["dsh-mgr.instance"] != i.ID || c.Config.Labels["dsh-mgr.owner"] != i.UID || c.Config.Labels["dsh-mgr.role"] != "dsh" || c.Image != i.ImageID {
return errors.New("runtime container identity mismatch")
}
if _, err = a.s.DB.ExecContext(ctx, "UPDATE instances SET container_id=? WHERE id=? AND container_id=''", c.ID, i.ID); err != nil {
return err
}
i.ContainerID = c.ID
if err = a.setRuntimeCreatePending(i.ID, "container_pending", false); err != nil {
return err
}
return nil
}
func (a *API) ensureRuntimeCreates() error {
_, err := a.s.DB.Exec("CREATE TABLE IF NOT EXISTS runtime_creates(instance_id TEXT PRIMARY KEY,container_pending INTEGER NOT NULL DEFAULT 0,network_pending INTEGER NOT NULL DEFAULT 0,container_pending_at TEXT NOT NULL DEFAULT '',network_pending_at TEXT NOT NULL DEFAULT '')")
if err != nil {
return err
}
if err = a.ensureColumn("runtime_creates", "container_pending_at"); err != nil {
return err
}
if err = a.ensureColumn("runtime_creates", "network_pending_at"); err != nil {
return err
}
_, err = a.s.DB.Exec("CREATE TABLE IF NOT EXISTS runtime_networks(instance_id TEXT PRIMARY KEY,network_id TEXT NOT NULL)")
return err
}
func (a *API) setRuntimeCreatePending(id, column string, pending bool) error {
if err := a.ensureRuntimeCreates(); err != nil {
return err
}
c, ok := pendingColumns[column]
if !ok {
return errors.New("unknown pending column")
}
stamp := ""
if pending {
stamp = time.Now().UTC().Format(time.RFC3339Nano)
}
_, err := a.s.DB.Exec("INSERT INTO runtime_creates(instance_id,"+c.flag+","+c.stamp+") VALUES(?,?,?) ON CONFLICT(instance_id) DO UPDATE SET "+c.flag+"=excluded."+c.flag+","+c.stamp+"=excluded."+c.stamp, id, pending, stamp)
return err
}
// pendingSince reports when a pending flag was raised. A legacy row without a
// timestamp reports a zero time without resolving the uncertainty.
func (a *API) pendingSince(id, column string) (time.Time, bool, error) {
if err := a.ensureRuntimeCreates(); err != nil {
return time.Time{}, false, err
}
c, ok := pendingColumns[column]
if !ok {
return time.Time{}, false, errors.New("unknown pending column")
}
var pending bool
var stamp string
err := a.s.DB.QueryRow("SELECT "+c.flag+",COALESCE("+c.stamp+",'') FROM runtime_creates WHERE instance_id=?", id).Scan(&pending, &stamp)
if errors.Is(err, sql.ErrNoRows) {
return time.Time{}, false, nil
}
if err != nil {
return time.Time{}, false, err
}
if !pending {
return time.Time{}, false, nil
}
if stamp == "" {
return time.Time{}, true, nil
}
since, perr := time.Parse(time.RFC3339Nano, stamp)
if perr != nil {
return time.Time{}, true, nil
}
return since, true, nil
}
func (a *API) recoverRuntimeNetwork(ctx context.Context, id string) error {
_, pending, err := a.pendingSince(id, "network_pending")
if err != nil {
return err
}
var recorded string
err = a.s.DB.QueryRowContext(ctx, "SELECT network_id FROM runtime_networks WHERE instance_id=?", id).Scan(&recorded)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return err
}
lookup := runtimeNetwork(id)
if recorded != "" {
lookup = recorded
}
var network struct {
ID, Name, Driver string
Labels map[string]string
Containers map[string]json.RawMessage
}
err = a.docker.Do(ctx, http.MethodGet, "/networks/"+url.PathEscape(lookup), nil, &network)
if dockerNotFound(err) {
if pending && recorded == "" {
return errors.New("network creation outcome remains uncertain")
}
return a.setRuntimeCreatePending(id, "network_pending", false)
}
if err != nil {
return err
}
if network.ID == "" || (recorded != "" && recorded != network.ID) || network.Name != runtimeNetwork(id) || network.Driver != "bridge" || network.Labels["dsh-mgr.instance"] != id {
return errors.New("runtime network identity mismatch")
}
if _, err = a.s.DB.ExecContext(ctx, "INSERT INTO runtime_networks(instance_id,network_id) VALUES(?,?) ON CONFLICT(instance_id) DO UPDATE SET network_id=excluded.network_id", id, network.ID); err != nil {
return err
}
if panel := os.Getenv("PANEL_CONTAINER_ID"); panel != "" && len(network.Containers) != 0 {
if _, connected := network.Containers[panel]; !connected {
c, err := a.docker.Inspect(ctx, panel)
if err != nil {
return err
}
panel = c.ID
}
if _, connected := network.Containers[panel]; connected {
err = a.docker.Do(ctx, http.MethodPost, "/networks/"+url.PathEscape(network.ID)+"/disconnect", map[string]any{"Container": panel, "Force": true}, nil)
if err != nil && !dockerNotFound(err) {
return err
}
}
}
err = a.docker.Do(ctx, http.MethodDelete, "/networks/"+url.PathEscape(network.ID), nil, nil)
if err != nil && !dockerNotFound(err) {
return err
}
return a.setRuntimeCreatePending(id, "network_pending", false)
}
func (a *API) ensureBuildChecks() error {
_, err := a.s.DB.Exec("CREATE TABLE IF NOT EXISTS image_build_checks(build_id TEXT PRIMARY KEY,image_id TEXT NOT NULL)")
if err != nil {
return err
}
if err = a.ensureColumn("image_build_checks", "container_id"); err != nil {
return err
}
return a.ensureColumn("image_build_checks", "created_at")
}
func buildCheckName(id string) string { return "dsh-build-check-" + id }
var errBuildCheckUncertain = errors.New("verification container creation outcome remains uncertain")
func (a *API) cleanupBuildCheck(ctx context.Context, id, image string) error {
var recorded string
if err := a.s.DB.QueryRowContext(ctx, "SELECT container_id FROM image_build_checks WHERE build_id=?", id).Scan(&recorded); err != nil {
return err
}
lookup := buildCheckName(id)
if recorded != "" {
lookup = recorded
}
c, err := a.docker.Inspect(ctx, lookup)
if dockerNotFound(err) {
if recorded != "" {
_, err := a.s.DB.ExecContext(ctx, "DELETE FROM image_build_checks WHERE build_id=?", id)
return err
}
// A name lookup cannot rule out a create still running in Docker.
return errBuildCheckUncertain
}
if err != nil {
return err
}
if c.ID == "" || (recorded != "" && c.ID != recorded) || strings.TrimPrefix(c.Name, "/") != buildCheckName(id) || c.Image != image || c.Config.Labels["dsh-mgr.role"] != "build-check" || c.Config.Labels["dsh-mgr.build"] != id {
return errors.New("build verification container identity mismatch")
}
return a.deleteBuildCheck(ctx, id, c.ID)
}
func (a *API) deleteBuildCheck(ctx context.Context, id, container string) error {
if _, err := a.s.DB.ExecContext(ctx, "UPDATE image_build_checks SET container_id=? WHERE build_id=?", container, id); err != nil {
return err
}
err := a.docker.Do(ctx, http.MethodDelete, "/containers/"+url.PathEscape(container)+"?force=1&v=1", nil, nil)
if err != nil && !dockerNotFound(err) {
return err
}
_, err = a.s.DB.ExecContext(ctx, "DELETE FROM image_build_checks WHERE build_id=?", id)
return err
}
func (a *API) recoverBuildChecks(ctx context.Context) error {
if err := a.ensureBuildChecks(); err != nil {
return err
}
rows, err := a.s.DB.QueryContext(ctx, "SELECT build_id,image_id,COALESCE(created_at,'') FROM image_build_checks")
if err != nil {
return err
}
type check struct{ id, image, createdAt string }
var checks []check
for rows.Next() {
var c check
if err = rows.Scan(&c.id, &c.image, &c.createdAt); err != nil {
break
}
checks = append(checks, c)
}
if err == nil {
err = rows.Err()
}
rows.Close()
if err != nil {
return err
}
var failures []error
missing := false
for _, c := range checks {
err := a.cleanupBuildCheck(ctx, c.id, c.image)
if errors.Is(err, errBuildCheckUncertain) {
missing = true
} else {
failures = append(failures, err)
}
}
if err := errors.Join(failures...); err != nil {
return err
}
if missing {
return errBuildCheckUncertain
}
return nil
}
func (a *API) persistBuildResult(b buildRecord, status, logs, detail, image, finished string) error {
tx, err := a.s.DB.Begin()
if err != nil {
return err
}
defer tx.Rollback()
if status == "success" && image != "" {
tag := "dsh-web:" + b.Version + "-" + imageRecipeRev
if b.ReleaseID != "" {
tag = "dsh-web:release-" + b.SourceCommit
}
_, err = tx.Exec("INSERT INTO images(image_id,version,recipe_rev,platform,tag,status,built_at) VALUES(?,?,?,?,?,?,?) ON CONFLICT(image_id) DO UPDATE SET version=excluded.version,recipe_rev=excluded.recipe_rev,platform=excluded.platform,tag=excluded.tag,status=excluded.status,built_at=excluded.built_at", image, b.Version, imageRecipeRev, imagePlatform, tag, "unverified", finished)
if err != nil {
return err
}
}
if _, err = tx.Exec("UPDATE image_builds SET status=?,logs=?,error=?,image_id=?,finished_at=? WHERE id=?", status, logs, detail, image, finished, b.ID); err != nil {
return err
}
return tx.Commit()
}
func (a *API) finishBuild(b buildRecord, status, logs, detail, image, finished string) {
for attempt := 0; ; attempt++ {
err := a.persistBuildResult(b, status, logs, detail, image, finished)
if err == nil {
return
}
if attempt%60 == 0 {
log.Printf("build %s: terminal transaction failed; retrying: %s", b.ID, sanitizeBuildLog(err.Error()))
}
select {
case <-a.s.Done():
return
case <-time.After(time.Second):
}
}
}
internal/api/resource_recovery_live_test.go
-407
View File
@@ -1,407 +0,0 @@
package api
import (
"bytes"
"context"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
const liveRecoveryLabel = "dsh-test.recovery-run"
type liveRecoveryResource struct{ kind, id string }
type liveRecoveryFault struct {
method, path string
release chan struct{}
done chan struct{}
once sync.Once
}
func (f *liveRecoveryFault) unblock() { f.once.Do(func() { close(f.release) }) }
type liveRecoveryProxy struct {
t *testing.T
run string
transport *http.Transport
mu sync.Mutex
handlers sync.WaitGroup
resources []liveRecoveryResource
fault *liveRecoveryFault
}
func liveRequire(t *testing.T, err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
func (p *liveRecoveryProxy) arm(method, path string, delayed bool) *liveRecoveryFault {
f := &liveRecoveryFault{method: method, path: path, release: make(chan struct{}), done: make(chan struct{})}
if !delayed {
f.unblock()
}
p.mu.Lock()
p.fault = f
p.mu.Unlock()
p.t.Cleanup(f.unblock)
return f
}
func (p *liveRecoveryProxy) ServeHTTP(w http.ResponseWriter, r *http.Request) {
p.handlers.Add(1)
defer p.handlers.Done()
body, err := io.ReadAll(r.Body)
if err != nil {
p.t.Error(err)
panic(http.ErrAbortHandler)
}
path := strings.TrimPrefix(r.URL.Path, "/v1.47")
create := r.Method == "POST" && (path == "/containers/create" || path == "/networks/create")
p.mu.Lock()
allowed := r.Method == "GET"
if create {
var spec struct {
Name string
Labels map[string]string
}
if json.Unmarshal(body, &spec) == nil {
name := spec.Name
if path == "/containers/create" {
name = r.URL.Query().Get("name")
}
allowed = spec.Labels[liveRecoveryLabel] == p.run && strings.Contains(name, p.run)
}
} else if !allowed {
for _, resource := range p.resources {
base := "/" + resource.kind + "/" + resource.id
if path == base || strings.HasPrefix(path, base+"/") {
allowed = true
}
}
if strings.HasSuffix(path, "/connect") || strings.HasSuffix(path, "/disconnect") {
var spec struct{ Container string }
_ = json.Unmarshal(body, &spec)
owned := false
for _, resource := range p.resources {
if resource.kind == "containers" && spec.Container == resource.id {
owned = true
}
}
allowed = allowed && owned
}
}
var fault *liveRecoveryFault
if p.fault != nil && p.fault.method == r.Method && p.fault.path == path {
fault = p.fault
p.fault = nil
}
p.mu.Unlock()
if !allowed {
p.t.Errorf("blocked out-of-scope request: %s %s", r.Method, path)
panic(http.ErrAbortHandler)
}
if fault != nil {
defer close(fault.done)
select {
case <-fault.release:
default:
conn, _, hijackErr := w.(http.Hijacker).Hijack()
if hijackErr != nil {
p.t.Error(hijackErr)
return
}
_ = conn.Close()
<-fault.release
}
}
// A disconnected test client must not cancel the deliberately late daemon operation.
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
req := r.Clone(ctx)
req.URL.Scheme, req.URL.Host, req.RequestURI = "http", "docker", ""
req.Body = io.NopCloser(bytes.NewReader(body))
if len(body) == 0 {
req.Body = http.NoBody
}
resp, err := p.transport.RoundTrip(req)
if err != nil {
p.t.Error(err)
panic(http.ErrAbortHandler)
}
defer resp.Body.Close()
data, err := io.ReadAll(resp.Body)
if err != nil {
p.t.Error(err)
panic(http.ErrAbortHandler)
}
if create && resp.StatusCode >= 200 && resp.StatusCode < 300 {
var result struct{ ID string }
if err := json.Unmarshal(data, &result); err != nil || result.ID == "" {
p.t.Errorf("missing real create ID: %s", data)
} else {
kind := "containers"
if path == "/networks/create" {
kind = "networks"
}
p.mu.Lock()
p.resources = append(p.resources, liveRecoveryResource{kind, result.ID})
p.mu.Unlock()
p.t.Logf("created %s ID=%s run=%s", kind, result.ID, p.run)
}
}
if fault != nil {
p.t.Logf("dropped real response: %s %s status=%d", r.Method, path, resp.StatusCode)
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
p.t.Errorf("fault did not follow successful daemon operation: %s", data)
}
panic(http.ErrAbortHandler)
}
for key, values := range resp.Header {
for _, value := range values {
w.Header().Add(key, value)
}
}
w.WriteHeader(resp.StatusCode)
_, _ = w.Write(data)
}
func newLiveRecovery(t *testing.T) (*API, *liveRecoveryProxy, string) {
t.Helper()
if os.Getenv("DSH_DOCKER_LIVE_TEST") != "1" {
t.Skip("opt in with DSH_DOCKER_LIVE_TEST=1; uses /var/run/docker.sock and local node:24-alpine only")
}
t.Setenv("PANEL_CONTAINER_ID", "")
var nonce [12]byte
_, err := rand.Read(nonce[:])
liveRequire(t, err)
p := &liveRecoveryProxy{t: t, run: "live-" + hex.EncodeToString(nonce[:]), transport: &http.Transport{DialContext: func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", "/var/run/docker.sock")
}}}
real := docker.New("unix:///var/run/docker.sock")
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
var image struct{ ID string }
liveRequire(t, real.Do(ctx, "GET", "/images/node:24-alpine/json", nil, &image))
s, err := store.Open(filepath.Join(t.TempDir(), "live.db"))
liveRequire(t, err)
t.Cleanup(s.Close)
server := httptest.NewServer(p)
t.Cleanup(func() {
server.Close()
p.handlers.Wait()
p.transport.CloseIdleConnections()
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
p.mu.Lock()
resources := append([]liveRecoveryResource(nil), p.resources...)
p.mu.Unlock()
for _, kind := range []string{"containers", "networks"} {
for _, resource := range resources {
if resource.kind != kind {
continue
}
path := "/" + kind + "/" + resource.id
err := real.Do(ctx, "DELETE", path+"?force=1", nil, nil)
if err != nil && !dockerNotFound(err) {
t.Errorf("cleanup %s: %v", path, err)
}
if kind == "containers" {
path += "/json"
}
if err := real.Do(ctx, "GET", path, nil, nil); !dockerNotFound(err) {
t.Errorf("residual %s: %v", path, err)
} else {
t.Logf("cleanup confirmed absent %s %s", kind, resource.id)
}
}
}
filters, _ := json.Marshal(map[string][]string{"label": {liveRecoveryLabel + "=" + p.run}})
for _, path := range []string{"/containers/json?all=1&", "/networks?"} {
var residual []json.RawMessage
if err := real.Do(ctx, "GET", path+"filters="+url.QueryEscape(string(filters)), nil, &residual); err != nil || len(residual) != 0 {
t.Errorf("label residual check: %s count=%d err=%v", p.run, len(residual), err)
} else {
t.Logf("label residual check empty: %s %s", p.run, path)
}
}
})
return &API{s: s, root: t.TempDir(), docker: docker.New(server.URL)}, p, image.ID
}
func liveContainerSpec(run, image string, labels map[string]string) map[string]any {
labels[liveRecoveryLabel] = run
return map[string]any{"Image": image, "Cmd": []string{"sleep", "600"}, "Labels": labels, "HostConfig": map[string]any{"NetworkMode": "none", "Memory": 64 << 20, "PidsLimit": 32}}
}
func TestDockerLiveRecoveryNetwork(t *testing.T) {
for _, panelMode := range []string{"unset", "unconnected", "connected-id", "connected-name"} {
t.Run(panelMode, func(t *testing.T) {
a, p, image := newLiveRecovery(t)
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
id := p.run
liveRequire(t, a.setRuntimeCreatePending(id, "network_pending", true))
p.arm("POST", "/networks/create", false)
err := a.docker.Do(ctx, "POST", "/networks/create", map[string]any{"Name": runtimeNetwork(id), "Driver": "bridge", "Labels": map[string]string{"dsh-mgr.instance": id, liveRecoveryLabel: p.run}}, nil)
if err == nil {
t.Fatal("create response was not lost")
}
var network struct{ ID string }
liveRequire(t, a.docker.Do(ctx, "GET", "/networks/"+runtimeNetwork(id), nil, &network))
if panelMode != "unset" {
name := p.run + "-panel"
var panel struct{ ID string }
spec := liveContainerSpec(p.run, image, map[string]string{})
if strings.HasPrefix(panelMode, "connected") {
spec["HostConfig"].(map[string]any)["NetworkMode"] = network.ID
}
liveRequire(t, a.docker.Do(ctx, "POST", "/containers/create?name="+name, spec, &panel))
liveRequire(t, a.docker.Do(ctx, "POST", "/containers/"+panel.ID+"/start", nil, nil))
t.Setenv("PANEL_CONTAINER_ID", panel.ID)
if panelMode == "connected-name" {
t.Setenv("PANEL_CONTAINER_ID", name)
}
var endpoints struct{ Containers map[string]json.RawMessage }
liveRequire(t, a.docker.Do(ctx, "GET", "/networks/"+network.ID, nil, &endpoints))
_, connected := endpoints.Containers[panel.ID]
if connected != strings.HasPrefix(panelMode, "connected") {
t.Fatalf("unexpected real panel connection: %v", connected)
}
}
p.arm("DELETE", "/networks/"+network.ID, false)
if err := a.removeRuntimeNetwork(ctx, id); err == nil {
t.Fatal("delete response was not lost")
}
var recorded string
liveRequire(t, a.s.DB.QueryRow("SELECT network_id FROM runtime_networks WHERE instance_id=?", id).Scan(&recorded))
if recorded != network.ID {
t.Fatalf("lost recorded identity: %s", recorded)
}
liveRequire(t, a.removeRuntimeNetwork(ctx, id))
_, pending, err := a.pendingSince(id, "network_pending")
liveRequire(t, err)
if pending {
t.Fatal("pending did not converge")
}
if err := a.docker.Do(ctx, "GET", "/networks/"+network.ID, nil, nil); !dockerNotFound(err) {
t.Fatalf("network still exists: %v", err)
}
})
}
}
func TestDockerLiveRecoveryLateContainer(t *testing.T) {
for _, build := range []bool{false, true} {
name := "runtime"
if build {
name = "build-check"
}
t.Run(name, func(t *testing.T) {
a, p, image := newLiveRecovery(t)
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
i := store.Instance{ID: p.run, UID: p.run, ImageID: image, Status: "error", Created: time.Now()}
name := runtimeNetwork(i.ID)
labels := map[string]string{"dsh-mgr.instance": i.ID, "dsh-mgr.owner": i.UID, "dsh-mgr.role": "dsh"}
if build {
name = buildCheckName(i.ID)
labels = map[string]string{"dsh-mgr.build": i.ID, "dsh-mgr.role": "build-check"}
liveRequire(t, a.ensureBuildChecks())
_, err := a.s.DB.Exec("INSERT INTO image_build_checks(build_id,image_id,created_at) VALUES(?,?,?)", i.ID, image, time.Now().Add(-time.Hour).Format(time.RFC3339Nano))
liveRequire(t, err)
} else {
liveRequire(t, a.s.AddInstance(i))
liveRequire(t, a.setRuntimeCreatePending(i.ID, "container_pending", true))
}
fault := p.arm("POST", "/containers/create", true)
if err := a.docker.Do(ctx, "POST", "/containers/create?name="+name, liveContainerSpec(p.run, image, labels), nil); err == nil {
t.Fatal("expected disconnected create")
}
if _, err := a.docker.Inspect(ctx, name); !dockerNotFound(err) {
t.Fatalf("expected real initial 404: %v", err)
}
for attempt := 0; attempt < 2; attempt++ {
if build {
if err := a.recoverBuildChecks(ctx); !errors.Is(err, errBuildCheckUncertain) {
t.Fatalf("uncertainty lost: %v", err)
}
var count int
liveRequire(t, a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks WHERE build_id=?", i.ID).Scan(&count))
if count != 1 {
t.Fatal("build record lost")
}
} else {
if err := a.recoverRuntimeContainer(ctx, &i); err == nil {
t.Fatal("uncertainty lost")
}
_, pending, err := a.pendingSince(i.ID, "container_pending")
liveRequire(t, err)
if !pending {
t.Fatal("pending lost")
}
}
}
fault.unblock()
select {
case <-fault.done:
case <-ctx.Done():
t.Fatal(ctx.Err())
}
container, err := a.docker.Inspect(ctx, name)
liveRequire(t, err)
p.arm("DELETE", "/containers/"+container.ID, false)
if build {
if err := a.recoverBuildChecks(ctx); err == nil {
t.Fatal("delete response was not lost")
}
var recorded string
liveRequire(t, a.s.DB.QueryRow("SELECT container_id FROM image_build_checks WHERE build_id=?", i.ID).Scan(&recorded))
if recorded != container.ID {
t.Fatal("build identity lost")
}
liveRequire(t, a.recoverBuildChecks(ctx))
var count int
liveRequire(t, a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&count))
if count != 0 {
t.Fatal("build record retained")
}
} else {
liveRequire(t, a.recoverRuntimeContainer(ctx, &i))
_, pending, err := a.pendingSince(i.ID, "container_pending")
liveRequire(t, err)
if pending {
t.Fatal("runtime pending did not converge")
}
stored, err := a.s.GetInstance(i.ID)
liveRequire(t, err)
if stored.ContainerID != container.ID {
t.Fatal("runtime identity lost")
}
if err := a.removeRuntimeContainer(ctx, i.ContainerID); err == nil {
t.Fatal("delete response was not lost")
}
liveRequire(t, a.removeRuntimeContainer(ctx, i.ContainerID))
}
if _, err := a.docker.Inspect(ctx, container.ID); !dockerNotFound(err) {
t.Fatalf("late resource not reclaimed: %v", err)
}
})
}
}
internal/api/resource_recovery_test.go
-765
View File
@@ -1,765 +0,0 @@
package api
import (
"context"
"encoding/json"
"errors"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"net/http"
"net/http/httptest"
"path/filepath"
"sync/atomic"
"testing"
"time"
)
func TestBuildCheckRenamedContainerRetainsIdentity(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" || r.URL.Path != "/v1.47/containers/known/json" {
t.Errorf("unsafe lookup %s %s", r.Method, r.URL.Path)
w.WriteHeader(404)
return
}
fmt.Fprint(w, `{"Id":"known","Name":"/renamed","Image":"image","Config":{"Labels":{"dsh-mgr.role":"build-check","dsh-mgr.build":"task"}}}`)
})
if err := a.ensureBuildChecks(); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO image_build_checks(build_id,image_id,container_id) VALUES('task','image','known')"); err != nil {
t.Fatal(err)
}
if err := a.recoverBuildChecks(context.Background()); err == nil {
t.Fatal("renamed identity accepted")
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&n); err != nil || n != 1 {
t.Fatalf("record lost %d %v", n, err)
}
}
func TestRuntimeNetworkDeletionLostResponseAndReplacement(t *testing.T) {
for _, foreign := range []bool{false, true} {
t.Run(fmt.Sprint(foreign), func(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
deleted := false
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/image/json" {
writeImageInspect(w, "image", "1.2.3")
return
}
if r.Method == "GET" {
if deleted {
if r.URL.Path != "/v1.47/networks/known" {
t.Error("did not use persisted ID")
}
w.WriteHeader(404)
return
}
owner := "net"
if foreign {
owner = "foreign"
}
fmt.Fprintf(w, `{"Id":"known","Name":"dsh-inst-net","Driver":"bridge","Labels":{"dsh-mgr.instance":%q}}`, owner)
return
}
if foreign {
t.Error("deleted foreign network")
}
deleted = true
w.WriteHeader(500)
})
if err := a.setRuntimeCreatePending("net", "network_pending", false); err != nil {
t.Fatal(err)
}
if err := a.removeRuntimeNetwork(context.Background(), "net"); err == nil {
t.Fatal("first error missing")
}
if foreign {
return
}
if err := a.removeRuntimeNetwork(context.Background(), "net"); err != nil {
t.Fatalf("confirmed deletion did not converge: %v", err)
}
})
}
}
func TestNetworkCreateLostResponse(t *testing.T) {
for _, mode := range []string{"present", "late", "foreign"} {
t.Run(mode, func(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
late := false
deletes := 0
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/image/json" {
writeImageInspect(w, "image", "1.2.3")
return
}
switch r.Method + " " + r.URL.Path {
case "POST /v1.47/networks/create":
w.WriteHeader(500)
case "GET /v1.47/containers/dsh-inst-net/json":
w.WriteHeader(404)
case "GET /v1.47/networks/dsh-inst-net":
if mode == "late" && !late {
w.WriteHeader(404)
return
}
id := "net"
if mode == "foreign" {
id = "other"
}
fmt.Fprintf(w, `{"Id":"network-id","Name":"dsh-inst-net","Driver":"bridge","Labels":{"dsh-mgr.instance":%q}}`, id)
case "DELETE /v1.47/networks/network-id":
deletes++
w.WriteHeader(204)
default:
t.Errorf("unexpected %s %s", r.Method, r.URL.Path)
w.WriteHeader(500)
}
})
seedVerifiedImage(t, a, "image", "1.2.3")
i := store.Instance{ID: "net", UID: "owner", Version: "1.2.3", ImageID: "image", Status: "creating"}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
if _, err := a.createRuntime(context.Background(), store.User{ID: "owner"}, i, "panel.test"); err == nil {
t.Fatal("lost response accepted")
}
if mode == "present" {
if deletes != 1 {
t.Fatal("network leaked")
}
return
}
saved, err := a.s.GetInstance("net")
if err != nil || saved.Status != "error" {
t.Fatalf("lost reservation %+v %v", saved, err)
}
late = true
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("DELETE", "/api/instances/net", "", sid))
if mode == "foreign" {
if deletes != 0 || w.Code != 502 {
t.Fatal("foreign network touched")
}
return
}
if w.Code != 200 || deletes != 1 {
t.Fatalf("late network retry %d %d", w.Code, deletes)
}
})
}
}
func TestBuildCheckReopenWorkerWithoutQueuedBuilds(t *testing.T) {
path := filepath.Join(t.TempDir(), "recovery.db")
s, err := store.Open(path)
if err != nil {
t.Fatal(err)
}
a := &API{s: s}
if err = a.ensureBuildChecks(); err != nil {
t.Fatal(err)
}
if _, err = s.DB.Exec("INSERT INTO image_build_checks(build_id,image_id) VALUES('task','image')"); err != nil {
t.Fatal(err)
}
s.Close()
s, err = store.Open(path)
if err != nil {
t.Fatal(err)
}
defer s.Close()
var attempts atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method == "GET" {
fmt.Fprint(w, `{"Id":"check","Name":"/dsh-build-check-task","Image":"image","Config":{"Labels":{"dsh-mgr.role":"build-check","dsh-mgr.build":"task"}}}`)
return
}
if r.Method == "DELETE" {
if attempts.Add(1) == 1 {
w.WriteHeader(500)
} else {
w.WriteHeader(204)
}
return
}
t.Errorf("unexpected %s", r.Method)
}))
defer srv.Close()
a = &API{s: s, docker: docker.New(srv.URL)}
if err = a.StartImageBuilds(); err != nil {
t.Fatal(err)
}
deadline := time.Now().Add(4 * time.Second)
for {
var n int
if err = s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&n); err != nil {
t.Fatal(err)
}
if n == 0 {
break
}
if time.Now().After(deadline) {
t.Fatal("reopen worker did not retry")
}
time.Sleep(10 * time.Millisecond)
}
if attempts.Load() != 2 {
t.Fatalf("attempts %d", attempts.Load())
}
}
func TestBuildTerminalFailureDoesNotBlockNextTask(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(500) })
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
t.Fatal(err)
}
for _, id := range []string{"a", "b"} {
if _, err = a.s.DB.Exec("INSERT INTO image_builds(id,version,status,created_at,release_id) VALUES(?,'1.2.3','queued',?,'release')", id, id); err != nil {
t.Fatal(err)
}
}
if _, err = a.s.DB.Exec("CREATE TRIGGER fail_a BEFORE UPDATE ON image_builds WHEN NEW.id='a' AND NEW.status='failed' BEGIN SELECT RAISE(FAIL,'injected'); END"); err != nil {
t.Fatal(err)
}
a.startBuildWorker()
deadline := time.Now().Add(4 * time.Second)
for {
var status string
if err = a.s.DB.QueryRow("SELECT status FROM image_builds WHERE id='b'").Scan(&status); err != nil {
t.Fatal(err)
}
if status == "failed" {
break
}
if time.Now().After(deadline) {
t.Fatal("A terminal retry blocked B")
}
time.Sleep(10 * time.Millisecond)
}
if _, err = a.s.DB.Exec("DROP TRIGGER fail_a"); err != nil {
t.Fatal(err)
}
for {
var status string
if err = a.s.DB.QueryRow("SELECT status FROM image_builds WHERE id='a'").Scan(&status); err != nil {
t.Fatal(err)
}
if status == "failed" {
break
}
if time.Now().After(deadline) {
t.Fatal("A never converged")
}
time.Sleep(10 * time.Millisecond)
}
}
func TestCreateLostResponseRecovery(t *testing.T) {
for _, mode := range []string{"cleanup", "retry", "foreign", "inspect-failure", "late"} {
t.Run(mode, func(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
deleted := 0
retry := false
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/image/json" {
writeImageInspect(w, "image", "1.2.3")
return
}
switch r.Method + " " + r.URL.Path {
case "POST /v1.47/networks/create":
fmt.Fprint(w, `{"Id":"dsh-inst-lost"}`)
case "GET /v1.47/networks/dsh-inst-lost":
fmt.Fprint(w, `{"Id":"dsh-inst-lost","Name":"dsh-inst-lost","Driver":"bridge","Labels":{"dsh-mgr.instance":"lost"}}`)
case "POST /v1.47/containers/create":
fmt.Fprint(w, "{")
case "GET /v1.47/containers/dsh-inst-lost/json":
if mode == "late" && !retry {
w.WriteHeader(404)
return
}
if mode == "inspect-failure" && !retry {
w.WriteHeader(500)
return
}
owner := "owner"
if mode == "foreign" {
owner = "another-panel"
}
fmt.Fprintf(w, `{"Id":"created","Name":"/dsh-inst-lost","Image":"image","Config":{"Labels":{"dsh-mgr.instance":"lost","dsh-mgr.owner":%q,"dsh-mgr.role":"dsh"}}}`, owner)
case "DELETE /v1.47/containers/created":
if mode == "retry" && !retry {
w.WriteHeader(500)
return
}
deleted++
w.WriteHeader(204)
case "DELETE /v1.47/networks/dsh-inst-lost":
w.WriteHeader(204)
default:
t.Errorf("unexpected %s %s", r.Method, r.URL.Path)
w.WriteHeader(500)
}
})
seedVerifiedImage(t, a, "image", "1.2.3")
i := store.Instance{ID: "lost", UID: "owner", Version: "1.2.3", ImageID: "image", Status: "creating"}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
if _, err := a.createRuntime(context.Background(), store.User{ID: "owner"}, i, "panel.test"); err == nil {
t.Fatal("accepted lost response")
}
if mode == "cleanup" {
if _, err := a.s.GetInstance(i.ID); err == nil || deleted != 1 {
t.Fatal("cleanup incomplete")
}
return
}
saved, err := a.s.GetInstance(i.ID)
if err != nil || saved.Status != "error" || deleted != 0 {
t.Fatalf("lost recovery record: %+v %v", saved, err)
}
retry = true
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("DELETE", "/api/instances/lost", "", sid))
if mode == "foreign" {
if w.Code != 502 || deleted != 0 {
t.Fatal("foreign resource touched")
}
return
}
if w.Code != 200 || deleted != 1 {
t.Fatalf("retry failed %d %s", w.Code, w.Body.String())
}
})
}
}
func TestBuildCheckRecoveryIdentityAndRetry(t *testing.T) {
for _, mode := range []string{"lost", "delete-failure", "foreign", "absent"} {
t.Run(mode, func(t *testing.T) {
deleted := 0
retry := false
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == "POST" && r.URL.Path == "/v1.47/containers/create":
if r.URL.Query().Get("name") != buildCheckName("task") {
t.Error("missing task name")
}
if mode == "delete-failure" {
fmt.Fprint(w, `{"Id":"check"}`)
} else {
fmt.Fprint(w, "{")
}
case r.Method == "GET":
if mode == "absent" && !retry {
w.WriteHeader(404)
return
}
if !retry && mode == "lost" {
w.WriteHeader(500)
return
}
id := "task"
if mode == "foreign" {
id = "another-panel"
}
fmt.Fprintf(w, `{"Id":"check","Name":"/dsh-build-check-task","Image":"image","Config":{"Labels":{"dsh-mgr.role":"build-check","dsh-mgr.build":%q}}}`, id)
case r.Method == "DELETE":
if !retry {
w.WriteHeader(500)
return
}
deleted++
w.WriteHeader(204)
default:
w.WriteHeader(500)
}
})
if err := a.verifyBuiltImage(context.Background(), "image", "1.2.3", "task"); err == nil {
t.Fatal("accepted failed validation")
}
var count int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&count); err != nil {
t.Fatal(err)
}
if count != 1 {
t.Fatal("lost cleanup record")
}
retry = true
err := a.recoverBuildChecks(context.Background())
if mode == "foreign" {
if err == nil || deleted != 0 {
t.Fatal("foreign container deleted")
}
return
}
if err != nil {
t.Fatal(err)
}
if deleted != 1 {
t.Fatal("not recovered")
}
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&count); err != nil || count != 0 {
t.Fatalf("record retained %d %v", count, err)
}
})
}
}
func TestReconcileRecoversLostRuntimeIdentity(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/image/json" {
writeImageInspect(w, "image", "1.2.3")
return
}
if r.Method != "GET" {
t.Fatalf("destructive recovery: %s", r.Method)
}
if r.URL.Path == "/v1.47/containers/json" {
fmt.Fprint(w, `[{"Id":"created","State":"running","Labels":{"dsh-mgr.instance":"lost","dsh-mgr.owner":"owner"}}]`)
return
}
fmt.Fprint(w, `{"Id":"created","Name":"/dsh-inst-lost","Image":"image","Config":{"Labels":{"dsh-mgr.instance":"lost","dsh-mgr.owner":"owner","dsh-mgr.role":"dsh"}}}`)
})
if err := a.s.AddInstance(store.Instance{ID: "lost", UID: "owner", ImageID: "image", Status: "creating"}); err != nil {
t.Fatal(err)
}
if err := a.Reconcile(context.Background()); err != nil {
t.Fatal(err)
}
i, err := a.s.GetInstance("lost")
if err != nil || i.ContainerID != "created" || i.Status != "error" {
t.Fatalf("not recovered: %+v %v", i, err)
}
}
func TestBuildResultTransactionAndRetry(t *testing.T) {
for _, table := range []string{"images", "image_builds"} {
t.Run(table, func(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker request") })
buildQueue.Lock()
err := a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
t.Fatal(err)
}
b := buildRecord{ID: "task", Version: "1.2.3"}
if _, err = a.s.DB.Exec("INSERT INTO image_builds(id,version,status,created_at) VALUES('task','1.2.3','validating','now')"); err != nil {
t.Fatal(err)
}
event := "INSERT"
if table == "image_builds" {
event = "UPDATE"
}
if _, err = a.s.DB.Exec("CREATE TRIGGER fail_result BEFORE " + event + " ON " + table + " BEGIN SELECT RAISE(FAIL,'injected'); END"); err != nil {
t.Fatal(err)
}
if err = a.persistBuildResult(b, "success", "logs", "", "image", "now"); err == nil {
t.Fatal("injection ignored")
}
var status string
var count int
if err = a.s.DB.QueryRow("SELECT status FROM image_builds WHERE id='task'").Scan(&status); err != nil || status != "validating" {
t.Fatalf("partial terminal write %s %v", status, err)
}
if err = a.s.DB.QueryRow("SELECT count(*) FROM images").Scan(&count); err != nil || count != 0 {
t.Fatalf("partial index write %d %v", count, err)
}
done := make(chan struct{})
go func() { a.finishBuild(b, "success", "logs", "", "image", "now"); close(done) }()
select {
case <-done:
t.Fatal("retry abandoned")
case <-time.After(30 * time.Millisecond):
}
if _, err = a.s.DB.Exec("DROP TRIGGER fail_result"); err != nil {
t.Fatal(err)
}
select {
case <-done:
case <-time.After(4 * time.Second):
t.Fatal("terminal write not retried")
}
if err = a.s.DB.QueryRow("SELECT status FROM image_builds WHERE id='task'").Scan(&status); err != nil || status != "success" {
t.Fatalf("terminal %s %v", status, err)
}
if err = a.s.DB.QueryRow("SELECT count(*) FROM images").Scan(&count); err != nil || count != 1 {
t.Fatalf("index %d %v", count, err)
}
})
}
}
func TestBuildTerminalRetryStopsWhenDatabaseCloses(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker call") })
a.s.Close()
done := make(chan struct{})
go func() { a.finishBuild(buildRecord{ID: "closed"}, "failed", "", "", "", "now"); close(done) }()
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("closed database retry did not stop")
}
}
func TestBuildCheckDefiniteRejectionAndExternalDeletion(t *testing.T) {
for _, rejected := range []bool{true, false} {
t.Run(fmt.Sprint(rejected), func(t *testing.T) {
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == "POST" && r.URL.Path == "/v1.47/containers/create" {
w.WriteHeader(400)
return
}
w.WriteHeader(404)
})
if rejected {
if err := a.verifyBuiltImage(context.Background(), "image", "1.2.3", "task"); err == nil {
t.Fatal("rejection accepted")
}
} else {
if err := a.ensureBuildChecks(); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO image_build_checks(build_id,image_id,container_id) VALUES('task','image','known')"); err != nil {
t.Fatal(err)
}
if err := a.recoverBuildChecks(context.Background()); err != nil {
t.Fatal(err)
}
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&n); err != nil || n != 0 {
t.Fatalf("definite outcome retained: %d %v", n, err)
}
})
}
}
func TestUncertainPendingRetainsLateResources(t *testing.T) {
for _, column := range []string{"container_pending", "network_pending"} {
for _, stamp := range []string{time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano), "", "invalid"} {
t.Run(column+"/"+stamp, func(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
late := false
deletes := 0
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/image/json" {
writeImageInspect(w, "image", "1.2.3")
return
}
switch r.Method + " " + r.URL.Path {
case "GET /v1.47/containers/dsh-inst-ghost/json":
if !late || column != "container_pending" {
w.WriteHeader(404)
return
}
fmt.Fprint(w, `{"Id":"container-id","Name":"/dsh-inst-ghost","Image":"image","Config":{"Labels":{"dsh-mgr.instance":"ghost","dsh-mgr.owner":"owner","dsh-mgr.role":"dsh"}}}`)
case "GET /v1.47/networks/dsh-inst-ghost":
if !late || column != "network_pending" {
w.WriteHeader(404)
return
}
fmt.Fprint(w, `{"Id":"network-id","Name":"dsh-inst-ghost","Driver":"bridge","Labels":{"dsh-mgr.instance":"ghost"}}`)
case "DELETE /v1.47/containers/container-id", "DELETE /v1.47/networks/network-id":
deletes++
w.WriteHeader(204)
default:
t.Errorf("unexpected %s %s", r.Method, r.URL.Path)
w.WriteHeader(500)
}
})
i := store.Instance{ID: "ghost", UID: "owner", Version: "1.2.3", ImageID: "image", Status: "error"}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
if err := a.setRuntimeCreatePending(i.ID, column, true); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("UPDATE runtime_creates SET "+pendingColumns[column].stamp+"=? WHERE instance_id=?", stamp, i.ID); err != nil {
t.Fatal(err)
}
for attempt := 0; attempt < 2; attempt++ {
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("DELETE", "/api/instances/ghost", "", sid))
if w.Code != 502 {
t.Fatalf("uncertainty discarded: %d %s", w.Code, w.Body.String())
}
if _, pending, err := a.pendingSince(i.ID, column); err != nil || !pending {
t.Fatalf("pending lost: %v %v", pending, err)
}
if _, err := a.s.GetInstance(i.ID); err != nil {
t.Fatal(err)
}
}
late = true
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("DELETE", "/api/instances/ghost", "", sid))
if w.Code != 200 || deletes != 1 {
t.Fatalf("late cleanup: %d %s deletes=%d", w.Code, w.Body.String(), deletes)
}
if _, err := a.s.GetInstance(i.ID); err == nil {
t.Fatal("instance retained")
}
})
}
}
}
func TestBuildCheckUncertainRowRetainsLateContainer(t *testing.T) {
for _, stamp := range []string{time.Now().Add(-time.Hour).UTC().Format(time.RFC3339Nano), "", "invalid"} {
t.Run(stamp, func(t *testing.T) {
late := false
deletes := 0
a, _ := imageTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch r.Method + " " + r.URL.Path {
case "GET /v1.47/containers/dsh-build-check-task/json":
if !late {
w.WriteHeader(404)
return
}
fmt.Fprint(w, `{"Id":"check","Name":"/dsh-build-check-task","Image":"image","Config":{"Labels":{"dsh-mgr.role":"build-check","dsh-mgr.build":"task"}}}`)
case "DELETE /v1.47/containers/check":
deletes++
w.WriteHeader(204)
default:
t.Errorf("unexpected %s %s", r.Method, r.URL.Path)
w.WriteHeader(500)
}
})
if err := a.ensureBuildChecks(); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO image_build_checks(build_id,image_id,created_at) VALUES('task','image',?)", stamp); err != nil {
t.Fatal(err)
}
for attempt := 0; attempt < 2; attempt++ {
if err := a.recoverBuildChecks(context.Background()); !errors.Is(err, errBuildCheckUncertain) {
t.Fatalf("uncertainty discarded: %v", err)
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&n); err != nil || n != 1 {
t.Fatalf("record lost: %d %v", n, err)
}
}
late = true
if err := a.recoverBuildChecks(context.Background()); err != nil {
t.Fatal(err)
}
var n int
if err := a.s.DB.QueryRow("SELECT count(*) FROM image_build_checks").Scan(&n); err != nil || n != 0 || deletes != 1 {
t.Fatalf("late cleanup: rows=%d deletes=%d err=%v", n, deletes, err)
}
})
}
}
func TestRuntimeNetworkPanelDisconnect(t *testing.T) {
for _, mode := range []string{"not-connected", "connected", "panel-name", "other-connected", "lost-disconnect", "forbidden", "inspect-forbidden", "foreign"} {
t.Run(mode, func(t *testing.T) {
panel := "panel-id"
if mode == "panel-name" || mode == "inspect-forbidden" {
panel = "panel"
}
t.Setenv("PANEL_CONTAINER_ID", panel)
connected := mode != "not-connected"
disconnects, deletes := 0, 0
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/v1.47/images/image/json" {
writeImageInspect(w, "image", "1.2.3")
return
}
switch r.Method + " " + r.URL.Path {
case "GET /v1.47/networks/dsh-inst-net", "GET /v1.47/networks/network-id":
owner := "net"
if mode == "foreign" {
owner = "other"
}
containers := "{}"
if connected {
containers = `{"panel-id":{}}`
}
if mode == "other-connected" {
containers = `{"other-id":{}}`
}
fmt.Fprintf(w, `{"Id":"network-id","Name":"dsh-inst-net","Driver":"bridge","Labels":{"dsh-mgr.instance":%q},"Containers":%s}`, owner, containers)
case "GET /v1.47/containers/panel/json", "GET /v1.47/containers/panel-id/json":
if mode == "inspect-forbidden" {
w.WriteHeader(403)
return
}
fmt.Fprint(w, `{"Id":"panel-id"}`)
case "POST /v1.47/networks/network-id/disconnect":
disconnects++
var body struct {
Container string
Force bool
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || body.Container != "panel-id" || !body.Force {
t.Errorf("disconnect identity: %+v %v", body, err)
}
if mode == "not-connected" || mode == "other-connected" || mode == "foreign" {
t.Error("unsafe disconnect")
w.WriteHeader(500)
return
}
if mode == "forbidden" {
w.WriteHeader(403)
return
}
connected = false
if mode == "lost-disconnect" {
w.WriteHeader(500)
return
}
w.WriteHeader(204)
case "DELETE /v1.47/networks/network-id":
deletes++
w.WriteHeader(204)
default:
t.Errorf("unexpected %s %s", r.Method, r.URL.Path)
w.WriteHeader(500)
}
})
if err := a.setRuntimeCreatePending("net", "network_pending", true); err != nil {
t.Fatal(err)
}
err := a.recoverRuntimeNetwork(context.Background(), "net")
wantError := mode == "lost-disconnect" || mode == "forbidden" || mode == "inspect-forbidden" || mode == "foreign"
if (err != nil) != wantError {
t.Fatalf("recovery error=%v wantError=%v", err, wantError)
}
if wantError {
if deletes != 0 {
t.Fatal("deleted after failure")
}
if _, pending, err := a.pendingSince("net", "network_pending"); err != nil || !pending {
t.Fatalf("pending lost: %v %v", pending, err)
}
if mode != "lost-disconnect" {
return
}
if err := a.recoverRuntimeNetwork(context.Background(), "net"); err != nil {
t.Fatalf("disconnect retry: %v", err)
}
}
wantDisconnects := 1
if mode == "not-connected" || mode == "other-connected" {
wantDisconnects = 0
}
if deletes != 1 || disconnects != wantDisconnects {
t.Fatalf("deletes=%d disconnects=%d", deletes, disconnects)
}
if _, pending, err := a.pendingSince("net", "network_pending"); err != nil || pending {
t.Fatalf("pending retained: %v %v", pending, err)
}
})
}
}
internal/api/response_buffer.go
-39
View File
@@ -1,39 +0,0 @@
package api
import (
"bytes"
"net/http"
)
// Only use for bounded JSON responses, never file or runtime streams. Register
// flush before lock-release defers so a slow client cannot hold storage locks.
func deferJSONResponse(w http.ResponseWriter) (http.ResponseWriter, func()) {
b := &jsonResponseBuffer{ResponseWriter: w}
return b, func() {
if b.status != 0 {
w.WriteHeader(b.status)
}
if b.body.Len() > 0 {
_, _ = w.Write(b.body.Bytes())
}
}
}
type jsonResponseBuffer struct {
http.ResponseWriter
body bytes.Buffer
status int
}
func (w *jsonResponseBuffer) WriteHeader(status int) {
if w.status == 0 {
w.status = status
}
}
func (w *jsonResponseBuffer) Write(p []byte) (int, error) {
if w.status == 0 {
w.status = http.StatusOK
}
return w.body.Write(p)
}
func (w *jsonResponseBuffer) Unwrap() http.ResponseWriter { return w.ResponseWriter }
internal/api/restore_snapshot_test.go
-117
View File
@@ -1,117 +0,0 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
)
func TestRestoreSnapshotCopyFailureDoesNotConsumeRetention(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
a, i, home := configTestAPI(t, false)
configWrite(t, filepath.Join(home, ".credentials.yaml"), "archive")
w := configCall(a, i, "POST", "backups", `{"password":"original-password"}`)
var result struct{ ID string }
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &result) != nil {
t.Fatal(w.Code, w.Body.String())
}
configWrite(t, filepath.Join(home, ".credentials.yaml"), "live")
base := filepath.Dir(home)
for _, name := range []string{"pre-restore-valuable", "failed-restore-valuable"} {
if err := os.Mkdir(filepath.Join(base, name), 0700); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(base, name, "keep"), name)
}
if err := syscall.Mkfifo(filepath.Join(home, "dangling"), 0600); err != nil {
t.Fatal(err)
}
body := fmt.Sprintf(`{"backup_id":%q,"password":"original-password"}`, result.ID)
for attempt := 0; attempt < 3; attempt++ {
w = configCall(a, i, "POST", "restore", body)
if w.Code != 400 || !strings.Contains(w.Body.String(), "cannot snapshot home") {
t.Fatalf("attempt %d: %d %s", attempt, w.Code, w.Body.String())
}
entries, err := os.ReadDir(base)
if err != nil || len(entries) != 3 {
t.Fatalf("partial snapshot leaked: %v %v", entries, err)
}
for _, name := range []string{"pre-restore-valuable", "failed-restore-valuable"} {
data, err := os.ReadFile(filepath.Join(base, name, "keep"))
if err != nil || string(data) != name {
t.Fatalf("valuable snapshot lost: %s %v", name, err)
}
}
data, err := os.ReadFile(filepath.Join(home, ".credentials.yaml"))
if err != nil || string(data) != "live" {
t.Fatalf("live home changed: %q %v", data, err)
}
}
if err := os.Remove(filepath.Join(home, "dangling")); err != nil {
t.Fatal(err)
}
if w = configCall(a, i, "POST", "restore", body); w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
entries, err := os.ReadDir(base)
if err != nil || len(entries) != 4 {
t.Fatalf("completed snapshot not retained: %v %v", entries, err)
}
if w = configCall(a, i, "POST", "restore", body); w.Code != 409 {
t.Fatal("retention not enforced", w.Code, w.Body.String())
}
}
func TestWorkspaceLeaseBlocksConfigAndUpgrade(t *testing.T) {
a, i, _ := configTestAPI(t, false)
a.workspaceReaders = map[string]int{i.ID: 1}
for _, action := range []string{"profile", "restore", "backups"} {
w := configCall(a, i, "POST", action, `{}`)
if w.Code != 409 || !strings.Contains(w.Body.String(), "instance files busy") {
t.Fatal(w.Code, w.Body.String())
}
}
w := configCall(a, i, "GET", "backups", "")
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
w = httptest.NewRecorder()
body, _ := json.Marshal(map[string]string{"image_id": "sha256:" + strings.Repeat("a", 64), "password": "original-password"})
r := httptest.NewRequest("POST", "/", strings.NewReader(string(body)))
a.upgrade(w, r, i)
if w.Code != 409 || !strings.Contains(w.Body.String(), "instance files busy") {
t.Fatal(w.Code, w.Body.String())
}
}
func TestUpgradeCopyMultipleDirectoryBatches(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
root, err := os.OpenRoot(t.TempDir())
if err != nil {
t.Fatal(err)
}
defer root.Close()
if err = root.Mkdir("home", 0700); err != nil {
t.Fatal(err)
}
for n := 0; n < 257; n++ {
configWrite(t, filepath.Join(root.Name(), "home", fmt.Sprint(n)), "x")
}
if err = upgradeCopy(context.Background(), root, "home", "snapshot", 257); err != nil {
t.Fatal(err)
}
entries, err := os.ReadDir(filepath.Join(root.Name(), "snapshot"))
if err != nil || len(entries) != 257 {
t.Fatal(len(entries), err)
}
}
internal/api/runtime.go
-1092
View File
File diff suppressed because it is too large Load Diff
internal/api/runtime_auth_test.go
-115
View File
@@ -1,115 +0,0 @@
package api
import (
"fmt"
"io"
"net/http"
"net/http/cookiejar"
"net/http/httptest"
"net/url"
"strings"
"testing"
"time"
)
func TestRuntimeExpiredNavigationBridge(t *testing.T) {
a, req := authFixture(t)
login := req("POST", "/api/login", `{"username":"alice","password":"password-123"}`, "", nil)
jar, _ := cookiejar.New(nil)
base, _ := url.Parse(a.origin)
cookies := login.Result().Cookies()
sid := cookieValue(cookies, "dsh_session")
csrf := cookieValue(cookies, "dsh_csrf")
for _, c := range cookies {
if c.Name == "dsh_access" {
c.Value = a.accessJWT(sid, "alice", time.Now().Add(-time.Minute))
}
}
jar.SetCookies(base, cookies)
call := func(method, path, mode, dest, origin, upgrade string) *httptest.ResponseRecorder {
u, _ := url.Parse(a.origin + path)
r := httptest.NewRequest(method, u.String(), nil)
r.Header.Set("Sec-Fetch-Mode", mode)
r.Header.Set("Sec-Fetch-Dest", dest)
r.Header.Set("Origin", origin)
r.Header.Set("Upgrade", upgrade)
for _, c := range jar.Cookies(u) {
r.AddCookie(c)
}
if _, err := r.Cookie("dsh_refresh"); err == nil {
t.Fatal("refresh leaked to runtime path")
}
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
return w
}
w := call("GET", "/dsh/demo?x=1&token=secret", "navigate", "document", "", "")
if w.Code != 303 || w.Header().Get("Cache-Control") != "no-store" || len(w.Result().Cookies()) != 0 {
t.Fatal(w.Code, w.Header())
}
location, _ := url.Parse(w.Header().Get("Location"))
values, _ := url.ParseQuery(location.Fragment[len("runtime-renew?"):])
if location.Path != "/" || values.Get("return") != "/dsh/demo?x=1" {
t.Fatal(location)
}
for _, tc := range []struct{ method, mode, dest, origin, upgrade string }{
{"GET", "cors", "empty", a.origin, ""},
{"GET", "navigate", "iframe", "https://evil.test", ""},
{"GET", "", "", a.origin, "websocket"},
{"GET", "navigate", "document", a.origin, "websocket"},
{"POST", "navigate", "document", a.origin, ""},
{"GET", "navigate", "document", "https://evil.test", ""},
} {
if w := call(tc.method, "/dsh/demo/", tc.mode, tc.dest, tc.origin, tc.upgrade); w.Code != 401 || w.Header().Get("Location") != "" {
t.Fatal(tc, w.Code)
}
}
refreshURL, _ := url.Parse(a.origin + "/api/refresh")
if w := call("GET", "/dsh/demo/", "navigate", "iframe", "", ""); w.Code != 303 {
t.Fatal("expired iframe did not reach renewal bridge", w.Code)
}
jar.SetCookies(base, []*http.Cookie{{Name: "dsh_access", Path: "/", MaxAge: -1}})
if w := call("GET", "/dsh/demo/", "navigate", "document", "", ""); w.Code != 303 {
t.Fatal("browser-expired cookie did not reach bridge", w.Code)
}
if w := call("GET", "/dsh/demo/ws", "websocket", "empty", a.origin, "websocket"); w.Code != 401 {
t.Fatal("bare sid accepted", w.Code)
}
expired := jar.Cookies(refreshURL)
if w := req("POST", "/api/refresh", "", "", expired); w.Code != 403 {
t.Fatal("CSRF bypass", w.Code)
}
fresh := req("POST", "/api/refresh", "", csrf, expired)
if fresh.Code != 200 {
t.Fatal(fresh.Code, fresh.Body.String())
}
jar.SetCookies(base, fresh.Result().Cookies())
if w := call("GET", "/dsh/demo/", "navigate", "document", "", ""); w.Code != 404 {
t.Fatal("renewed navigation", w.Code)
}
if w := call("GET", "/dsh/demo/ws", "websocket", "empty", a.origin, "websocket"); w.Code != 404 {
t.Fatal("renewed WS authentication", w.Code)
}
a.s.DeleteSession(sid)
if w := call("GET", "/dsh/demo/ws", "websocket", "empty", a.origin, "websocket"); w.Code != 401 {
t.Fatal("revoked WS accepted", w.Code)
}
}
func TestRuntimeSessionBootstrap(t *testing.T) {
target, _ := url.Parse("http://runtime.test")
proxy := runtimeReverseProxy(target, "/dsh/demo", "panel.test", "runtime=secret")
response := &http.Response{StatusCode: 200, Header: http.Header{"Content-Type": {"text/html"}, "Set-Cookie": {"secret=runtime"}, "Etag": {"old"}, "Content-Length": {"1"}, "Cache-Control": {"public"}}, Body: io.NopCloser(strings.NewReader("<html><head></head><body>runtime</body></html>"))}
if err := proxy.ModifyResponse(response); err != nil {
t.Fatal(err)
}
data, _ := io.ReadAll(response.Body)
if !strings.Contains(string(data), `<head><script src="/api/runtime-session.js"></script>`) || response.ContentLength != int64(len(data)) || response.Header.Get("Set-Cookie") != "" || response.Header.Get("ETag") != "" || response.Header.Get("Cache-Control") != "no-store" || response.Header.Get("Content-Length") != fmt.Sprint(len(data)) {
t.Fatal(string(data), response.Header)
}
w := httptest.NewRecorder()
runtimeSessionScript(w, httptest.NewRequest("GET", "/api/runtime-session.js", nil))
if w.Code != 200 || !strings.Contains(w.Body.String(), "240000") || !strings.Contains(w.Body.String(), `"X-CSRF-Token":csrf.slice(9)`) || strings.Contains(w.Body.String(), "dsh_refresh=") {
t.Fatal(w.Code, w.Body.String())
}
}
internal/api/runtime_free_search_test.go
-141
View File
@@ -1,141 +0,0 @@
package api
import (
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"os/exec"
"strings"
"sync/atomic"
"testing"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func TestRuntimeFreeSearchProxy(t *testing.T) {
var forwarded atomic.Int32
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("token") == "launch" {
http.SetCookie(w, &http.Cookie{Name: "runtime", Value: r.Host})
w.WriteHeader(303)
return
}
forwarded.Add(1)
cookie, err := r.Cookie("runtime")
if err != nil || cookie.Value != r.Host {
t.Error("Host-bound cookie mismatch", r.Host, cookie)
}
if r.Header.Get("Authorization") != "" || r.Header.Get("X-CSRF-Token") != "" || strings.Contains(r.Header.Get("Cookie"), "dsh_session") {
t.Error("panel credentials leaked")
}
if runtimeFreeSearchPath(r.URL.Path) {
remote, _, _ := net.SplitHostPort(r.RemoteAddr)
if !net.ParseIP(remote).IsLoopback() || r.Host != "localhost" || r.Header.Get("Origin") != "http://localhost" {
t.Error("loopback guard failed", r.Host, r.Header)
}
data, _ := io.ReadAll(r.Body)
if string(data) != "payload" || r.URL.RawQuery != "keep=1" {
t.Error("request changed", string(data), r.URL.RawQuery)
}
} else if r.Host != "public.example:8443" || r.Header.Get("Origin") != "https://public.example:8443" {
t.Error("ordinary resource authority changed", r.Host)
}
w.WriteHeader(204)
}))
defer upstream.Close()
old := runtimeTransport
transport := old.Clone()
transport.DialContext = func(ctx context.Context, network, address string) (net.Conn, error) {
if address != "192.0.2.10:8080" {
return nil, fmt.Errorf("not the Caddy target: %s", address)
}
return (&net.Dialer{}).DialContext(ctx, network, strings.TrimPrefix(upstream.URL, "http://"))
}
runtimeTransport = transport
defer func() { runtimeTransport = old; transport.CloseIdleConnections() }()
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/json") {
fmt.Fprintf(w, `{"State":{"Running":true,"StartedAt":"1970-01-01T00:00:01Z"},"NetworkSettings":{"Networks":{%q:{"IPAddress":"192.0.2.10"}}}}`, runtimeNetwork("demo"))
} else {
io.WriteString(w, "http://localhost/?token=launch\n")
}
})
a.origin = "https://public.example:8443"
defer runtimeAuthCache.Delete(runtimeAuthKey{a, "demo"})
for _, i := range []store.Instance{{ID: "demo", UID: "owner", ContainerID: "container"}, {ID: "foreign", UID: "other"}} {
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
}
call := func(method, path, origin, session, site string, want int) {
t.Helper()
r := runtimeRequest(method, path, "payload", session)
r.Host = "public.example:8443"
r.Header.Set("Origin", origin)
r.Header.Set("Sec-Fetch-Site", site)
r.Header.Set("Authorization", "Bearer panel-secret")
r.Header.Set("X-CSRF-Token", "panel-secret")
w := httptest.NewRecorder()
a.Handler().ServeHTTP(w, r)
if w.Code != want {
t.Fatalf("%s %s origin=%q: %d %s", method, path, origin, w.Code, w.Body.String())
}
}
const bridge = "/api/dsh-free-search-settings/"
for _, endpoint := range []string{"describe", "mutate", "credentials-status", "credentials-set", "credentials-unset", "check-update", "update", "raw-search"} {
call("POST", "/dsh/demo"+bridge+endpoint+"?keep=1&token=browser", a.origin, sid, "same-origin", 204)
}
for _, path := range []string{"/assets/client.js", bridge + "unknown", bridge + "describe/", bridge + "describe-extra", "/api/dsh-free-search-settings-evil/describe"} {
call("POST", "/dsh/demo"+path, a.origin, sid, "same-origin", 204)
}
call("POST", "/dsh/demo"+bridge+"describe?keep=1", a.origin, sid, "same-origin", 204)
count := forwarded.Load()
for _, origin := range []string{"", "null", "https://evil.example", "http://localhost", "https://public.example:8443.evil"} {
call("POST", "/dsh/demo"+bridge+"mutate", origin, sid, "", 403)
}
call("POST", "/dsh/demo"+bridge+"mutate", a.origin, sid, "cross-site", 403)
call("POST", "/dsh/demo"+bridge+"mutate", a.origin, "", "", 401)
call("POST", "/dsh/foreign"+bridge+"mutate", a.origin, sid, "", 404)
call("GET", "/dsh/demo"+bridge+"mutate", "", sid, "", 405)
call("POST", bridge+"mutate", a.origin, "", "", 403)
call("GET", bridge+"describe", "", "", "", 404)
if forwarded.Load() != count {
t.Fatal("denied request reached runtime")
}
}
func TestRuntimeFreeSearchFetchSemantics(t *testing.T) {
node, err := exec.LookPath("node")
if err != nil {
t.Skip("Node required for Fetch API semantics")
}
script := `const assert = require("node:assert/strict"); global.window=global; global.location=new URL("https://panel.test/dsh/demo/"); global.document={baseURI:location.href,addEventListener(){}}; global.addEventListener=()=>{}; global.setInterval=()=>{}; let last; global.fetch=async (input,init)=>{last=[input,init]; return new Request(input,init)};` + runtimeSessionJS +
`(async()=>{
const path='/api/dsh-free-search-settings/mutate';
for (const input of [path,new URL(path,location)]) {
const result=await fetch(input,{method:'POST',body:'string-body',headers:{'X-Test':'yes'},credentials:'include'});
assert.equal(result.url,location.origin+'/dsh/demo'+path);
assert.equal(await result.text(),'string-body'); assert.equal(result.headers.get('X-Test'),'yes'); assert.equal(result.credentials,'include');
}
const abort=new AbortController();
const input=new Request(new URL(path,location),{method:'POST',body:'request-body',headers:{'X-Test':'request'},signal:abort.signal,credentials:'omit',redirect:'manual',cache:'no-store'});
const result=await fetch(input);
assert.equal(await result.text(),'request-body'); assert.equal(result.headers.get('X-Test'),'request'); assert.equal(result.credentials,'omit'); assert.equal(result.redirect,'manual'); assert.equal(result.cache,'no-store'); abort.abort(); assert.equal(result.signal.aborted,true);
const override=await fetch(new Request(new URL(path,location),{method:'POST',body:'old'}),{body:'new',headers:{'X-Test':'override'}});
assert.equal(await override.text(),'new'); assert.equal(override.headers.get('X-Test'),'override');
const stream = new ReadableStream({start(c){c.enqueue(new TextEncoder().encode('stream-body'));c.close()}});
const streamed=await fetch(new Request(new URL(path,location),{method:'POST',body:stream,duplex:'half'}));
assert.equal(await streamed.text(),'stream-body');
const used=new Request(new URL(path,location),{method:'POST',body:'used'}); await used.text();
await assert.rejects(fetch(used),TypeError);
for(const path of ['/api/refresh','/api/dsh-free-search-settings/mutate/','/api/dsh-free-search-settings/mutate-extra','/dsh/demo/api/dsh-free-search-settings/mutate','https://evil.test/api/dsh-free-search-settings/mutate']) {
const input=new URL(path,location); const init={method:'POST',body:'unchanged'}; await fetch(input,init); assert.equal(last[0],input); assert.equal(last[1],init);
}
})().catch(e=>{console.error(e);process.exit(1)});`
if output, err := exec.Command(node, "-e", script).CombinedOutput(); err != nil {
t.Fatalf("%v: %s", err, output)
}
}
internal/api/runtime_office_test.go
-167
View File
@@ -1,167 +0,0 @@
package api
import (
"context"
"io"
"net/http"
"net/http/httptest"
"net/http/httputil"
"net/url"
"strings"
"sync/atomic"
"testing"
"time"
)
func officeTestProxy(t *testing.T, upstream string, officeTimeout time.Duration) *httputil.ReverseProxy {
t.Helper()
target, err := url.Parse(upstream)
if err != nil {
t.Fatal(err)
}
normal := runtimeTransport.Clone()
office := runtimeOfficeTransport.Clone()
normal.ResponseHeaderTimeout = 30 * time.Millisecond
office.ResponseHeaderTimeout = officeTimeout
t.Cleanup(normal.CloseIdleConnections)
t.Cleanup(office.CloseIdleConnections)
proxy := runtimeReverseProxy(target, "/dsh/office-test", "panel.test", "runtime=secret")
proxy.Transport = runtimeProxyTransport{normal: normal, office: office}
return proxy
}
func TestRuntimeOfficeTransportDefaults(t *testing.T) {
if got := runtimeTransport.ResponseHeaderTimeout; got != 20*time.Second {
t.Errorf("normal timeout = %s, want 20s", got)
}
if got := runtimeOfficeTransport.ResponseHeaderTimeout; got != 120*time.Second {
t.Errorf("office timeout = %s, want 120s", got)
}
if runtimeTransport == runtimeOfficeTransport {
t.Fatal("office transport must be separate")
}
target, _ := url.Parse("http://panel.test")
proxy := runtimeReverseProxy(target, "/dsh/office-test", "panel.test", "")
transport, ok := proxy.Transport.(runtimeProxyTransport)
if !ok || transport.normal != runtimeTransport || transport.office != runtimeOfficeTransport {
t.Fatalf("unexpected default transport: %#v", proxy.Transport)
}
}
func TestRuntimeOfficeProxyRouting(t *testing.T) {
for _, path := range []string{
"/api/officeToPdf/render",
"/api/status",
"/api/officeToPdf/render/",
"/api/officeToPdf/render/child",
"/api/officeToPdf/renderOther",
"/api/officetopdf/render",
"/api/officeToPdf/status",
} {
t.Run(path, func(t *testing.T) {
const body = "document payload"
const query = "file=report%20one.docx&mode=pdf&mode=preview"
var calls atomic.Int32
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
data, err := io.ReadAll(r.Body)
if err != nil || string(data) != body || r.Method != http.MethodPost {
t.Errorf("upstream method/body = %s %q, error %v", r.Method, data, err)
}
if r.URL.Path != path || r.URL.RawQuery != query {
t.Errorf("upstream URL = %s", r.URL)
}
if r.Host != "panel.test" || r.Header.Get("X-Forwarded-Prefix") != "/dsh/office-test" || r.Header.Get("Cookie") != "runtime=secret" {
t.Errorf("upstream host/headers = %s %v", r.Host, r.Header)
}
time.Sleep(100 * time.Millisecond)
w.Header().Set("Content-Type", "application/pdf")
_, _ = io.WriteString(w, "rendered PDF")
}))
defer upstream.Close()
proxy := officeTestProxy(t, upstream.URL, 500*time.Millisecond)
req := httptest.NewRequest(http.MethodPost, "/dsh/office-test"+path+"?"+query+"&token=removed", strings.NewReader(body))
req.Header.Set("Cookie", "panel=private")
originalURL := req.URL.String()
w := httptest.NewRecorder()
proxy.ServeHTTP(w, req)
want := http.StatusBadGateway
if path == "/api/officeToPdf/render" {
want = http.StatusOK
if w.Body.String() != "rendered PDF" {
t.Errorf("response body = %q", w.Body.String())
}
}
if w.Code != want {
t.Errorf("status = %d, want %d: %s", w.Code, want, w.Body.String())
}
if req.URL.String() != originalURL || req.Header.Get("Cookie") != "panel=private" {
t.Error("proxy mutated incoming URL or cookies")
}
upstream.Close()
if got := calls.Load(); got != 1 {
t.Errorf("mutating request sent %d times, want exactly once", got)
}
})
}
}
func TestRuntimeOfficeProxyFiniteTimeoutAndCancellation(t *testing.T) {
for _, name := range []string{"header timeout", "caller cancellation"} {
t.Run(name, func(t *testing.T) {
started := make(chan struct{}, 2)
canceled := make(chan struct{}, 2)
var calls atomic.Int32
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
_, _ = io.Copy(io.Discard, r.Body)
started <- struct{}{}
select {
case <-r.Context().Done():
canceled <- struct{}{}
case <-time.After(2 * time.Second):
}
}))
defer upstream.Close()
officeTimeout := 100 * time.Millisecond
if name == "caller cancellation" {
officeTimeout = 5 * time.Second
}
proxy := officeTestProxy(t, upstream.URL, officeTimeout)
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
req := httptest.NewRequest(http.MethodPost, "/dsh/office-test/api/officeToPdf/render?mode=pdf", strings.NewReader("document")).WithContext(ctx)
w := httptest.NewRecorder()
done := make(chan struct{})
go func() {
proxy.ServeHTTP(w, req)
close(done)
}()
select {
case <-started:
case <-time.After(time.Second):
t.Fatal("request never reached upstream")
}
if name == "caller cancellation" {
cancel()
}
select {
case <-done:
case <-time.After(500 * time.Millisecond):
t.Fatal("proxy did not promptly finish")
}
if w.Code != http.StatusBadGateway {
t.Errorf("status = %d, want 502", w.Code)
}
select {
case <-canceled:
case <-time.After(500 * time.Millisecond):
t.Fatal("upstream context was not canceled")
}
upstream.Close()
if got := calls.Load(); got != 1 {
t.Errorf("mutating request sent %d times, want exactly once", got)
}
})
}
}
internal/api/runtime_persistence_test.go
-195
View File
@@ -1,195 +0,0 @@
package api
import (
"bytes"
"context"
"encoding/binary"
"io"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"testing/iotest"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func runtimeTestFrame(payload []byte) []byte {
var h [8]byte
h[0] = 1
binary.BigEndian.PutUint32(h[4:], uint32(len(payload)))
return append(h[:], payload...)
}
func TestRuntimeTokenFromStreamLargeRawAndFramed(t *testing.T) {
for _, tc := range []struct {
name string
data []byte
token string
}{
{"raw", append(bytes.Repeat([]byte("x"), 9<<20), []byte("\ndsh web: http://runtime/?token=raw-tail\n")...), "raw-tail"},
{"framed", runtimeTestFrame(append(bytes.Repeat([]byte("x"), 9<<20), []byte("\ndsh web: http://runtime/?token=framed-tail\n")...)), "framed-tail"},
} {
t.Run(tc.name, func(t *testing.T) {
got, err := runtimeTokenFromStream(bytes.NewReader(tc.data))
if err != nil || got != tc.token {
t.Fatalf("token=%q err=%v", got, err)
}
})
}
}
func TestRuntimeTokenFromStreamSplitLatestAndTruncatedFrame(t *testing.T) {
data := []byte("dsh web: http://runtime/?token=first-token\ndsh web: http://runtime/?token=second-token\n")
got, err := runtimeTokenFromStream(iotest.OneByteReader(bytes.NewReader(data)))
if err != nil || got != "second-token" {
t.Fatalf("token=%q err=%v", got, err)
}
framed := runtimeTestFrame([]byte("dsh web: http://runtime/?token=truncated"))
framed = framed[:len(framed)-2]
if _, err := runtimeTokenFromStream(bytes.NewReader(framed)); err == nil {
t.Fatal("truncated frame accepted")
}
}
func TestRuntimeCookiePersistenceAndHostExchange(t *testing.T) {
var mu sync.Mutex
logs, exchanges := 0, 0
var seen []string
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
defer mu.Unlock()
if r.URL.Path == "/" {
exchanges++
token := r.URL.Query().Get("token")
seen = append(seen, token)
w.Header().Set("Set-Cookie", "session="+token+"-"+r.Host+"; Max-Age=60")
w.WriteHeader(http.StatusSeeOther)
return
}
if strings.HasSuffix(r.URL.Path, "/logs") {
logs++
io.WriteString(w, "dsh web: http://runtime/?token=persist-token\n")
return
}
w.WriteHeader(http.StatusNoContent)
}))
defer upstream.Close()
t.Setenv("DOCKER_HOST", upstream.URL)
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {})
a.docker = docker.New(upstream.URL)
i := store.Instance{ID: "persist", UID: "owner", ContainerID: "container"}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
target, _ := url.Parse(upstream.URL)
ctx := context.Background()
if got, err := a.runtimeCookie(ctx, i, "2025-01-01T00:00:00Z", target, "panel.test"); err != nil || got != "session=persist-token-panel.test" {
t.Fatalf("first cookie=%q err=%v", got, err)
}
b := New(a.s, a.root)
b.docker = docker.New(upstream.URL)
if got, err := b.runtimeCookie(ctx, i, "2025-01-01T00:00:00Z", target, "panel.test"); err != nil || got != "session=persist-token-panel.test" {
t.Fatalf("persisted cookie=%q err=%v", got, err)
}
if got, err := b.runtimeCookie(ctx, i, "2025-01-01T00:00:00Z", target, "localhost"); err != nil || got != "session=persist-token-localhost" {
t.Fatalf("localhost cookie=%q err=%v", got, err)
}
if got, err := b.runtimeCookie(ctx, i, "2025-01-01T00:00:00Z", target, "panel.test"); err != nil || got != "session=persist-token-panel.test" {
t.Fatalf("host cookie=%q err=%v", got, err)
}
proxy := runtimeReverseProxy(target, "/dsh/persist", "panel.test", "session=persist-token-panel.test")
b.invalidateRejectedRuntimeCookie(proxy, i, "2025-01-01T00:00:00Z", "panel.test", "session=persist-token-panel.test")
if err := proxy.ModifyResponse(&http.Response{StatusCode: 401, Header: make(http.Header), Body: io.NopCloser(strings.NewReader("denied"))}); err != nil {
t.Fatal(err)
}
if got, err := b.runtimeCookie(ctx, i, "2025-01-01T00:00:00Z", target, "panel.test"); err != nil || got != "session=persist-token-panel.test" {
t.Fatalf("reexchange: %q %v", got, err)
}
if _, err := b.runtimeCookie(ctx, i, "2025-01-02T00:00:00Z", target, "panel.test"); err != nil {
t.Fatalf("changed start time: %v", err)
}
mu.Lock()
defer mu.Unlock()
if logs != 2 {
t.Fatalf("log reads=%d, want 2", logs)
}
if exchanges != 4 {
t.Fatalf("exchanges=%d, want 4", exchanges)
}
for _, token := range seen {
if token != "persist-token" {
t.Fatalf("exchanged token=%q", token)
}
}
}
func TestRuntimeAuthDiskEncryptedAndTamperRejected(t *testing.T) {
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {})
entry := runtimeAuthEntry{Container: "container", Started: "2025-01-01T00:00:00Z", Token: "super-secret-token", Host: "panel.test", Cookies: "session=super-secret-token"}
key := entry.Container + "/" + entry.Started
if err := a.saveRuntimeAuth("instance", key, entry); err != nil {
t.Fatal(err)
}
path := a.runtimeAuthPath("instance", key)
st, err := os.Stat(path)
if err != nil || st.Mode().Perm() != 0600 {
t.Fatalf("stat=%v mode=%v", err, st.Mode().Perm())
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if bytes.Contains(data, []byte(entry.Token)) || bytes.Contains(data, []byte(entry.Cookies)) {
t.Fatal("runtime token stored in plaintext")
}
data[len(data)/2] ^= 1
if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err)
}
if _, ok := a.loadRuntimeAuth("instance", key); ok {
t.Fatal("tampered runtime auth accepted")
}
}
func TestRuntimeCookieSaveFailureDoesNotCache(t *testing.T) {
var mu sync.Mutex
logs := 0
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/logs") {
mu.Lock()
logs++
mu.Unlock()
io.WriteString(w, "dsh web: http://runtime/?token=save-failure-token\n")
return
}
w.Header().Set("Set-Cookie", "session=save-failure; Max-Age=60")
w.WriteHeader(http.StatusSeeOther)
}))
defer upstream.Close()
t.Setenv("DOCKER_HOST", upstream.URL)
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {})
badRoot := filepath.Join(t.TempDir(), "not-a-directory")
a.docker = docker.New(upstream.URL)
if err := os.WriteFile(badRoot, []byte("x"), 0600); err != nil {
t.Fatal(err)
}
a.root = badRoot
i := store.Instance{ID: "save-failure", UID: "owner", ContainerID: "container"}
target, _ := url.Parse(upstream.URL)
for n := 0; n < 2; n++ {
if _, err := a.runtimeCookie(context.Background(), i, "2025-01-01T00:00:00Z", target, "panel.test"); err == nil {
t.Fatal("save failure accepted")
}
}
mu.Lock()
defer mu.Unlock()
if logs != 2 {
t.Fatalf("log reads=%d after save failures, want 2", logs)
}
}
internal/api/runtime_session.go
-60
View File
@@ -1,60 +0,0 @@
package api
import "net/http"
func runtimeSessionScript(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
w.Header().Set("Content-Type", "application/javascript; charset=utf-8")
w.Header().Set("Cache-Control", "no-store")
w.Write([]byte(runtimeSessionJS))
}
const runtimeSessionJS = `(function () {
const prefix = location.pathname.match(/^\/dsh\/[^/]+(?=\/|$)/);
if (prefix) {
const originalFetch = window.fetch;
const bridge = /^\/api\/dsh-free-search-settings\/(describe|mutate|credentials-status|credentials-set|credentials-unset|check-update|update|raw-search)$/;
window.fetch = async function (input, init) {
let url;
try { url = new URL(input instanceof Request ? input.url : input, document.baseURI); } catch (_) {}
if (url && url.origin === location.origin && bridge.test(url.pathname)) {
url.pathname = prefix[0] + url.pathname;
if (input instanceof Request) {
const request = new Request(input, init);
return originalFetch.call(this, new Request(url, request));
}
input = url;
}
return originalFetch.call(this, input, init);
};
}
let pending;
async function renew() {
if (pending) return pending;
const run = async function () {
// Renew proactively: a successful /me does not extend the five-minute
// access lifetime. Web Locks serialize cookie rotation across pages.
const csrf = document.cookie.split("; ").find(c => c.startsWith("dsh_csrf="));
if (!csrf) return;
const response = await fetch("/api/refresh", {method:"POST", credentials:"same-origin", headers:{"X-CSRF-Token":csrf.slice(9)}});
if (response.status === 401 || response.status === 403 || response.status === 409) {
for (const delay of [0, 100, 250, 500]) {
if (delay) await new Promise(resolve => setTimeout(resolve, delay));
const check = await fetch("/api/me", {credentials:"same-origin"});
if (check.status !== 401) return;
}
if (response.status === 401 || response.status === 403) location.reload();
}
};
pending = (navigator.locks ? navigator.locks.request("dsh-session-refresh", run) : run())
.catch(function () {}).finally(function () { pending = undefined; });
return pending;
}
setInterval(renew, 240000);
window.addEventListener("pageshow", renew);
window.addEventListener("online", renew);
document.addEventListener("visibilitychange", function () { if (!document.hidden) renew(); });
})();`
internal/api/runtime_test.go
-635
View File
@@ -1,635 +0,0 @@
package api
import (
"bufio"
"context"
"encoding/binary"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func runtimeTestAPI(t *testing.T, handler http.HandlerFunc) (*API, string) {
t.Helper()
s, err := store.Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatal(err)
}
t.Cleanup(s.Close)
if err := s.CreateUser(store.User{ID: "owner", Username: "owner"}, "unused"); err != nil {
t.Fatal(err)
}
sid, _, err := s.Session("owner")
if err != nil {
t.Fatal(err)
}
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
a := &API{s: s, root: t.TempDir(), origin: "http://panel.test", docker: docker.New(server.URL)}
if err := a.initAuth(); err != nil {
t.Fatal(err)
}
registerTestAccess(t, a, sid)
return a, sid
}
func runtimeRequest(method, path, body, sid string) *http.Request {
r := httptest.NewRequest(method, path, strings.NewReader(body))
r.Host = "panel.test"
r.AddCookie(&http.Cookie{Name: "dsh_session", Value: sid})
addTestAccess(r, sid)
return r
}
func TestRuntimeProxyRejectsErrorState(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
t.Error("error-state proxy must not contact Docker")
})
if err := a.s.AddInstance(store.Instance{ID: "broken", UID: "owner", Status: "error", Created: time.Now()}); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.proxy(w, runtimeRequest("GET", "/dsh/broken/", "", sid))
if w.Code != http.StatusConflict {
t.Fatalf("status = %d", w.Code)
}
}
// createRuntime holds the instance lock during creation; a concurrent DELETE
// must wait instead of removing the row while the container is still starting.
func TestInstanceDeleteWaitsForInFlightCreate(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method == "GET" && strings.Contains(r.URL.Path, "/networks/") {
w.WriteHeader(404)
return
}
if r.Method == http.MethodGet && r.URL.Path == "/v1.47/containers/dsh-inst-creating/json" {
w.WriteHeader(http.StatusNotFound)
return
}
if r.Method == http.MethodDelete && strings.Contains(r.URL.Path, "/networks/") {
w.WriteHeader(http.StatusNoContent)
return
}
t.Errorf("unexpected Docker call %s %s", r.Method, r.URL.Path)
})
if err := a.s.AddInstance(store.Instance{ID: "creating", UID: "owner", Status: "creating", Created: time.Now()}); err != nil {
t.Fatal(err)
}
inst := a.instanceLocks.Acquire("creating")
done := make(chan *httptest.ResponseRecorder, 1)
go func() {
w := httptest.NewRecorder()
a.instance(w, runtimeRequest(http.MethodDelete, "/api/instances/creating", "", sid))
done <- w
}()
select {
case w := <-done:
t.Fatalf("delete interleaved with creation: %d %s", w.Code, w.Body.String())
case <-time.After(300 * time.Millisecond):
}
inst.Release()
select {
case w := <-done:
if w.Code != 200 {
t.Fatalf("status %d: %s", w.Code, w.Body.String())
}
case <-time.After(5 * time.Second):
t.Fatal("delete stalled after creation finished")
}
if _, err := a.s.GetInstance("creating"); err == nil {
t.Fatal("instance row retained")
}
}
func TestRuntimeCreateLimitAndPolicy(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("chown requires root")
}
t.Setenv("HOST_DATA_ROOT", "/host/data")
t.Setenv("PANEL_CONTAINER_ID", "panel")
imageID := "sha256:" + strings.Repeat("a", 64)
var mu sync.Mutex
var bodies []map[string]any
var calls []string
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
mu.Lock()
defer mu.Unlock()
calls = append(calls, r.Method+" "+r.URL.Path)
switch {
case strings.Contains(r.URL.Path, "/images/"):
writeJSON(w, map[string]any{"Id": imageID, "Os": "linux", "Architecture": "amd64", "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": "runtime", "dsh-mgr.version": "1.2.3", "dsh-mgr.recipe": imageRecipeRev}}})
case strings.HasSuffix(r.URL.Path, "/containers/create"):
var body map[string]any
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Error(err)
}
bodies = append(bodies, body)
writeJSON(w, map[string]string{"Id": fmt.Sprintf("container%d", len(bodies))})
default:
w.WriteHeader(204)
}
})
body := fmt.Sprintf("{\"name\":\"one\",\"image_id\":%q}", imageID)
seedVerifiedImage(t, a, imageID, "1.2.3")
var wg sync.WaitGroup
statuses := make(chan int, 4)
for n := 0; n < 4; n++ {
wg.Add(1)
go func() {
defer wg.Done()
w := httptest.NewRecorder()
a.instances(w, runtimeRequest("POST", "/api/instances", body, sid))
statuses <- w.Code
}()
}
wg.Wait()
close(statuses)
successes, limits := 0, 0
for status := range statuses {
if status == 200 {
successes++
} else if status == 409 {
limits++
} else {
t.Errorf("status %d", status)
}
}
if successes != 2 || limits != 2 {
t.Fatalf("success=%d limit=%d", successes, limits)
}
instances, err := a.s.Instances("owner")
if err != nil || len(instances) != 2 {
t.Fatalf("instances=%v err=%v", instances, err)
}
for _, i := range instances {
if i.Version != "1.2.3" || i.ContainerID == "" || i.Status != "running" {
t.Errorf("%+v", i)
}
if _, err := os.Stat(filepath.Join(a.root, "workspaces", "owner", "instances", i.ID, "home")); err != nil {
t.Error(err)
}
}
for _, b := range bodies {
h := b["HostConfig"].(map[string]any)
mounts := h["Mounts"].([]any)
if len(mounts) != 1 {
t.Fatalf("unexpected mounts: %v", mounts)
}
mount := mounts[0].(map[string]any)
if mount["Target"] != "/workspace" || !strings.HasSuffix(mount["Source"].(string), "/home") {
t.Fatal(mount)
}
if !strings.Contains(fmt.Sprint(b["Env"]), "DSH_HOME=/workspace") {
t.Fatal("wrong data root", b["Env"])
}
if b["User"] != "1000:1000" || h["ReadonlyRootfs"] != true || h["Memory"] != float64(2<<30) || h["NanoCpus"] != float64(2e9) || h["PidsLimit"] != float64(256) {
t.Errorf("policy %v", b)
}
if h["PortBindings"] != nil || h["PublishAllPorts"] != nil {
t.Error("ports published")
}
if fmt.Sprint(h["CapDrop"]) != "[ALL]" || !strings.Contains(fmt.Sprint(h["Tmpfs"]), "exec") {
t.Error(h)
}
if strings.Contains(fmt.Sprint(b["Cmd"]), "public-url") || !strings.Contains(fmt.Sprint(b["Cmd"]), "--trusted-host panel.test") {
t.Error(b["Cmd"])
}
if !strings.Contains(fmt.Sprint(h["Mounts"]), "/host/data/workspaces/owner") {
t.Error(h["Mounts"])
}
}
if !strings.Contains(strings.Join(calls, "\n"), "/connect") {
t.Fatal("panel not connected")
}
}
func TestRuntimeLifecycleAndOwnership(t *testing.T) {
t.Setenv("PANEL_CONTAINER_ID", "")
var calls []string
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
calls = append(calls, r.Method+" "+r.URL.Path)
if r.Method == "GET" && strings.Contains(r.URL.Path, "/networks/") {
w.WriteHeader(404)
return
}
if strings.HasSuffix(r.URL.Path, "/json") {
io.WriteString(w, "{\"State\":{\"Status\":\"running\"}}")
} else if strings.HasSuffix(r.URL.Path, "/logs") {
io.WriteString(w, "dsh web: http://localhost/?token=secret-value\nAuthorization: Bearer another-secret\n")
} else {
w.WriteHeader(204)
}
})
i := store.Instance{ID: "instance", UID: "owner", ContainerID: "container"}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
dir := filepath.Join(a.root, "workspaces", "owner", "instances", i.ID, "home")
if err := os.MkdirAll(dir, 0755); err != nil {
t.Fatal(err)
}
for _, action := range []string{"start", "stop", "restart"} {
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("POST", "/api/instances/instance/"+action, "", sid))
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
w = httptest.NewRecorder()
a.instance(w, runtimeRequest("GET", "/api/instances/instance/"+action, "", sid))
if w.Code != 405 {
t.Fatal(w.Code)
}
}
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("GET", "/api/instances/instance/logs", "", sid))
if strings.Contains(w.Body.String(), "secret-value") || strings.Contains(w.Body.String(), "another-secret") {
t.Fatal("credentials in logs")
}
foreign := store.Instance{ID: "foreign", UID: "someone", ContainerID: "other"}
a.s.AddInstance(foreign)
before := len(calls)
for _, path := range []string{"/api/instances/foreign/start", "/api/instances/foreign/logs"} {
w = httptest.NewRecorder()
a.instance(w, runtimeRequest("POST", path, "", sid))
if w.Code != 404 {
t.Fatal(w.Code)
}
}
w = httptest.NewRecorder()
a.proxy(w, runtimeRequest("GET", "/dsh/foreign/", "", sid))
if w.Code != 404 || len(calls) != before {
t.Fatal("owner check bypass")
}
w = httptest.NewRecorder()
a.instance(w, runtimeRequest("DELETE", "/api/instances/instance", "", sid))
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
if _, err := os.Stat(dir); !os.IsNotExist(err) {
t.Fatal("instance data retained", err)
}
if _, err := a.s.GetInstance(i.ID); err == nil {
t.Fatal("row retained")
}
if !strings.Contains(strings.Join(calls, "\n"), "DELETE /v1.47/containers/container") {
t.Fatal(calls)
}
}
func TestRuntimeTokenOriginAndExchange(t *testing.T) {
logs := "dsh web: http://localhost/?token=old\ndsh web: http://localhost/?token=latest%2Btoken\x1b[0m\n"
if got := latestRuntimeToken(logs); got != "latest+token" {
t.Fatal(got)
}
if got := latestRuntimeToken(logs + `{"uri":"/?token=old"}`); got != "latest+token" {
t.Fatalf("proxy log replaced startup token: %q", got)
}
if strings.Contains(redactRuntimeLogs(logs), "latest") || strings.Contains(redactRuntimeLogs(logs), "old") {
t.Fatal("token not redacted")
}
a := &API{origin: "http://panel.test"}
for _, tc := range []struct {
method, origin, upgrade string
want bool
}{{"GET", "", "", true}, {"POST", "", "", false}, {"POST", "http://panel.test", "", true}, {"POST", "http://evil.test", "", false}, {"GET", "", "websocket", false}, {"GET", "http://panel.test", "websocket", true}} {
r := runtimeRequest(tc.method, "/dsh/id/", "", "")
r.Header.Set("Origin", tc.origin)
r.Header.Set("Upgrade", tc.upgrade)
if a.runtimeOriginAllowed(r) != tc.want {
t.Errorf("%+v", tc)
}
}
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Method != "GET" || r.URL.Query().Get("token") != "latest+token" || r.Host != "panel.test" {
t.Error("invalid exchange")
}
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: "server-secret", Path: "/"})
w.Header().Set("Location", "/")
w.WriteHeader(303)
}))
defer server.Close()
target, _ := url.Parse(server.URL)
cookie, err := exchangeRuntimeCookie(context.Background(), target, "panel.test", "latest+token")
if err != nil || cookie != "dsh_auth=server-secret" {
t.Fatalf("exchange failed: %v", err)
}
}
func TestRuntimeProxyHeadersRedirectAndSSE(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Cookie") != "dsh_auth=server-secret" || r.Header.Get("Authorization") != "" || r.Header.Get("X-CSRF-Token") != "" || r.URL.Query().Get("token") != "" {
t.Error("credential boundary violated")
}
if r.Host != "panel.test" {
t.Error(r.Host)
}
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: "not-for-browser"})
if r.URL.Path == "/redirect" {
w.Header().Set("Location", "/next?token=not-for-browser")
w.WriteHeader(303)
return
}
if r.URL.Path != "/events" {
t.Error(r.URL.Path)
}
w.Header().Set("Content-Type", "text/event-stream")
io.WriteString(w, "data: first\n\n")
w.(http.Flusher).Flush()
<-r.Context().Done()
}))
defer upstream.Close()
target, _ := url.Parse(upstream.URL)
proxy := httptest.NewServer(runtimeReverseProxy(target, "/dsh/id", "panel.test", "dsh_auth=server-secret"))
defer proxy.Close()
client := &http.Client{Timeout: 3 * time.Second, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
for _, path := range []string{"redirect", "events"} {
req, _ := http.NewRequest("GET", proxy.URL+"/dsh/id/"+path+"?token=browser-token", nil)
req.Header.Set("Cookie", "dsh_session=panel-secret")
req.Header.Set("Authorization", "Bearer panel-secret")
req.Header.Set("X-CSRF-Token", "panel-secret")
resp, err := client.Do(req)
if err != nil {
t.Fatal(err)
}
if resp.Header.Get("Set-Cookie") != "" {
t.Fatal("cookie escaped")
}
if path == "redirect" {
if resp.Header.Get("Location") != "/dsh/id/next" {
t.Fatal(resp.Header.Get("Location"))
}
} else {
line, err := bufio.NewReader(resp.Body).ReadString('\n')
if err != nil || line != "data: first\n" {
t.Fatal("SSE did not flush", err)
}
}
resp.Body.Close()
}
}
func TestRuntimeProxyPreservesComboQuery(t *testing.T) {
const query = "?@deepseek-ai/dsh-client-modules/client.js,@deepseek-ai/dsh-client-ui/client.js&rev=abc"
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.RawQuery != query {
t.Errorf("combo query changed: %q", r.URL.RawQuery)
}
w.Header().Set("Location", "/plugins/?"+query+"&%74oken=secret")
w.WriteHeader(http.StatusTemporaryRedirect)
}))
defer upstream.Close()
target, _ := url.Parse(upstream.URL)
proxy := runtimeReverseProxy(target, "/dsh/id", "panel.test", "")
for _, suffix := range []string{"", "&token=secret", "&%74oken=secret&token=second"} {
w := httptest.NewRecorder()
proxy.ServeHTTP(w, httptest.NewRequest("GET", "http://panel.test/dsh/id/plugins/?"+query+suffix, nil))
if w.Code != http.StatusTemporaryRedirect || w.Header().Get("Location") != "/dsh/id/plugins/?"+query {
t.Fatalf("unexpected combo redirect: %d %q", w.Code, w.Header().Get("Location"))
}
}
}
func TestRuntimeProxyWebSocketUpgrade(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/socket" || r.Header.Get("Cookie") != "dsh_auth=secret" || r.Header.Get("Authorization") != "" {
t.Error("invalid upgraded request")
}
conn, rw, err := w.(http.Hijacker).Hijack()
if err != nil {
t.Error(err)
return
}
defer conn.Close()
rw.WriteString("HTTP/1.1 101 Switching Protocols\r\nConnection: Upgrade\r\nUpgrade: websocket\r\n\r\n")
rw.Flush()
data := make([]byte, 4)
if _, err := io.ReadFull(rw, data); err != nil {
t.Error(err)
return
}
rw.Write(data)
rw.Flush()
}))
defer upstream.Close()
target, _ := url.Parse(upstream.URL)
proxy := httptest.NewServer(runtimeReverseProxy(target, "/dsh/id", "panel.test", "dsh_auth=secret"))
defer proxy.Close()
conn, err := net.DialTimeout("tcp", strings.TrimPrefix(proxy.URL, "http://"), time.Second)
if err != nil {
t.Fatal(err)
}
defer conn.Close()
conn.SetDeadline(time.Now().Add(3 * time.Second))
fmt.Fprint(conn, "GET /dsh/id/socket HTTP/1.1\r\nHost: panel.test\r\nConnection: Upgrade\r\nUpgrade: websocket\r\nAuthorization: Bearer panel-secret\r\n\r\n")
reader := bufio.NewReader(conn)
resp, err := http.ReadResponse(reader, nil)
if err != nil || resp.StatusCode != 101 {
t.Fatalf("upgrade: %v %v", resp, err)
}
fmt.Fprint(conn, "ping")
data := make([]byte, 4)
if _, err := io.ReadFull(reader, data); err != nil || string(data) != "ping" {
t.Fatal("upgrade tunnel failed", err)
}
}
func TestRuntimeCreateRollbackAndImageRejection(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("chown requires root")
}
t.Setenv("PANEL_CONTAINER_ID", "")
t.Setenv("HOST_DATA_ROOT", "")
imageID := "sha256:" + strings.Repeat("b", 64)
for _, role := range []string{"foreign", "runtime"} {
t.Run(role, func(t *testing.T) {
var calls []string
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
calls = append(calls, r.Method+" "+r.URL.Path)
switch {
case r.Method == "GET" && strings.Contains(r.URL.Path, "/networks/"):
name := strings.TrimPrefix(r.URL.Path, "/v1.47/networks/")
writeJSON(w, map[string]any{"Id": name, "Name": name, "Driver": "bridge", "Labels": map[string]string{"dsh-mgr.instance": strings.TrimPrefix(name, "dsh-inst-")}})
case strings.Contains(r.URL.Path, "/images/"):
writeJSON(w, map[string]any{"Id": imageID, "Os": "linux", "Architecture": "amd64", "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": role, "dsh-mgr.version": "1", "dsh-mgr.recipe": imageRecipeRev}}})
case strings.HasSuffix(r.URL.Path, "/containers/create"):
writeJSON(w, map[string]string{"Id": "container"})
case strings.HasSuffix(r.URL.Path, "/start"):
w.WriteHeader(500)
io.WriteString(w, "token=must-not-leak")
default:
w.WriteHeader(204)
}
})
seedVerifiedImage(t, a, imageID, "1")
w := httptest.NewRecorder()
a.instances(w, runtimeRequest("POST", "/api/instances", fmt.Sprintf("{\"name\":\"test\",\"image_id\":%q}", imageID), sid))
want := 400
if role == "runtime" {
want = 502
}
if w.Code != want {
t.Fatal(w.Code, w.Body.String())
}
if strings.Contains(w.Body.String(), "must-not-leak") {
t.Fatal("docker error leaked")
}
instances, err := a.s.Instances("owner")
if err != nil || len(instances) != 0 {
t.Fatal("reservation not rolled back", err)
}
if role == "foreign" && len(calls) != 1 {
t.Fatal("untrusted image used")
}
if role == "runtime" && !strings.Contains(strings.Join(calls, "\n"), "DELETE /v1.47/containers/container") {
t.Fatal("container not cleaned")
}
})
}
}
func TestRuntimeProxyMissingNetwork(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "{\"State\":{\"Running\":true},\"NetworkSettings\":{\"Networks\":null}}")
})
if err := a.s.AddInstance(store.Instance{ID: "instance", UID: "owner", ContainerID: "container"}); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.proxy(w, runtimeRequest("GET", "/dsh/instance/", "", sid))
if w.Code != 502 {
t.Fatal(w.Code)
}
}
func TestRuntimeCookieCacheRestart(t *testing.T) {
exchanges, logs := 0, 0
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
exchanges++
token := r.URL.Query().Get("token")
if token != "first" && token != "second" {
t.Error("wrong token")
}
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: token})
w.WriteHeader(303)
}))
defer upstream.Close()
target, _ := url.Parse(upstream.URL)
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
logs++
token := "first"
if r.URL.Query().Get("since") == "2.000000000" {
token = "second"
} else if r.URL.Query().Get("since") != "1.000000000" {
t.Error("missing startup filter")
}
io.WriteString(w, "dsh web: http://localhost/?token="+token+"\n")
})
i := store.Instance{ID: "cache-test", ContainerID: "container"}
defer runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
for n, started := range []string{"1970-01-01T00:00:01Z", "1970-01-01T00:00:01Z", "1970-01-01T00:00:02Z"} {
cookie, err := a.runtimeCookie(context.Background(), i, started, target, "panel.test")
want := "dsh_auth=first"
if n == 2 {
want = "dsh_auth=second"
}
if err != nil || cookie != want {
t.Fatal("cookie exchange", err)
}
}
if exchanges != 2 || logs != 2 {
t.Fatalf("exchanges=%d logs=%d", exchanges, logs)
}
}
func TestRuntimeCookieCurrentStartupLargeMultiplexedLogs(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("token") != "current" {
t.Error("stale token exchanged")
}
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: "current"})
w.WriteHeader(303)
}))
defer upstream.Close()
target, _ := url.Parse(upstream.URL)
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("since") != "2.123456789" {
t.Error("logs not filtered by StartedAt")
io.WriteString(w, strings.Repeat("old logs\n", 200000))
return
}
for _, payload := range []string{strings.Repeat("startup noise\n", 100000), "http://localhost/?token=current\n"} {
header := [8]byte{1}
binary.BigEndian.PutUint32(header[4:], uint32(len(payload)))
w.Write(header[:])
io.WriteString(w, payload)
}
})
i := store.Instance{ID: "large-log", ContainerID: "container"}
defer runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
cookie, err := a.runtimeCookie(context.Background(), i, "1970-01-01T00:00:02.123456789Z", target, "panel.test")
if err != nil || cookie != "dsh_auth=current" {
t.Fatalf("cookie=%s err=%v", cookie, err)
}
}
func TestRuntimeCookieRejectsTruncatedLogs(t *testing.T) {
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "http://localhost/?token=partial\n"+strings.Repeat("x", 8<<20))
})
i := store.Instance{ID: "oversize", ContainerID: "container"}
defer runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
if _, err := a.runtimeCookie(context.Background(), i, "1970-01-01T00:00:01Z", nil, "panel.test"); err == nil {
t.Fatal("truncated log accepted")
}
}
func TestRuntimeCreateRejectsTrailingJSON(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
t.Error("invalid request reached Docker")
w.WriteHeader(500)
})
body := fmt.Sprintf(`{"name":"test","image_id":%q} {}`, "sha256:"+strings.Repeat("a", 64))
w := httptest.NewRecorder()
a.instances(w, runtimeRequest("POST", "/api/instances", body, sid))
if w.Code != 400 {
t.Fatal(w.Code, w.Body.String())
}
}
func TestRuntimeBlocksPendingUpgrade(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
t.Error("pending upgrade reached Docker")
w.WriteHeader(500)
})
if err := a.s.AddInstance(store.Instance{ID: "pending", UID: "owner", ContainerID: "old"}); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("CREATE TABLE upgrade_journal(instance_id TEXT,phase TEXT); INSERT INTO upgrade_journal VALUES('pending','snapshotting')"); err != nil {
t.Fatal(err)
}
for _, tc := range []struct{ method, path string }{{"POST", "/api/instances/pending/start"}, {"POST", "/api/instances/pending/restart"}, {"DELETE", "/api/instances/pending"}, {"GET", "/dsh/pending/"}} {
w := httptest.NewRecorder()
r := runtimeRequest(tc.method, tc.path, "", sid)
if strings.HasPrefix(tc.path, "/dsh/") {
a.proxy(w, r)
} else {
a.instance(w, r)
}
if w.Code != 409 {
t.Fatalf("%s: %d %s", tc.path, w.Code, w.Body.String())
}
}
}
internal/api/snapshot_download.go
-230
View File
@@ -1,230 +0,0 @@
package api
import (
"archive/tar"
"compress/gzip"
"context"
"errors"
"io"
"mime"
"net/http"
"os"
"path/filepath"
"syscall"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
var errSnapshotDownloadLimit = errors.New("snapshot archive exceeds size, entry or depth limit")
var errUnsafeSnapshotArchive = errors.New("unsafe snapshot link or unsupported file type")
type snapshotArchiveWriter struct {
ctx context.Context
w io.Writer
remaining int64
}
func (w *snapshotArchiveWriter) Write(p []byte) (int, error) {
if err := w.ctx.Err(); err != nil {
return 0, err
}
if int64(len(p)) > w.remaining {
return 0, errSnapshotDownloadLimit
}
n, err := w.w.Write(p)
w.remaining -= int64(n)
return n, err
}
func snapshotArchive(ctx context.Context, root *os.Root, out io.Writer, limit int64) error {
// Bound headers, padding and gzip expansion independently of file content.
gz := gzip.NewWriter(&snapshotArchiveWriter{ctx, out, limit + (256 << 20)})
tw := tar.NewWriter(&snapshotArchiveWriter{ctx, gz, limit + (128 << 20)})
remaining := limit
entries := 100000
var walk func(*os.Root, string, string, int) error
walk = func(parent *os.Root, name, rel string, depth int) error {
if err := ctx.Err(); err != nil {
return err
}
entries--
if entries < 0 || depth > 128 {
return errSnapshotDownloadLimit
}
st, err := parent.Lstat(name)
if err != nil {
return err
}
link := ""
if st.Mode()&os.ModeSymlink != 0 {
link, err = parent.Readlink(name)
if err != nil {
return err
}
// Archive link text only; do not read or traverse its target.
} else if !st.IsDir() && !st.Mode().IsRegular() {
return errUnsafeSnapshotArchive
}
if st.Mode().IsRegular() && st.Size() > remaining {
return errSnapshotDownloadLimit
}
if st.Mode().IsRegular() {
remaining -= st.Size()
}
h, err := tar.FileInfoHeader(st, link)
if err != nil {
return err
}
h.Name = filepath.ToSlash(rel)
if st.IsDir() {
h.Name += "/"
}
if err = tw.WriteHeader(h); err != nil {
return err
}
if st.Mode()&os.ModeSymlink != 0 {
return nil
}
if st.IsDir() {
child, err := parent.OpenRoot(name)
if err != nil {
return err
}
defer child.Close()
actual, err := child.Stat(".")
if err != nil || !os.SameFile(st, actual) {
return errors.New("snapshot directory changed")
}
f, err := child.Open(".")
if err != nil {
return err
}
defer f.Close()
for {
children, err := f.ReadDir(128)
if err != nil && err != io.EOF {
return err
}
for _, entry := range children {
if err := walk(child, entry.Name(), filepath.Join(rel, entry.Name()), depth+1); err != nil {
return err
}
}
if err == io.EOF {
return nil
}
}
}
f, err := parent.OpenFile(name, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if err != nil {
return err
}
defer f.Close()
actual, err := f.Stat()
if err != nil || !actual.Mode().IsRegular() || !os.SameFile(st, actual) || actual.Size() != st.Size() {
return errors.New("snapshot file changed")
}
if _, err = io.CopyN(tw, f, st.Size()); err != nil {
return err
}
after, err := f.Stat()
if err != nil || after.Size() != st.Size() || !after.ModTime().Equal(st.ModTime()) {
return errors.New("snapshot file changed")
}
return nil
}
if err := walk(root, ".", ".", 0); err != nil {
return err
}
if err := tw.Close(); err != nil {
return err
}
return gz.Close()
}
func (a *API) snapshotDownload(w http.ResponseWriter, r *http.Request, i store.Instance) {
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
ids := r.URL.Query()["download"]
if len(ids) != 1 || !snapshotIDPattern.MatchString(ids[0]) {
fail(w, 400, "invalid snapshot ID")
return
}
id := ids[0]
select {
case zipSlots <- struct{}{}:
defer func() { <-zipSlots }()
default:
fail(w, 429, "too many archive downloads")
return
}
ctx, cancel := context.WithTimeout(r.Context(), zipTimeout)
defer cancel()
archive, err := os.CreateTemp("", "dsh-snapshot-*.tar.gz")
if err != nil {
fail(w, 500, "temporary archive unavailable")
return
}
defer os.Remove(archive.Name())
defer archive.Close()
code := 500
err = func() error {
lock := a.instanceLocks.Acquire(i.ID)
defer lock.Release()
if err := ctx.Err(); err != nil {
return err
}
current, err := a.s.GetInstance(i.ID)
if err != nil || current.UID != i.UID {
code = 404
return errors.New("not found")
}
root, err := managementRoot(a.root, "workspaces", current.UID, "instances", current.ID, id)
if err != nil {
if errors.Is(err, os.ErrNotExist) {
code = 404
}
if errors.Is(err, errUnsafeManagementPath) {
code = 400
}
return errors.New("snapshot unavailable or unsafe path")
}
defer root.Close()
return snapshotArchive(ctx, root, archive, upgradeSnapshotLimit)
}()
if err != nil {
if errors.Is(err, errUnsafeSnapshotArchive) {
code = 400
}
if errors.Is(err, errSnapshotDownloadLimit) {
code = 413
}
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
code = 408
}
fail(w, code, err.Error())
return
}
if ctx.Err() != nil {
fail(w, 408, "snapshot download timed out or canceled")
return
}
if _, err = archive.Seek(0, io.SeekStart); err != nil {
fail(w, 500, "archive unavailable")
return
}
controller := http.NewResponseController(w)
if err = controller.SetWriteDeadline(time.Now().Add(zipTimeout)); err != nil && !errors.Is(err, http.ErrNotSupported) {
fail(w, 500, "download unavailable")
return
}
defer controller.SetWriteDeadline(time.Time{})
a.s.AddAudit(i.UID, "snapshot.download", i.ID, id)
w.Header().Set("Content-Type", "application/gzip")
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": id + ".tar.gz"}))
http.ServeContent(w, r, id+".tar.gz", time.Time{}, archive)
}
internal/api/snapshot_download_test.go
-297
View File
@@ -1,297 +0,0 @@
package api
import (
"archive/tar"
"bytes"
"compress/gzip"
"context"
"errors"
"io"
"mime"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
)
type snapshotTransferWriter struct {
*httptest.ResponseRecorder
check func()
}
func (w *snapshotTransferWriter) WriteHeader(code int) {
w.check()
w.ResponseRecorder.WriteHeader(code)
}
func TestSnapshotDownloadTransferUnlockedAndOversize(t *testing.T) {
a, i, home := configTestAPI(t, false)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
id := "failed-home-" + strings.Repeat("c", 32)
dir := filepath.Join(filepath.Dir(home), id)
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(dir, "data"), "saved")
checked := false
w := &snapshotTransferWriter{httptest.NewRecorder(), func() {
checked = true
if a.instanceLocks.held() {
t.Fatal("instance locked during transfer")
}
files, err := os.ReadDir(tmp)
if err != nil || len(files) != 1 {
t.Fatal(files, err)
}
st, err := files[0].Info()
if err != nil || st.Mode().Perm() != 0600 {
t.Fatal(st, err)
}
if err := os.RemoveAll(dir); err != nil {
t.Fatal(err)
}
}}
a.snapshotDownload(w, httptest.NewRequest(http.MethodGet, "/?download="+id, nil), i)
if !checked || w.Code != 200 {
t.Fatal(checked, w.Code)
}
gz, err := gzip.NewReader(bytes.NewReader(w.Body.Bytes()))
if err != nil {
t.Fatal(err)
}
defer gz.Close()
if _, err := io.Copy(io.Discard, gz); err != nil {
t.Fatal(err)
}
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
f, err := os.Create(filepath.Join(dir, "huge"))
if err != nil {
t.Fatal(err)
}
err = f.Truncate(upgradeSnapshotLimit + 1)
f.Close()
if err != nil {
t.Fatal(err)
}
rec := httptest.NewRecorder()
a.snapshotDownload(rec, httptest.NewRequest("GET", "/?download="+id, nil), i)
if rec.Code != 413 || !strings.Contains(rec.Body.String(), "limit") {
t.Fatal(rec.Code, rec.Body.String())
}
files, err := os.ReadDir(tmp)
if err != nil || len(files) != 0 {
t.Fatal(files, err)
}
}
func TestSnapshotDownloadContentsAndCleanup(t *testing.T) {
a, i, home := configTestAPI(t, true)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
id := "pre-restore-" + strings.Repeat("a", 32)
dir := filepath.Join(filepath.Dir(home), id)
if err := os.MkdirAll(filepath.Join(dir, "node_modules/.pnpm/pkg"), 0750); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(dir, "node_modules/.pnpm/pkg/bin"), "#!/bin/sh")
if err := os.Chmod(filepath.Join(dir, "node_modules/.pnpm/pkg/bin"), 0751); err != nil {
t.Fatal(err)
}
if err := os.Symlink(".pnpm/pkg", filepath.Join(dir, "node_modules/pkg")); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(dir, ".credentials.yaml"), "secret")
if _, err := a.s.DB.Exec("UPDATE instances SET status='error' WHERE id=?", i.ID); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
a.config(w, httptest.NewRequest("GET", "/?download="+id, nil), i, "snapshots")
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
_, params, err := mime.ParseMediaType(w.Header().Get("Content-Disposition"))
if err != nil || params["filename"] != id+".tar.gz" {
t.Fatal(params, err)
}
gz, err := gzip.NewReader(bytes.NewReader(w.Body.Bytes()))
if err != nil {
t.Fatal(err)
}
defer gz.Close()
tr := tar.NewReader(gz)
seen := map[string]bool{}
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatal(err)
}
seen[h.Name] = true
b, err := io.ReadAll(tr)
if err != nil {
t.Fatal(err)
}
switch h.Name {
case "node_modules/.pnpm/pkg/bin":
if string(b) != "#!/bin/sh" || h.Mode != 0751 {
t.Fatal(h, string(b))
}
case "node_modules/pkg":
if h.Typeflag != tar.TypeSymlink || h.Linkname != ".pnpm/pkg" {
t.Fatal(h)
}
case ".credentials.yaml":
if string(b) != "secret" {
t.Fatal(string(b))
}
}
}
if len(seen) != 7 || !seen["node_modules/pkg"] || seen["node_modules/pkg/bin"] {
t.Fatal(seen)
}
files, err := os.ReadDir(tmp)
if err != nil || len(files) != 0 {
t.Fatal(files, err)
}
r := httptest.NewRequest("GET", "/?download="+id, nil)
r.Header.Set("Range", "bytes=0-9")
w = httptest.NewRecorder()
a.snapshotDownload(w, r, i)
if w.Code != 206 || w.Body.Len() != 10 {
t.Fatal(w.Code, w.Body.Len())
}
}
func TestSnapshotDownloadRejectsUnsafeAndForeign(t *testing.T) {
a, i, home := configTestAPI(t, false)
id := "pre-restore-" + strings.Repeat("b", 32)
dir := filepath.Join(filepath.Dir(home), id)
if err := os.Mkdir(dir, 0700); err != nil {
t.Fatal(err)
}
for _, query := range []string{"", "../home", "/home", id + "/../home", "pre-restore-ABC"} {
w := httptest.NewRecorder()
a.snapshotDownload(w, httptest.NewRequest("GET", "/?download="+url.QueryEscape(query), nil), i)
if w.Code != 400 {
t.Fatal(query, w.Code)
}
}
foreign := i
foreign.UID = "someone-else"
w := httptest.NewRecorder()
a.snapshotDownload(w, httptest.NewRequest("GET", "/?download="+id, nil), foreign)
if w.Code != 404 {
t.Fatal(w.Code)
}
for _, target := range []string{"../home", "/workspace/example", "../../elsewhere", "missing", "link"} {
if err := os.Symlink(target, filepath.Join(dir, "link")); err != nil {
t.Fatal(err)
}
w = httptest.NewRecorder()
a.snapshotDownload(w, httptest.NewRequest("GET", "/?download="+id, nil), i)
if w.Code != 200 || w.Header().Get("Content-Disposition") == "" {
t.Fatal(target, w.Code)
}
gz, err := gzip.NewReader(w.Body)
if err != nil {
t.Fatal(err)
}
tr := tar.NewReader(gz)
found := false
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
t.Fatal(err)
}
if h.Name == "link" {
found = h.Typeflag == tar.TypeSymlink && h.Linkname == target && h.Size == 0
} else if h.Name != "./" {
t.Fatalf("unexpected archived entry: %s", h.Name)
}
}
gz.Close()
if !found {
t.Fatal("link text was not preserved")
}
if err := os.Remove(filepath.Join(dir, "link")); err != nil {
t.Fatal(err)
}
}
if err := syscall.Mkfifo(filepath.Join(dir, "fifo"), 0600); err != nil {
t.Fatal(err)
}
w = httptest.NewRecorder()
a.snapshotDownload(w, httptest.NewRequest("GET", "/?download="+id, nil), i)
if w.Code != 400 {
t.Fatal(w.Code)
}
if err := os.Remove(filepath.Join(dir, "fifo")); err != nil {
t.Fatal(err)
}
if err := os.Remove(dir); err != nil {
t.Fatal(err)
}
if err := os.Symlink(home, dir); err != nil {
t.Fatal(err)
}
w = httptest.NewRecorder()
a.snapshotDownload(w, httptest.NewRequest("GET", "/?download="+id, nil), i)
if w.Code != 400 {
t.Fatal(w.Code)
}
}
func TestSnapshotArchiveLimitsAndCancellation(t *testing.T) {
dir := t.TempDir()
configWrite(t, filepath.Join(dir, "file"), strings.Repeat("x", 2048))
root, err := os.OpenRoot(dir)
if err != nil {
t.Fatal(err)
}
defer root.Close()
if err := snapshotArchive(context.Background(), root, io.Discard, 1024); !errors.Is(err, errSnapshotDownloadLimit) {
t.Fatal(err)
}
if err := snapshotArchive(context.Background(), root, io.Discard, 2048); err != nil {
t.Fatal("exact content limit should allow archive overhead", err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if err := snapshotArchive(ctx, root, io.Discard, upgradeSnapshotLimit); !errors.Is(err, context.Canceled) {
t.Fatal(err)
}
a, i, _ := configTestAPI(t, false)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
r := httptest.NewRequest("GET", "/?download=pre-restore-"+strings.Repeat("a", 32), nil).WithContext(ctx)
w := httptest.NewRecorder()
a.snapshotDownload(w, r, i)
if w.Code != 408 {
t.Fatal(w.Code)
}
files, err := os.ReadDir(tmp)
if err != nil || len(files) != 0 {
t.Fatal(files, err)
}
if len(zipSlots) != 0 || a.instanceLocks.held() {
t.Fatal("resource leak")
}
t.Setenv("TMPDIR", filepath.Join(tmp, "missing"))
w = httptest.NewRecorder()
a.snapshotDownload(w, httptest.NewRequest("GET", r.URL.String(), nil), i)
if w.Code != 500 {
t.Fatal(w.Code)
}
}
internal/api/snapshot_restore.go
-394
View File
@@ -1,394 +0,0 @@
package api
import (
"context"
"database/sql"
"encoding/json"
"errors"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"time"
)
const snapshotUnknownReason = "snapshot source image/version unknown; explicit acknowledgement required"
func snapshotRuntimeLayout(config, host map[string]json.RawMessage, mounts []struct{ Type, Source, Destination string }, home string) bool {
workspace := 0
for _, mount := range mounts {
if mount.Type == "tmpfs" && mount.Destination == "/tmp" {
continue
}
if mount.Type != "bind" || mount.Source != home || mount.Destination != "/workspace" {
return false
}
workspace++
}
if workspace != 1 {
return false
}
var env []string
if json.Unmarshal(config["Env"], &env) != nil {
return false
}
values := map[string]string{}
for _, entry := range env {
key, value, _ := strings.Cut(entry, "=")
values[key] = value
}
for key, want := range map[string]string{"DSH_HOME": "/workspace", "HOME": "/workspace", "XDG_DATA_HOME": "/workspace/.local/share", "XDG_CONFIG_HOME": "/workspace/.config"} {
if values[key] != want {
return false
}
}
for index, raw := range []json.RawMessage{host["Tmpfs"], config["Volumes"]} {
if len(raw) == 0 {
continue
}
var mounts map[string]json.RawMessage
if json.Unmarshal(raw, &mounts) != nil {
return false
}
if index == 0 {
delete(mounts, "/tmp")
}
if len(mounts) != 0 {
return false
}
}
return true
}
func (a *API) recordSnapshot(i store.Instance, id, version, image string) error {
_, err := a.s.DB.Exec("CREATE TABLE IF NOT EXISTS snapshot_metadata (instance_id TEXT NOT NULL, id TEXT NOT NULL, source_version TEXT NOT NULL, source_image TEXT NOT NULL, PRIMARY KEY(instance_id,id))")
if err != nil {
return err
}
_, err = a.s.DB.Exec("INSERT INTO snapshot_metadata(instance_id,id,source_version,source_image) VALUES(?,?,?,?)", i.ID, id, version, image)
return err
}
func (a *API) snapshotCompatibility(i store.Instance, id string) (version, image, reason string) {
var exists int
if err := a.s.DB.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='snapshot_copy_state'").Scan(&exists); err != nil {
return "", "", "snapshot completion metadata unavailable"
}
if exists != 0 {
var ready int
err := a.s.DB.QueryRow("SELECT ready FROM snapshot_copy_state WHERE instance_id=? AND id=?", i.ID, id).Scan(&ready)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return "", "", "snapshot completion metadata unavailable"
}
if err == nil && ready != 1 {
return "", "", "snapshot copy incomplete"
}
}
if err := a.s.DB.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='snapshot_metadata'").Scan(&exists); err != nil {
return "", "", "snapshot metadata unavailable"
}
if exists != 0 {
err := a.s.DB.QueryRow("SELECT source_version,source_image FROM snapshot_metadata WHERE instance_id=? AND id=?", i.ID, id).Scan(&version, &image)
if err != nil && !errors.Is(err, sql.ErrNoRows) {
return "", "", "snapshot metadata unavailable"
}
}
if (version != "" && version != i.Version) || (image != "" && image != i.ImageID) {
return version, image, "snapshot source image/version is incompatible"
}
if version == "" || image == "" {
return version, image, snapshotUnknownReason
}
return version, image, ""
}
func (a *API) snapshotRestore(w http.ResponseWriter, r *http.Request, i store.Instance) {
controller := http.NewResponseController(w)
w, flush := deferJSONResponse(w)
defer flush()
select {
case configBodySlots <- struct{}{}:
defer func() { r.Body = http.NoBody; <-configBodySlots }()
default:
fail(w, 503, "configuration uploads busy; retry later")
return
}
if !configReadBody(w, r, controller, 4096) {
return
}
var in struct {
ID string
Password string
Confirm bool
AllowUnknown bool `json:"allow_unknown_version"`
}
dec := json.NewDecoder(r.Body)
if dec.Decode(&in) != nil || dec.Decode(new(any)) != io.EOF || !snapshotIDPattern.MatchString(in.ID) || !in.Confirm {
fail(w, 400, "valid snapshot ID and explicit confirmation required")
return
}
lock := a.instanceLocks.Acquire(i.ID)
defer lock.Release()
a.readersMu.Lock()
readers := a.workspaceReaders[i.ID]
a.readersMu.Unlock()
if readers > 0 {
fail(w, 409, "instance files busy; retry later")
return
}
user := a.userLocks.Acquire(i.UID)
defer user.Release()
current, err := a.s.GetInstance(i.ID)
if err != nil || current.UID != i.UID {
fail(w, 404, "not found")
return
}
i = current
if !a.reauthenticate(w, i.UID, in.Password) {
return
}
if reason := a.backupProtection(i); reason != "" {
fail(w, 409, reason)
return
}
admissionMu.RLock()
defer admissionMu.RUnlock()
buildQueue.Lock()
err = a.ensureBuilds()
buildQueue.Unlock()
if err != nil {
fail(w, 500, "database error")
return
}
var blocked int
if err = a.s.DB.QueryRowContext(r.Context(), "SELECT count(*) FROM image_admissions WHERE image_id=? AND state='blocked'", i.ImageID).Scan(&blocked); err != nil {
fail(w, 500, "database error")
return
}
if blocked > 0 {
fail(w, 409, "image is blocked; restore denied")
return
}
version, image, reason := a.snapshotCompatibility(i, in.ID)
if reason != "" && !(reason == snapshotUnknownReason && in.AllowUnknown) {
fail(w, 409, reason)
return
}
root, err := managementRoot(a.root, "workspaces", i.UID, "instances", i.ID)
if errors.Is(err, os.ErrNotExist) {
fail(w, 404, "home or snapshot not found")
return
}
if err != nil {
managementPathError(w, err)
return
}
defer root.Close()
for _, name := range []string{"home", in.ID} {
st, e := root.Lstat(name)
if errors.Is(e, os.ErrNotExist) {
fail(w, 404, "home or snapshot not found")
return
}
if e != nil {
managementPathError(w, e)
return
}
if !st.IsDir() {
fail(w, 400, "unsafe home or snapshot")
return
}
}
f, err := root.Open(".")
if err != nil {
fail(w, 500, "snapshot storage unavailable")
return
}
entries, err := f.ReadDir(-1)
f.Close()
count := 0
for _, entry := range entries {
if strings.HasPrefix(entry.Name(), "pre-restore-") || strings.HasPrefix(entry.Name(), "failed-restore-") {
count++
}
}
if err != nil {
fail(w, 500, "snapshot listing failed")
return
}
if count >= 3 {
fail(w, 409, "restore snapshot limit reached; cleanup required")
return
}
size, err := quotaTreeBytes(root, in.ID)
var pathError *os.PathError
if errors.As(err, &pathError) {
managementPathError(w, err)
return
}
if err != nil || a.quotaGrowth(i.UID, size) != nil {
fail(w, 413, "quota exceeded or snapshot size unavailable")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Minute)
defer cancel()
var container struct {
Image string
Config, HostConfig map[string]json.RawMessage
Mounts []struct{ Type, Source, Destination string }
State struct{ Running, Paused, Restarting bool }
}
err = a.docker.Do(ctx, "GET", "/containers/"+url.PathEscape(i.ContainerID)+"/json", nil, &container)
if err != nil {
fail(w, 502, "container inspection failed")
return
}
if container.State.Paused || container.State.Restarting {
fail(w, 409, "unsupported container state")
return
}
var runtimeImage struct {
ID string `json:"Id"`
Config struct{ Labels map[string]string }
}
hostRoot := os.Getenv("HOST_DATA_ROOT")
if hostRoot == "" {
hostRoot = a.root
}
homeSource, homeErr := toHostPath(a.root, hostRoot, filepath.Join(root.Name(), "home"))
if homeErr != nil || !snapshotRuntimeLayout(container.Config, container.HostConfig, container.Mounts, homeSource) {
fail(w, 409, "incompatible effective runtime workspace layout")
return
}
if i.ImageID == "" || container.Image != i.ImageID || container.Config == nil || container.HostConfig == nil || homeErr != nil || a.docker.Do(ctx, "GET", "/images/"+url.PathEscape(i.ImageID)+"/json", nil, &runtimeImage) != nil || runtimeImage.ID != i.ImageID || runtimeImage.Config.Labels["dsh-mgr.version"] != i.Version || upgradeWorkspace(container.Config, container.HostConfig, runtimeImage.Config.Labels["dsh-mgr.recipe"], homeSource) != nil {
fail(w, 409, "incompatible runtime image or workspace layout")
return
}
wasRunning := container.State.Running
op := store.ID()
staging := ".snapshot-restore-" + op
previous := "pre-restore-" + op
failed := "failed-restore-" + op
if err = upgradeCopy(ctx, root, in.ID, staging, size); err != nil {
if root.RemoveAll(staging) != nil {
a.s.SetInstance(i.ID, "error")
fail(w, 500, "staging cleanup requires administrator recovery")
return
}
if errors.As(err, &pathError) {
managementPathError(w, err)
return
}
fail(w, 400, "unsafe or oversized snapshot")
return
}
defer func() {
if root.RemoveAll(staging) != nil {
a.s.SetInstance(i.ID, "error")
}
}()
if err = a.recordSnapshot(i, previous, i.Version, i.ImageID); err != nil {
fail(w, 500, "snapshot metadata unavailable")
return
}
if err = a.recordSnapshot(i, failed, version, image); err != nil {
fail(w, 500, "snapshot metadata unavailable")
return
}
if _, err = a.s.DB.Exec("UPDATE instances SET status='error' WHERE id=?", i.ID); err != nil {
fail(w, 500, "cannot record restore state")
return
}
if _, err = a.s.DB.Exec("INSERT INTO audit(uid,action,target,detail,at) VALUES(?,?,?,?,?)", i.UID, "snapshot.restore.started", i.ID, in.ID+" "+previous+" "+failed, time.Now().Unix()); err != nil {
fail(w, 500, "restore audit unavailable; recovery required")
return
}
if err = a.configStop(ctx, i.ContainerID); err != nil {
fail(w, 502, "runtime stop failed; recovery required")
return
}
moved, installed := false, false
var owner store.User
owner.ID = i.UID
err = a.s.DB.QueryRow("SELECT quota FROM users WHERE id=?", i.UID).Scan(&owner.Quota)
if err == nil {
var used int64
used, err = a.userUsage(owner)
if err == nil && used > owner.Quota {
err = errors.New("quota exceeded after runtime stop")
}
}
if err == nil {
err = root.Rename("home", previous)
}
if err == nil {
moved = true
err = upgradeSync(root)
}
if err == nil {
err = root.Rename(staging, "home")
installed = err == nil
}
if err == nil {
err = upgradeSync(root)
}
startReady := func(c context.Context) error {
runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
if e := a.docker.Do(c, "POST", "/containers/"+url.PathEscape(i.ContainerID)+"/start", nil, nil); e != nil {
return e
}
return a.upgradeReady(c, i.ContainerID, i.ID, a.runtimeHost(r))
}
if err == nil {
err = startReady(ctx)
}
if err == nil && !wasRunning {
err = a.configStop(ctx, i.ContainerID)
}
rejected := err != nil
if rejected {
rollback, done := context.WithTimeout(context.Background(), 90*time.Second)
defer done()
if a.configStop(rollback, i.ContainerID) != nil {
fail(w, 500, "restore rollback requires runtime stop; recovery required")
return
}
if installed {
if root.Rename("home", failed) != nil || upgradeSync(root) != nil {
fail(w, 500, "restore rollback requires administrator recovery")
return
}
}
if moved {
if root.Rename(previous, "home") != nil || upgradeSync(root) != nil {
fail(w, 500, "restore rollback requires administrator recovery")
return
}
}
if wasRunning {
if startReady(rollback) != nil {
fail(w, 500, "restore rollback readiness failed; recovery required")
return
}
}
}
status := "exited"
if wasRunning {
status = "running"
}
if _, err = a.s.DB.Exec("UPDATE instances SET status=? WHERE id=?", status, i.ID); err != nil {
fail(w, 500, "restore state requires administrator recovery")
return
}
runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
if rejected {
a.s.AddAudit(i.UID, "snapshot.restore.rolled_back", i.ID, in.ID)
fail(w, 400, "restore rejected; previous home restored")
return
}
a.s.AddAudit(i.UID, "snapshot.restore.completed", i.ID, in.ID)
writeJSON(w, map[string]any{"ok": true, "snapshot_id": previous})
}
internal/api/snapshot_restore_test.go
-267
View File
@@ -1,267 +0,0 @@
package api
import (
"encoding/json"
"fmt"
"git.misaka.ren/m1saka/dsh-web/internal/docker"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
)
func snapshotRestoreFixture(t *testing.T, running bool, failure string) (*API, store.Instance, string, string) {
t.Helper()
a, i, home := configTestAPI(t, running, failure)
i.ImageID = "sha256:" + strings.Repeat("a", 64)
i.Version = "1.0"
if _, err := a.s.DB.Exec("UPDATE instances SET image_id=?,version=? WHERE id=?", i.ImageID, i.Version, i.ID); err != nil {
t.Fatal(err)
}
original := a.docker
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if strings.Contains(r.URL.Path, "/images/") {
writeJSON(w, map[string]any{"Id": i.ImageID, "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.recipe": "7", "dsh-mgr.version": i.Version}}})
return
}
if strings.HasSuffix(r.URL.Path, "/json") {
var data map[string]any
if err := original.Do(r.Context(), "GET", r.URL.RequestURI(), nil, &data); err != nil {
t.Error(err)
w.WriteHeader(500)
return
}
data["Image"] = i.ImageID
data["Config"] = map[string]any{"Env": []string{"DSH_HOME=/workspace", "HOME=/workspace", "XDG_DATA_HOME=/workspace/.local/share", "XDG_CONFIG_HOME=/workspace/.config"}}
data["Mounts"] = []map[string]string{{"Type": "bind", "Source": home, "Destination": "/workspace"}}
data["HostConfig"] = map[string]any{"Mounts": []map[string]string{{"Type": "bind", "Source": home, "Target": "/workspace"}}}
data["HostConfig"].(map[string]any)["Tmpfs"] = map[string]string{"/tmp": "rw,noexec,nosuid,size=64m"}
writeJSON(w, data)
return
}
resp, err := original.Stream(r.Context(), r.Method, r.URL.RequestURI(), nil, "")
if err != nil {
t.Error(err)
w.WriteHeader(500)
return
}
defer resp.Body.Close()
w.WriteHeader(resp.StatusCode)
io.Copy(w, resp.Body)
}))
t.Cleanup(proxy.Close)
a.docker = docker.New(proxy.URL)
id := "upgrade-" + strings.Repeat("b", 32) + "-snapshot"
if err := os.Mkdir(filepath.Join(filepath.Dir(home), id), 0700); err != nil {
t.Fatal(err)
}
configWrite(t, filepath.Join(filepath.Dir(home), id, "data"), "snapshot")
configWrite(t, filepath.Join(home, "data"), "current")
return a, i, home, id
}
func snapshotRestoreBody(id string, unknown bool) string {
return fmt.Sprintf(`{"id":%q,"password":"original-password","confirm":true,"allow_unknown_version":%t}`, id, unknown)
}
func TestSnapshotRuntimeLayout(t *testing.T) {
for _, test := range []string{"valid", "env", "tmpfs", "volume", "effective"} {
t.Run(test, func(t *testing.T) {
env := []string{"DSH_HOME=/workspace", "HOME=/workspace", "XDG_DATA_HOME=/workspace/.local/share", "XDG_CONFIG_HOME=/workspace/.config"}
if test == "env" {
env = append(env, "DSH_HOME=/old-shared")
}
raw, _ := json.Marshal(env)
config := map[string]json.RawMessage{"Env": raw}
host := map[string]json.RawMessage{"Tmpfs": json.RawMessage(`{"/tmp":"rw"}`)}
mounts := []struct{ Type, Source, Destination string }{{"bind", "/home", "/workspace"}}
mounts = append(mounts, struct{ Type, Source, Destination string }{"tmpfs", "", "/tmp"})
if test == "tmpfs" {
host["Tmpfs"] = json.RawMessage(`{"/workspace/data":"rw"}`)
}
if test == "volume" {
config["Volumes"] = json.RawMessage(`{"/workspace/data":{}}`)
}
if test == "effective" {
mounts = append(mounts, struct{ Type, Source, Destination string }{"volume", "/other", "/workspace/data"})
}
if snapshotRuntimeLayout(config, host, mounts, "/home") != (test == "valid") {
t.Fatal(test)
}
})
}
}
func TestSnapshotListProvenance(t *testing.T) {
a, i, _, id := snapshotRestoreFixture(t, false, "")
for _, known := range []bool{false, true} {
if known {
if err := a.recordSnapshot(i, id, "old-version", i.ImageID); err != nil {
t.Fatal(err)
}
}
w := configCall(a, i, "GET", "snapshots", "")
var list []snapshotInfo
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &list) != nil || len(list) != 1 {
t.Fatal(w.Code, w.Body.String())
}
if list[0].Restorable == known || list[0].RestoreReason == "" {
t.Fatal(list)
}
if !known && (list[0].SourceVersion != "" || list[0].SourceImage != "") {
t.Fatal("inferred provenance", list)
}
}
}
func TestSnapshotFullRestore(t *testing.T) {
for _, running := range []bool{false, true} {
t.Run(fmt.Sprint(running), func(t *testing.T) {
a, i, home, id := snapshotRestoreFixture(t, running, "")
links := map[string]string{"absolute": "/workspace/example", "dangling": "missing", "loop": "loop", "relative": "data"}
for name, target := range links {
if err := os.Symlink(target, filepath.Join(filepath.Dir(home), id, name)); err != nil {
t.Fatal(err)
}
}
if err := a.recordSnapshot(i, id, i.Version, i.ImageID); err != nil {
t.Fatal(err)
}
w := configCall(a, i, "POST", "snapshots", snapshotRestoreBody(id, false))
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
var result struct {
SnapshotID string `json:"snapshot_id"`
}
json.Unmarshal(w.Body.Bytes(), &result)
for path, want := range map[string]string{filepath.Join(home, "data"): "snapshot", filepath.Join(filepath.Dir(home), id, "data"): "snapshot", filepath.Join(filepath.Dir(home), result.SnapshotID, "data"): "current"} {
got, err := os.ReadFile(path)
if err != nil || string(got) != want {
t.Fatal(path, string(got), err)
}
}
current, _ := a.s.GetInstance(i.ID)
for name, target := range links {
got, err := os.Readlink(filepath.Join(home, name))
if err != nil || got != target {
t.Fatalf("restored link %s: %q %v", name, got, err)
}
}
want := "exited"
if running {
want = "running"
}
if current.Status != want {
t.Fatal(current.Status)
}
})
}
}
func TestSnapshotRestoreRollback(t *testing.T) {
a, i, home, id := snapshotRestoreFixture(t, false, "reconcile")
w := configCall(a, i, "POST", "snapshots", snapshotRestoreBody(id, true))
if w.Code != 400 || !strings.Contains(w.Body.String(), "previous home restored") {
t.Fatal(w.Code, w.Body.String())
}
for path, want := range map[string]string{filepath.Join(home, "data"): "current", filepath.Join(filepath.Dir(home), id, "data"): "snapshot"} {
got, err := os.ReadFile(path)
if err != nil || string(got) != want {
t.Fatal(path, string(got), err)
}
}
matches, _ := filepath.Glob(filepath.Join(filepath.Dir(home), "failed-restore-*", "data"))
if len(matches) != 1 {
t.Fatal(matches)
}
}
func TestIncompleteSnapshotNotRestorable(t *testing.T) {
a, i, _, id := snapshotRestoreFixture(t, false, "")
if _, err := a.s.DB.Exec("CREATE TABLE snapshot_copy_state (instance_id TEXT, id TEXT, ready INTEGER)"); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("INSERT INTO snapshot_copy_state VALUES(?,?,0)", i.ID, id); err != nil {
t.Fatal(err)
}
w := configCall(a, i, "GET", "snapshots", "")
var list []snapshotInfo
if w.Code != 200 || json.Unmarshal(w.Body.Bytes(), &list) != nil || len(list) != 1 || list[0].Restorable || list[0].RestoreReason != "snapshot copy incomplete" {
t.Fatal(w.Code, w.Body.String())
}
w = configCall(a, i, "POST", "snapshots", snapshotRestoreBody(id, true))
if w.Code != 409 || !strings.Contains(w.Body.String(), "snapshot copy incomplete") {
t.Fatal(w.Code, w.Body.String())
}
if err := a.recordSnapshot(i, id, i.Version, i.ImageID); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("UPDATE snapshot_copy_state SET ready=1"); err != nil {
t.Fatal(err)
}
if _, _, reason := a.snapshotCompatibility(i, id); reason != "" {
t.Fatal(reason)
}
if _, err := a.s.DB.Exec("ALTER TABLE snapshot_copy_state RENAME COLUMN ready TO unavailable"); err != nil {
t.Fatal(err)
}
if _, _, reason := a.snapshotCompatibility(i, id); reason != "snapshot completion metadata unavailable" {
t.Fatal(reason)
}
}
func TestSnapshotRestoreGuards(t *testing.T) {
for _, test := range []string{"unknown", "mismatch", "password", "confirm", "reader", "quota", "error", "unsafe", "foreign", "oversized"} {
t.Run(test, func(t *testing.T) {
a, i, home, id := snapshotRestoreFixture(t, false, "")
body := snapshotRestoreBody(id, true)
want := 409
switch test {
case "unknown":
body = snapshotRestoreBody(id, false)
case "mismatch":
if err := a.recordSnapshot(i, id, "different", i.ImageID); err != nil {
t.Fatal(err)
}
case "password":
body = strings.ReplaceAll(body, "original-password", "wrong")
want = 403
case "confirm":
body = strings.ReplaceAll(body, `"confirm":true`, `"confirm":false`)
want = 400
case "reader":
a.workspaceReaders = map[string]int{i.ID: 1}
case "quota":
a.s.DB.Exec("UPDATE users SET quota=1 WHERE id=?", i.UID)
want = 413
case "error":
a.s.SetInstance(i.ID, "error")
case "unsafe":
if err := syscall.Mkfifo(filepath.Join(filepath.Dir(home), id, "unsafe"), 0600); err != nil {
t.Fatal(err)
}
want = 400
case "foreign":
i.UID = "someone-else"
want = 404
case "oversized":
body = strings.Repeat("x", 4097)
want = 413
}
w := configCall(a, i, "POST", "snapshots", body)
if w.Code != want {
t.Fatal(w.Code, w.Body.String())
}
got, err := os.ReadFile(filepath.Join(home, "data"))
if err != nil || string(got) != "current" {
t.Fatal(string(got), err)
}
})
}
}
internal/api/upgrade.go
-634
View File
@@ -1,634 +0,0 @@
package api
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/url"
"os"
"path/filepath"
"strings"
"syscall"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/backup"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
const upgradeSnapshotLimit int64 = 1 << 30
func upgradeWorkspace(config, host map[string]json.RawMessage, recipe, home string) error {
var mounts []map[string]json.RawMessage
if err := json.Unmarshal(host["Mounts"], &mounts); err != nil {
return errors.New("unsupported runtime mounts")
}
get := func(m map[string]json.RawMessage, key string) string {
var value string
_ = json.Unmarshal(m[key], &value)
return value
}
if !supportedLayout(recipe) {
return errors.New("unsupported template layout; template 7 required")
}
if len(mounts) != 1 || get(mounts[0], "Type") != "bind" || get(mounts[0], "Source") != home || get(mounts[0], "Target") != "/workspace" {
return errors.New("independent instance home at /workspace required")
}
for _, key := range []string{"Binds", "VolumesFrom"} {
var values []string
if raw := host[key]; len(raw) != 0 {
if json.Unmarshal(raw, &values) != nil || len(values) != 0 {
return errors.New("unsupported additional mounts")
}
}
}
var env []string
if raw := config["Env"]; len(raw) != 0 {
if err := json.Unmarshal(raw, &env); err != nil {
return err
}
}
kept := make([]string, 0, len(env)+4)
for _, entry := range env {
key, _, _ := strings.Cut(entry, "=")
switch key {
case "DSH_HOME", "HOME", "XDG_DATA_HOME", "XDG_CONFIG_HOME":
default:
kept = append(kept, entry)
}
}
kept = append(kept, "DSH_HOME=/workspace", "HOME=/workspace", "XDG_DATA_HOME=/workspace/.local/share", "XDG_CONFIG_HOME=/workspace/.config")
config["Env"], _ = json.Marshal(kept)
return nil
}
func upgradeRoot(root string, i store.Instance) (*os.Root, error) {
r, err := os.OpenRoot(root)
if err != nil {
return nil, err
}
for _, part := range []string{"workspaces", i.UID, "instances", i.ID} {
if part == "" || part == "." || part == ".." || strings.ContainsAny(part, "/\\") {
r.Close()
return nil, errors.New("invalid storage path")
}
st, e := r.Lstat(part)
if e != nil || !st.IsDir() {
r.Close()
return nil, errors.New("unsafe storage path")
}
next, e := r.OpenRoot(part)
r.Close()
if e != nil {
return nil, e
}
r = next
}
return r, nil
}
func upgradeSync(r *os.Root) error {
f, err := r.Open(".")
if err != nil {
return err
}
defer f.Close()
return f.Sync()
}
func upgradeCopy(ctx context.Context, r *os.Root, source, target string, limit int64) error {
remaining, entries := limit, 100000
var copyTree func(string, string, int) error
copyTree = func(src, dst string, depth int) (copyErr error) {
defer func() {
if copyErr != nil {
if pathErr, ok := copyErr.(*os.PathError); ok {
copyErr = fmt.Errorf("snapshot copy: %w", &os.PathError{Op: pathErr.Op, Path: src, Err: pathErr.Err})
return
}
copyErr = fmt.Errorf("%s: %w", src, copyErr)
}
}()
if err := ctx.Err(); err != nil {
return err
}
entries--
if entries < 0 || depth > 128 {
return errors.New("snapshot entry limit")
}
st, err := r.Lstat(src)
if err != nil {
return err
}
if st.IsDir() {
if err = r.Mkdir(dst, 0700); err != nil {
return err
}
f, err := r.OpenFile(src, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_DIRECTORY, 0)
if err != nil {
return err
}
defer f.Close()
for {
children, err := f.ReadDir(128)
if err != nil && err != io.EOF {
return err
}
for _, child := range children {
if err := copyTree(filepath.Join(src, child.Name()), filepath.Join(dst, child.Name()), depth+1); err != nil {
return err
}
}
if err == io.EOF {
break
}
}
if err = r.Chown(dst, 1000, 1000); err != nil {
return err
}
if err = r.Chmod(dst, st.Mode().Perm()); err != nil {
return err
}
d, err := r.Open(dst)
if err != nil {
return err
}
defer d.Close()
return d.Sync()
}
if st.Mode()&os.ModeSymlink != 0 && depth > 0 {
link, err := r.Readlink(src)
if err != nil {
return err
}
// Preserve the link itself, including dangling and container-absolute
// targets. Never resolve it against the panel's filesystem.
if err = r.Symlink(link, dst); err != nil {
return err
}
return r.Lchown(dst, 1000, 1000)
}
if !st.Mode().IsRegular() || st.Size() > remaining {
return errors.New("unsafe or oversized snapshot")
}
in, err := r.OpenFile(src, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if err != nil {
return err
}
defer in.Close()
actual, err := in.Stat()
if err != nil || !actual.Mode().IsRegular() || !os.SameFile(st, actual) {
return errors.New("snapshot source changed")
}
out, err := r.OpenFile(dst, os.O_WRONLY|os.O_CREATE|os.O_EXCL|syscall.O_NOFOLLOW, 0600)
if err != nil {
return err
}
defer out.Close()
n, err := io.Copy(out, io.LimitReader(in, remaining+1))
if err != nil {
return err
}
remaining -= n
if remaining < 0 {
return errors.New("snapshot size limit")
}
if err = out.Chown(1000, 1000); err != nil {
return err
}
if err = out.Chmod(st.Mode().Perm()); err != nil {
return err
}
return out.Sync()
}
if err := copyTree(source, target, 0); err != nil {
return err
}
return upgradeSync(r)
}
// Call while holding the instance lock before any operation that can start or mutate a runtime.
func (a *API) upgradeRecoveryRequired(i store.Instance) bool {
var exists int
if err := a.s.DB.QueryRow("SELECT count(*) FROM sqlite_master WHERE type='table' AND name='upgrade_journal'").Scan(&exists); err != nil {
return true
}
if exists == 0 {
return false
}
var pending int
if err := a.s.DB.QueryRow("SELECT count(*) FROM upgrade_journal WHERE instance_id=? AND phase NOT IN ('complete','rolled_back')", i.ID).Scan(&pending); err != nil {
return true
}
if pending != 0 {
a.s.DB.Exec("UPDATE instances SET status='error' WHERE id=?", i.ID)
}
return pending != 0
}
func (a *API) upgrade(w http.ResponseWriter, r *http.Request, i store.Instance) {
if r.Method != http.MethodPost {
fail(w, 405, "method not allowed")
return
}
var in struct {
ImageID string `json:"image_id"`
Password string `json:"password"`
}
dec := json.NewDecoder(http.MaxBytesReader(w, r.Body, 64<<10))
dec.DisallowUnknownFields()
if dec.Decode(&in) != nil || !runtimeImageID.MatchString(in.ImageID) || dec.Decode(new(any)) != io.EOF {
fail(w, 400, "invalid image_id")
return
}
inst := a.instanceLocks.Acquire(i.ID)
defer inst.Release()
a.readersMu.Lock()
readers := a.workspaceReaders[i.ID]
a.readersMu.Unlock()
if readers > 0 {
fail(w, 409, "instance files busy; retry later")
return
}
user := a.userLocks.Acquire(i.UID)
defer user.Release()
admissionMu.RLock()
defer admissionMu.RUnlock()
current, err := a.s.GetInstance(i.ID)
if err != nil || current.UID != i.UID {
fail(w, 404, "not found")
return
}
i = current
if len(in.Password) == 0 || len(in.Password) > 72 {
fail(w, 400, "login password required for pre-upgrade backup")
return
}
if !a.reauthenticate(w, i.UID, in.Password) {
return
}
if a.upgradeRecoveryRequired(i) || i.Status == "error" {
fail(w, 409, "instance requires recovery")
return
}
ctx, cancel := context.WithTimeout(r.Context(), 3*time.Minute)
defer cancel()
var old struct {
Config map[string]json.RawMessage
HostConfig map[string]json.RawMessage
State struct {
Status string
Running, Paused, Restarting bool
}
NetworkSettings struct{ Networks map[string]json.RawMessage }
}
if i.ContainerID == "" || a.docker.Do(ctx, "GET", "/containers/"+url.PathEscape(i.ContainerID)+"/json", nil, &old) != nil {
fail(w, 502, "container inspection failed")
return
}
if old.State.Paused || old.State.Restarting || (old.State.Status != "running" && old.State.Status != "exited" && old.State.Status != "created") {
fail(w, 409, "unsupported container state")
return
}
if old.Config == nil || old.HostConfig == nil {
fail(w, 502, "container configuration unavailable")
return
}
verified, err := a.verifiedImages(ctx)
if err != nil {
fail(w, 500, "database error")
return
}
var image struct {
ID string `json:"Id"`
Config struct{ Labels map[string]string }
}
if a.docker.Do(ctx, "GET", "/images/"+url.PathEscape(in.ImageID)+"/json", nil, &image) != nil || image.ID != in.ImageID || image.Config.Labels["dsh-mgr.role"] != "runtime" || !supportedLayout(image.Config.Labels["dsh-mgr.recipe"]) || verified[image.ID] == "" || verified[image.ID] != image.Config.Labels["dsh-mgr.version"] {
fail(w, 400, "local runtime image required")
return
}
root, err := upgradeRoot(a.root, i)
if err != nil {
fail(w, 500, "storage unavailable")
return
}
defer root.Close()
hostRoot := os.Getenv("HOST_DATA_ROOT")
if hostRoot == "" {
hostRoot = a.root
}
homeSource, homeErr := toHostPath(a.root, hostRoot, filepath.Join(root.Name(), "home"))
if homeErr != nil {
fail(w, 500, "host storage unavailable")
return
}
if err = upgradeWorkspace(old.Config, old.HostConfig, image.Config.Labels["dsh-mgr.recipe"], homeSource); err != nil {
fail(w, 409, "incompatible runtime workspace layout")
return
}
if _, err = a.quotaSnapshot(i.UID, root, 0); err != nil {
fail(w, 413, "quota exceeded or snapshot size unavailable")
return
}
op := store.ID()
snapshot, failedHome := "upgrade-"+op+"-snapshot", "failed-home-"+op
if a.recordSnapshot(i, snapshot, i.Version, i.ImageID) != nil || a.recordSnapshot(i, failedHome, image.Config.Labels["dsh-mgr.version"], image.ID) != nil {
fail(w, 500, "snapshot metadata unavailable")
return
}
newName := runtimeNetwork(i.ID) + "-upgrade-" + op
_, err = a.s.DB.Exec(`CREATE TABLE IF NOT EXISTS upgrade_journal (id TEXT PRIMARY KEY, instance_id TEXT NOT NULL, old_container TEXT NOT NULL, new_container TEXT NOT NULL DEFAULT '', new_name TEXT NOT NULL, image_id TEXT NOT NULL, snapshot TEXT NOT NULL, failed_home TEXT NOT NULL, phase TEXT NOT NULL)`)
if err != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
_, err = a.s.DB.Exec("CREATE TABLE IF NOT EXISTS upgrade_state (id TEXT PRIMARY KEY, old_status TEXT NOT NULL, was_running INTEGER NOT NULL)")
if err != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
tx, err := a.s.DB.Begin()
if err != nil {
fail(w, 500, "database error")
return
}
_, err = tx.Exec("INSERT INTO upgrade_journal(id,instance_id,old_container,new_name,image_id,snapshot,failed_home,phase) VALUES(?,?,?,?,?,?,?,'snapshotting')", op, i.ID, i.ContainerID, newName, image.ID, snapshot, failedHome)
if err == nil {
_, err = tx.Exec("INSERT INTO upgrade_state VALUES(?,?,?)", op, old.State.Status, old.State.Running)
}
if err == nil {
_, err = tx.Exec("UPDATE instances SET status='error' WHERE id=?", i.ID)
}
if err != nil {
tx.Rollback()
fail(w, 500, "database error")
return
}
if tx.Commit() != nil {
fail(w, 500, "database error")
return
}
phase := func(p string) error {
_, err := a.s.DB.Exec("UPDATE upgrade_journal SET phase=? WHERE id=?", p, op)
return err
}
newID, complete, snapReady, createAttempted := "", false, false, false
stopAttempted := false
defer func() {
if complete {
return
}
cleanup, done := context.WithTimeout(context.Background(), 30*time.Second)
defer done()
if phase("rolling_back") != nil {
return
}
if createAttempted {
id := newID
if id == "" {
id = newName
}
// A timed-out create can still have succeeded; the journaled unique name is authoritative.
_ = a.docker.Do(cleanup, "POST", "/containers/"+url.PathEscape(id)+"/stop?t=10", nil, nil)
if a.removeRuntimeContainer(cleanup, id) != nil {
return
}
}
if snapReady && createAttempted {
if phase("saving_failed_home") != nil {
return
}
if root.Rename("home", failedHome) != nil || upgradeSync(root) != nil {
return
}
if phase("restoring_home") != nil {
return
}
if root.Rename(snapshot, "home") != nil || upgradeSync(root) != nil {
return
}
}
if stopAttempted && old.State.Running {
if phase("restoring_runtime") != nil || a.docker.Do(cleanup, "POST", "/containers/"+url.PathEscape(i.ContainerID)+"/start", nil, nil) != nil {
return
}
}
runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
tx, err := a.s.DB.Begin()
if err != nil {
return
}
defer tx.Rollback()
if _, err = tx.Exec("UPDATE upgrade_journal SET phase='rolled_back' WHERE id=?", op); err != nil {
return
}
if _, err = tx.Exec("UPDATE instances SET status=? WHERE id=?", old.State.Status, i.ID); err != nil {
return
}
_ = tx.Commit()
}()
if old.State.Running {
if phase("stopping") != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
stopAttempted = true
if a.docker.Do(ctx, "POST", "/containers/"+url.PathEscape(i.ContainerID)+"/stop?t=10", nil, nil) != nil {
fail(w, 502, "old runtime stop failed")
return
}
}
stopped, inspectErr := a.docker.Inspect(ctx, i.ContainerID)
if inspectErr != nil || stopped.State.Running || stopped.State.Paused || stopped.State.Restarting {
fail(w, 502, "old runtime did not stop")
return
}
if phase("snapshotting") != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
snapshotBytes, quotaErr := a.quotaSnapshot(i.UID, root, 0)
if quotaErr != nil {
fail(w, 413, "quota exceeded or snapshot size unavailable")
return
}
// Only publish a completed copy under a name the snapshot API recognizes.
staging := "." + snapshot + "-partial"
if _, err := a.s.DB.Exec("CREATE TABLE IF NOT EXISTS snapshot_copy_state (instance_id TEXT NOT NULL, id TEXT NOT NULL, ready INTEGER NOT NULL, PRIMARY KEY(instance_id,id))"); err != nil {
fail(w, 500, "snapshot completion metadata unavailable")
return
}
if _, err := a.s.DB.Exec("INSERT INTO snapshot_copy_state VALUES(?,?,0)", i.ID, snapshot); err != nil {
fail(w, 500, "snapshot completion metadata unavailable")
return
}
if copyErr := upgradeCopy(ctx, root, "home", staging, snapshotBytes); copyErr != nil {
if cleanupErr := root.RemoveAll(staging); cleanupErr != nil {
a.s.AddAudit(i.UID, "upgrade.snapshot.cleanup_failed", i.ID, staging)
fail(w, 500, "home snapshot failed: "+copyErr.Error()+"; partial snapshot cleanup required")
return
}
fail(w, 500, "home snapshot failed: "+copyErr.Error())
return
}
if err := root.Rename(staging, snapshot); err != nil {
if root.RemoveAll(staging) != nil {
a.s.AddAudit(i.UID, "upgrade.snapshot.cleanup_failed", i.ID, staging)
}
fail(w, 500, "home snapshot publication failed")
return
}
if upgradeSync(root) != nil {
fail(w, 500, "home snapshot publication sync failed")
return
}
if _, err := a.s.DB.Exec("UPDATE snapshot_copy_state SET ready=1 WHERE instance_id=? AND id=?", i.ID, snapshot); err != nil {
fail(w, 500, "snapshot completion metadata unavailable")
return
}
snapReady = true
backupDir := filepath.Join(a.root, "backups", i.UID, i.ID)
storage, err := os.OpenRoot(a.root)
if err != nil {
fail(w, 500, "backup storage unavailable")
return
}
err = configCheckPath(storage, filepath.ToSlash(filepath.Join("backups", i.UID, i.ID, "archive")))
storage.Close()
if err != nil || os.MkdirAll(backupDir, 0700) != nil {
fail(w, 500, "unsafe backup storage")
return
}
entries, err := os.ReadDir(backupDir)
if err != nil || len(entries) >= 20 {
fail(w, 409, "backup retention limit reached")
return
}
backupID := store.ID()
backupPath := filepath.Join(backupDir, backupID+".dshbak")
if err = backup.CreateWithPassword(filepath.Join(root.Name(), "home"), backupPath, in.Password, backup.Metadata{DSHVersion: i.Version, CreatedAt: time.Now().UTC()}); err != nil {
fail(w, 500, "pre-upgrade backup failed")
return
}
backupStat, err := os.Stat(backupPath)
if err != nil {
fail(w, 500, "pre-upgrade backup unavailable")
return
}
if err = a.indexBackup(i, backupID, "pre-upgrade", 2, backupStat.Size()); err != nil {
os.Remove(backupPath)
fail(w, 500, "backup index unavailable")
return
}
delete(old.Config, "Hostname")
old.Config["Image"], _ = json.Marshal(image.ID)
old.Config["Cmd"], _ = json.Marshal([]string{"web", "--no-open", "--port", "3080", "--trusted-host", a.runtimeHost(r)})
old.Config["HostConfig"], _ = json.Marshal(old.HostConfig)
endpoints := map[string]any{}
for network := range old.NetworkSettings.Networks {
endpoints[network] = map[string]any{}
}
old.Config["NetworkingConfig"], _ = json.Marshal(map[string]any{"EndpointsConfig": endpoints})
if phase("creating") != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
createAttempted = true
var created struct {
ID string `json:"Id"`
}
if a.docker.Do(ctx, "POST", "/containers/create?name="+url.QueryEscape(newName), old.Config, &created) != nil || created.ID == "" {
fail(w, 502, "upgrade container creation failed")
return
}
newID = created.ID
if _, err = a.s.DB.Exec("UPDATE upgrade_journal SET new_container=?, phase='starting' WHERE id=?", newID, op); err != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
if a.docker.Do(ctx, "POST", "/containers/"+url.PathEscape(newID)+"/start", nil, nil) != nil {
fail(w, 502, "upgrade start failed")
return
}
if phase("readiness") != nil {
fail(w, 500, "upgrade journal unavailable")
return
}
if a.upgradeReady(ctx, newID, i.ID, a.runtimeHost(r)) != nil {
fail(w, 502, "upgrade readiness failed")
return
}
tx, err = a.s.DB.Begin()
if err != nil {
fail(w, 500, "database error")
return
}
defer tx.Rollback()
_, err = tx.Exec("UPDATE instances SET container_id=?,image_id=?,version=?,status='running' WHERE id=?", newID, image.ID, image.Config.Labels["dsh-mgr.version"], i.ID)
if err == nil {
_, err = tx.Exec("UPDATE upgrade_journal SET phase='committed' WHERE id=?", op)
}
if err != nil {
fail(w, 500, "database error")
return
}
if tx.Commit() != nil {
fail(w, 500, "database error")
return
}
complete = true
if a.removeRuntimeContainer(ctx, i.ContainerID) != nil || phase("complete") != nil {
a.s.SetInstance(i.ID, "error")
fail(w, 502, "upgrade committed; old container cleanup requires recovery")
return
}
runtimeAuthCache.Delete(runtimeAuthKey{a, i.ID})
i.ContainerID, i.ImageID, i.Version, i.Status = newID, image.ID, image.Config.Labels["dsh-mgr.version"], "running"
a.s.AddAudit(i.UID, "instance.upgrade", i.ID, image.ID)
writeJSON(w, i)
}
func (a *API) upgradeReady(ctx context.Context, id, instanceID, host string) error {
ctx, cancel := context.WithTimeout(ctx, 150*time.Second)
defer cancel()
ticker := time.NewTicker(250 * time.Millisecond)
defer ticker.Stop()
for {
c, err := a.docker.Inspect(ctx, id)
if err != nil {
return errors.New("readiness inspection failed")
}
if !c.State.Running {
return errors.New("runtime stopped")
}
logs, err := a.docker.Logs(ctx, id)
if err == nil && latestRuntimeToken(logs) != "" && (c.State.Health == nil || c.State.Health.Status == "healthy") {
{
ip := c.NetworkSettings.Networks[runtimeNetwork(instanceID)].IPAddress
if net.ParseIP(ip) != nil {
target := &url.URL{Scheme: "http", Host: net.JoinHostPort(ip, "8080")}
if cookies, e := exchangeRuntimeCookie(ctx, target, host, latestRuntimeToken(logs)); e == nil {
req, _ := http.NewRequestWithContext(ctx, "GET", target.String()+"/", nil)
req.Host = host
req.Header.Set("Cookie", cookies)
client := &http.Client{Transport: runtimeTransport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
if resp, e := client.Do(req); e == nil {
resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return nil
}
}
}
}
}
}
select {
case <-ctx.Done():
return ctx.Err()
case <-ticker.C:
}
}
}
internal/api/upgrade_test.go
-583
View File
@@ -1,583 +0,0 @@
package api
import (
"context"
"encoding/json"
"fmt"
"net"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"syscall"
"testing"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
"golang.org/x/crypto/bcrypt"
)
func TestUpgradeWorkspaceCompatibility(t *testing.T) {
for _, tc := range []struct {
name, recipe, target string
shared, reject bool
}{
{"legacy-to-seven", "7", "/data/dsh-home", true, true},
{"single-to-seven", "7", "/workspace", false, false},
{"migrated-to-seven", "7", "/workspace", true, true},
{"legacy-to-six", "6", "/data/dsh-home", true, true},
{"legacy-to-unknown", "", "/data/dsh-home", true, true},
{"single-to-six", "6", "/workspace", false, true},
{"single-to-unknown", "", "/workspace", false, true},
} {
t.Run(tc.name, func(t *testing.T) {
mounts := []map[string]any{{"Type": "bind", "Source": "/instance/home", "Target": tc.target, "ReadOnly": false}}
if tc.shared {
target := "/workspace"
if tc.target == "/workspace" {
target = "/shared"
}
mounts = append(mounts, map[string]any{"Type": "bind", "Source": "/user/files", "Target": target, "ReadOnly": true})
}
raw, _ := json.Marshal(mounts)
host := map[string]json.RawMessage{"Mounts": raw}
config := map[string]json.RawMessage{"Env": json.RawMessage(`["CUSTOM=yes","DSH_HOME=/data/dsh-home","HOME=/data/dsh-home","XDG_DATA_HOME=/old/data","XDG_CONFIG_HOME=/old/config"]`)}
before := string(config["Env"])
err := upgradeWorkspace(config, host, tc.recipe, "/instance/home")
if (err != nil) != tc.reject {
t.Fatalf("error=%v", err)
}
if tc.reject {
if string(host["Mounts"]) != string(raw) || string(config["Env"]) != before {
t.Fatal("legacy or rejected config changed")
}
return
}
if !strings.Contains(string(config["Env"]), "CUSTOM=yes") || strings.Contains(string(config["Env"]), "/data/dsh-home") {
t.Fatal(string(config["Env"]))
}
var got []map[string]any
if err := json.Unmarshal(host["Mounts"], &got); err != nil {
t.Fatal(err)
}
if got[0]["Source"] != "/instance/home" || got[0]["Target"] != "/workspace" || len(got) != 1 {
t.Fatal(got)
}
first, _ := json.Marshal([]any{config, host})
if err := upgradeWorkspace(config, host, tc.recipe, "/instance/home"); err != nil {
t.Fatal(err)
}
second, _ := json.Marshal([]any{config, host})
if string(first) != string(second) {
t.Fatal("conversion is not idempotent")
}
})
}
}
func TestUpgradeWorkspaceRejectsAmbiguousMounts(t *testing.T) {
for _, extra := range []string{
`{"Type":"bind","Source":"/other","Target":"/shared"}`,
`{"Type":"bind","Source":"/other","Target":"/workspace"}`,
`{"Type":"volume","Source":"/user/files","Target":"/workspace"}`,
`{"Type":"bind","Source":"/instance/home","Target":"/workspace"}`,
} {
config := map[string]json.RawMessage{"Env": json.RawMessage(`["CUSTOM=yes"]`)}
host := map[string]json.RawMessage{"Mounts": json.RawMessage(`[{"Type":"bind","Source":"/instance/home","Target":"/workspace"},` + extra + `]`)}
before, _ := json.Marshal([]any{config, host})
if err := upgradeWorkspace(config, host, "7", "/instance/home"); err == nil {
t.Fatal("accepted ambiguous mounts", extra)
}
after, _ := json.Marshal([]any{config, host})
if string(before) != string(after) {
t.Fatal("rejected config mutated")
}
}
}
func TestUpgradeLifecycle(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
for _, mode := range []string{"snapshot-fifo", "snapshot-dangling-link", "success", "legacy-layout", "shared-layout", "start-failure", "readiness-failure", "cleanup-failure", "running", "running-failure", "restart-failure", "delete-failure", "invalid-image", "unverified-image", "failed-build", "version-mismatch", "image-id-mismatch", "db-failure"} {
t.Run(mode, func(t *testing.T) {
image := "sha256:" + strings.Repeat("b", 64)
var a *API
var home, shared string
var calls []string
var body map[string]json.RawMessage
running := mode == "running" || mode == "running-failure" || mode == "restart-failure"
stopped := false
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Host != "panel.test" {
t.Error("wrong trusted host")
}
if r.URL.Query().Get("token") == "must-not-leak" {
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: "ready"})
w.WriteHeader(303)
return
}
if r.Header.Get("Cookie") != "dsh_auth=ready" {
t.Error("missing authenticated readiness")
}
w.WriteHeader(200)
}))
defer upstream.Close()
transport := runtimeTransport
runtimeTransport = transport.Clone()
runtimeTransport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, network, strings.TrimPrefix(upstream.URL, "http://"))
}
defer func() { runtimeTransport.CloseIdleConnections(); runtimeTransport = transport }()
handler := func(w http.ResponseWriter, r *http.Request) {
p := strings.TrimPrefix(r.URL.Path, "/v1.47")
calls = append(calls, r.Method+" "+p)
switch {
case p == "/containers/new/exec":
writeJSON(w, map[string]string{"Id": "reconcile"})
case p == "/exec/reconcile/json":
writeJSON(w, map[string]any{"Running": false, "ExitCode": 0})
case p == "/containers/old/json":
state := "exited"
if running {
state = "running"
}
mounts := []map[string]string{{"Type": "bind", "Source": home, "Target": "/workspace"}}
if mode == "legacy-layout" {
mounts[0]["Target"] = "/data/dsh-home"
mounts = append(mounts, map[string]string{"Type": "bind", "Source": shared, "Target": "/workspace"})
}
if mode == "shared-layout" {
mounts = append(mounts, map[string]string{"Type": "bind", "Source": shared, "Target": "/shared"})
}
writeJSON(w, map[string]any{"Config": map[string]any{"Hostname": "old", "Cmd": []string{"obsolete-flag"}, "Image": "old-image", "Env": []string{"CUSTOM=yes"}, "User": "1000:1000"}, "HostConfig": map[string]any{"NetworkMode": "dsh-inst-one", "Memory": 9007199254740993, "Mounts": mounts}, "State": map[string]any{"Status": state, "Running": running && !stopped}, "NetworkSettings": map[string]any{"Networks": map[string]any{"dsh-inst-one": map[string]any{"IPAddress": "172.18.0.3", "Aliases": []string{"old"}}}}})
case strings.HasPrefix(p, "/images/"):
role := "runtime"
if mode == "invalid-image" {
role = "builder"
}
writeJSON(w, map[string]any{"Id": image, "Os": "linux", "Architecture": "amd64", "Config": map[string]any{"Labels": map[string]string{"dsh-mgr.role": role, "dsh-mgr.version": "v2", "dsh-mgr.recipe": imageRecipeRev}}})
case p == "/containers/create":
if !strings.HasPrefix(r.URL.Query().Get("name"), "dsh-inst-one-upgrade-") {
t.Error("missing unique name")
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
t.Error(err)
}
before, _ := a.s.GetInstance("one")
if before.ContainerID != "old" || before.Status != "error" {
t.Errorf("premature switch: %+v", before)
}
writeJSON(w, map[string]string{"Id": "new"})
case p == "/containers/old/stop":
stopped = true
w.WriteHeader(204)
case p == "/containers/old/start":
restored, _ := os.ReadFile(filepath.Join(home, ".config", "state"))
if string(restored) != "original" {
t.Error("old runtime restarted before home restore")
}
if mode == "restart-failure" {
http.Error(w, "failed", 500)
return
}
w.WriteHeader(204)
case p == "/containers/old" && r.Method == "DELETE":
current, _ := a.s.GetInstance("one")
if current.ContainerID != "new" {
t.Error("old removed before commit")
}
if mode == "delete-failure" {
http.Error(w, "failed", 500)
return
}
w.WriteHeader(204)
case p == "/containers/new/start":
if err := os.WriteFile(filepath.Join(home, ".config", "state"), []byte("migrated"), 0600); err != nil {
t.Error(err)
}
if err := os.WriteFile(filepath.Join(shared, "file"), []byte("shared-new"), 0600); err != nil {
t.Error(err)
}
if mode == "start-failure" || mode == "cleanup-failure" || mode == "running-failure" || mode == "restart-failure" {
http.Error(w, "token=must-not-leak", 500)
return
}
w.WriteHeader(204)
case p == "/containers/new/json":
writeJSON(w, map[string]any{"State": map[string]any{"Running": mode != "readiness-failure"}, "NetworkSettings": map[string]any{"Networks": map[string]any{"dsh-inst-one": map[string]string{"IPAddress": "127.0.0.1"}}}})
case p == "/containers/new/logs":
if mode == "db-failure" {
_, err := a.s.DB.Exec("CREATE TRIGGER reject_switch BEFORE UPDATE OF container_id ON instances BEGIN SELECT RAISE(FAIL, 'blocked'); END")
if err != nil {
t.Error(err)
}
}
fmt.Fprint(w, "http://runtime/?token=must-not-leak\n")
case p == "/containers/new" && r.Method == "DELETE":
if mode == "cleanup-failure" {
http.Error(w, "token=must-not-leak", 500)
return
}
w.WriteHeader(204)
default:
w.WriteHeader(204)
}
}
var sid string
a, sid = runtimeTestAPI(t, handler)
hash, _ := bcrypt.GenerateFromPassword([]byte("upgrade-password"), bcrypt.MinCost)
a.s.DB.Exec("UPDATE users SET pass_hash=?,quota=10737418240 WHERE id='owner'", string(hash))
if _, err := a.verifiedImages(context.Background()); err != nil {
t.Fatal(err)
}
if mode != "unverified-image" {
status, version, builtImage := "success", "v2", image
switch mode {
case "failed-build":
status = "failed"
case "version-mismatch":
version = "v3"
case "image-id-mismatch":
builtImage = "sha256:" + strings.Repeat("c", 64)
}
if _, err := a.s.DB.Exec("INSERT INTO image_builds(id,version,status,image_id,created_at,recipe_rev,platform) VALUES(?,?,?,?,?,?,?)", store.ID(), version, status, builtImage, time.Now().UTC().Format(time.RFC3339Nano), imageRecipeRev, imagePlatform); err != nil {
t.Fatal(err)
}
}
if _, err := a.s.DB.Exec("INSERT INTO image_admissions(image_id,version,recipe_rev,platform,state,evidence,admitted_by,admitted_at) VALUES(?,'v2',?,?,'compatible','test acceptance','test','now')", image, imageRecipeRev, imagePlatform); err != nil {
t.Fatal(err)
}
base := filepath.Join(a.root, "workspaces", "owner", "instances", "one")
home = filepath.Join(base, "home")
shared = filepath.Join(a.root, "workspaces", "owner", "files")
for _, p := range []string{filepath.Join(home, ".config"), shared} {
if err := os.MkdirAll(p, 0750); err != nil {
t.Fatal(err)
}
}
if err := os.WriteFile(filepath.Join(home, ".config", "state"), []byte("original"), 0600); err != nil {
t.Fatal(err)
}
if mode == "snapshot-dangling-link" {
if err := os.Symlink("missing", filepath.Join(home, "dangling")); err != nil {
t.Fatal(err)
}
}
if mode == "snapshot-fifo" {
if err := syscall.Mkfifo(filepath.Join(home, "pipe"), 0600); err != nil {
t.Fatal(err)
}
}
i := store.Instance{ID: "one", UID: "owner", Name: "one", ContainerID: "old", ImageID: "old-image", Version: "v1", Status: "exited", Created: time.Now()}
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
w := httptest.NewRecorder()
stale := i
stale.ContainerID = "stale"
a.upgrade(w, runtimeRequest("POST", "/api/instances/one/upgrade", fmt.Sprintf(`{"image_id":%q,"password":"upgrade-password"}`, image), sid), stale)
if strings.Contains(w.Body.String(), "must-not-leak") {
t.Fatal("leaked token")
}
got, err := a.s.GetInstance(i.ID)
if err != nil {
t.Fatal(err)
}
if mode == "legacy-layout" || mode == "shared-layout" || mode == "invalid-image" || mode == "unverified-image" || mode == "failed-build" || mode == "version-mismatch" || mode == "image-id-mismatch" {
want := 400
if mode == "legacy-layout" || mode == "shared-layout" {
want = 409
}
if w.Code != want {
t.Fatalf("%d %s", w.Code, w.Body.String())
}
if got.ContainerID != i.ContainerID || got.ImageID != i.ImageID || got.Version != i.Version || got.Status != i.Status {
t.Fatalf("rejected upgrade changed instance: %+v", got)
}
for _, call := range calls {
if !strings.HasPrefix(call, "GET ") {
t.Errorf("rejected upgrade mutated runtime: %s", call)
}
}
return
}
var phase, snapshot, failed string
if err := a.s.DB.QueryRow("SELECT phase,snapshot,failed_home FROM upgrade_journal WHERE instance_id='one'").Scan(&phase, &snapshot, &failed); err != nil {
t.Fatal(err)
}
if mode == "snapshot-fifo" {
if w.Code != 500 || !strings.Contains(w.Body.String(), "home/pipe") || phase != "rolled_back" || got.ContainerID != "old" {
t.Fatalf("%d %s phase=%s", w.Code, w.Body.String(), phase)
}
entries, err := os.ReadDir(base)
if err != nil || len(entries) != 1 || entries[0].Name() != "home" {
t.Fatalf("partial snapshot leaked: %v %v", entries, err)
}
listed := configCall(a, got, "GET", "snapshots", "")
if listed.Code != 200 || strings.TrimSpace(listed.Body.String()) != "[]" {
t.Fatal(listed.Code, listed.Body.String())
}
return
}
if mode == "snapshot-dangling-link" {
if w.Code != 200 || phase != "complete" || got.ContainerID != "new" {
t.Fatalf("unexpected result: %d %s phase=%s instance=%+v", w.Code, w.Body.String(), phase, got)
}
saved, err := os.ReadFile(filepath.Join(base, snapshot, ".config", "state"))
if err != nil || string(saved) != "original" {
t.Fatalf("copied data missing: %q %v", saved, err)
}
link, err := os.Readlink(filepath.Join(base, snapshot, "dangling"))
if err != nil || link != "missing" {
t.Fatalf("copied link missing: %q %v", link, err)
}
listed := configCall(a, got, "GET", "snapshots", "")
if listed.Code != 200 || !strings.Contains(listed.Body.String(), snapshot) {
t.Fatalf("snapshot not listed: %d %s", listed.Code, listed.Body.String())
}
return
}
data, err := os.ReadFile(filepath.Join(home, ".config", "state"))
if err != nil {
t.Fatal(err)
}
if mode == "success" || mode == "running" {
if w.Code != 200 || got.ContainerID != "new" || got.Version != "v2" || phase != "complete" || string(data) != "migrated" {
t.Fatalf("%d %s %+v phase=%s home=%s", w.Code, w.Body.String(), got, phase, data)
}
saved, err := os.ReadFile(filepath.Join(base, snapshot, ".config", "state"))
if err != nil || string(saved) != "original" {
t.Fatalf("snapshot %q %v", saved, err)
}
st, _ := os.Stat(filepath.Join(base, snapshot, ".config", "state"))
if st.Sys().(*syscall.Stat_t).Uid != 1000 {
t.Error("snapshot ownership")
}
} else if mode == "delete-failure" {
if w.Code != 502 || got.ContainerID != "new" || got.Status != "error" || phase != "committed" || string(data) != "migrated" {
t.Fatalf("unsafe delete failure %+v %s", got, phase)
}
} else if mode == "restart-failure" {
if w.Code < 400 || got.ContainerID != "old" || got.Status != "error" || phase != "restoring_runtime" || string(data) != "original" {
t.Fatalf("unsafe restart failure %+v %s", got, phase)
}
} else if mode == "cleanup-failure" {
if got.Status != "error" || phase != "rolling_back" || string(data) != "migrated" || !a.upgradeRecoveryRequired(got) {
t.Fatalf("unsafe cleanup %+v %s %s", got, phase, data)
}
} else {
if w.Code < 400 || got.ContainerID != "old" || got.ImageID != "old-image" || got.Status != map[bool]string{true: "running", false: "exited"}[running] || phase != "rolled_back" || string(data) != "original" {
t.Fatalf("%d %+v phase=%s home=%s", w.Code, got, phase, data)
}
failedData, err := os.ReadFile(filepath.Join(base, failed, ".config", "state"))
if err != nil || string(failedData) != "migrated" {
t.Fatalf("failed home %s %v", failedData, err)
}
}
sharedData, _ := os.ReadFile(filepath.Join(shared, "file"))
if string(sharedData) != "shared-new" {
t.Error("shared workspace rolled back")
}
if _, ok := body["Hostname"]; ok {
t.Error("hostname copied")
}
if string(body["Env"]) != `["CUSTOM=yes","DSH_HOME=/workspace","HOME=/workspace","XDG_DATA_HOME=/workspace/.local/share","XDG_CONFIG_HOME=/workspace/.config"]` || !strings.Contains(string(body["HostConfig"]), "9007199254740993") {
t.Error("raw config not preserved")
}
if strings.Contains(string(body["HostConfig"]), `"Target":"/shared"`) || strings.Contains(string(body["HostConfig"]), "/data/dsh-home") {
t.Error("workspace mounts not migrated")
}
if strings.Contains(string(body["NetworkingConfig"]), "172.18") || strings.Contains(string(body["NetworkingConfig"]), "Aliases") {
t.Error("dynamic endpoint copied")
}
if string(body["Cmd"]) != `["web","--no-open","--port","3080","--trusted-host","panel.test"]` {
t.Errorf("command not recalculated: %s", body["Cmd"])
}
has := func(want string) bool {
for _, call := range calls {
if call == want {
return true
}
}
return false
}
if has("DELETE /containers/old") != (mode == "success" || mode == "running" || mode == "delete-failure") {
t.Error("old deletion lifecycle")
}
if has("POST /containers/old/start") != (mode == "running-failure" || mode == "restart-failure") {
t.Error("old restart lifecycle")
}
if has("POST /containers/old/stop") != running {
t.Error("old stop lifecycle")
}
})
}
}
func TestUpgradeSnapshotRejectsUnsafeFiles(t *testing.T) {
for _, kind := range []string{"fifo", "oversize"} {
t.Run(kind, func(t *testing.T) {
base := t.TempDir()
if err := os.Mkdir(filepath.Join(base, "home"), 0700); err != nil {
t.Fatal(err)
}
p := filepath.Join(base, "home", "bad")
var err error
switch kind {
case "fifo":
err = syscall.Mkfifo(p, 0600)
case "oversize":
err = os.WriteFile(p, []byte("12345"), 0600)
}
if err != nil {
t.Fatal(err)
}
root, err := os.OpenRoot(base)
if err != nil {
t.Fatal(err)
}
defer root.Close()
if upgradeCopy(context.Background(), root, "home", "snapshot", 4) == nil {
t.Fatal("unsafe file accepted")
}
})
}
}
func TestUpgradeSnapshotRelativeLinks(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("requires chown")
}
for _, mode := range []string{"pnpm", "absolute", "escape", "chain-escape", "loop", "dangling"} {
t.Run(mode, func(t *testing.T) {
base := t.TempDir()
home := filepath.Join(base, "home")
pkg := filepath.Join(home, "node_modules/.pnpm/pkg/node_modules/pkg")
if err := os.MkdirAll(pkg, 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(pkg, "index.js"), []byte("original"), 0640); err != nil {
t.Fatal(err)
}
link := ".pnpm/pkg/node_modules/pkg"
switch mode {
case "absolute":
link = "/workspace/example"
case "escape":
link = "../../outside"
case "chain-escape":
if err := os.Symlink(".", filepath.Join(home, "alias")); err != nil {
t.Fatal(err)
}
link = "../alias/../outside"
case "loop":
link = "pkg"
case "dangling":
link = "missing"
}
if err := os.Symlink(link, filepath.Join(home, "node_modules/pkg")); err != nil {
t.Fatal(err)
}
root, err := os.OpenRoot(base)
if err != nil {
t.Fatal(err)
}
defer root.Close()
err = upgradeCopy(context.Background(), root, "home", "snapshot", 1024)
if err != nil {
t.Fatal(err)
}
got, err := root.Readlink("snapshot/node_modules/pkg")
if err != nil || got != link {
t.Fatalf("link not preserved: %q %v", got, err)
}
if err := root.Rename("home", "failed-home"); err != nil {
t.Fatal(err)
}
if err := root.Rename("snapshot", "home"); err != nil {
t.Fatal(err)
}
if mode != "pnpm" {
got, err := root.Readlink("home/node_modules/pkg")
if err != nil || got != link {
t.Fatalf("restored link changed: %q %v", got, err)
}
return
}
data, err := os.ReadFile(filepath.Join(home, "node_modules/pkg/index.js"))
if err != nil || string(data) != "original" {
t.Fatalf("rollback link broken: %q %v", data, err)
}
})
}
}
func TestUpgradeReadinessRequiresToken(t *testing.T) {
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/json") {
writeJSON(w, map[string]any{"State": map[string]bool{"Running": true}})
} else {
fmt.Fprint(w, "starting without token")
}
})
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Millisecond)
defer cancel()
if a.upgradeReady(ctx, "new", "one", "panel.test") == nil {
t.Fatal("accepted without token")
}
}
func TestUpgradeReadinessRequiresAuthenticatedHTTP(t *testing.T) {
for _, mode := range []string{"success", "token-rejected", "missing-cookie", "unauthorized", "redirect", "unavailable"} {
t.Run(mode, func(t *testing.T) {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Query().Get("token") != "" {
if mode == "token-rejected" {
w.WriteHeader(401)
return
}
if mode != "missing-cookie" {
http.SetCookie(w, &http.Cookie{Name: "dsh_auth", Value: "ready"})
}
w.WriteHeader(303)
return
}
if r.Header.Get("Cookie") != "dsh_auth=ready" {
t.Error("cookie not sent")
}
switch mode {
case "unauthorized":
w.WriteHeader(401)
case "redirect":
w.WriteHeader(302)
case "unavailable":
w.WriteHeader(503)
default:
w.WriteHeader(200)
}
}))
defer upstream.Close()
transport := runtimeTransport
runtimeTransport = transport.Clone()
runtimeTransport.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, network, strings.TrimPrefix(upstream.URL, "http://"))
}
defer func() { runtimeTransport.CloseIdleConnections(); runtimeTransport = transport }()
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
if strings.HasSuffix(r.URL.Path, "/json") {
writeJSON(w, map[string]any{"State": map[string]bool{"Running": true}, "NetworkSettings": map[string]any{"Networks": map[string]any{"dsh-inst-one": map[string]string{"IPAddress": "127.0.0.1"}}}})
} else {
fmt.Fprint(w, "http://runtime/?token=ready\n")
}
})
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
err := a.upgradeReady(ctx, "new", "one", "panel.test")
if (err == nil) != (mode == "success") {
t.Fatalf("readiness=%v", err)
}
})
}
}
internal/api/workspace_concurrency_test.go
-298
View File
@@ -1,298 +0,0 @@
package api
import (
"errors"
"fmt"
"io"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
type pausedDownload struct {
*httptest.ResponseRecorder
started chan struct{}
release chan struct{}
once sync.Once
}
func (w *pausedDownload) Write(p []byte) (int, error) {
w.once.Do(func() { close(w.started) })
<-w.release
return w.ResponseRecorder.Write(p)
}
func TestSlowDownloadDoesNotHoldLifecycleLock(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker call") })
for _, id := range []string{"one", "two"} {
if err := a.s.AddInstance(store.Instance{ID: id, UID: "owner"}); err != nil {
t.Fatal(err)
}
dir := filepath.Join(a.root, "workspaces/owner/instances", id, "home")
if err := os.MkdirAll(dir, 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, "file.txt"), []byte("content"), 0600); err != nil {
t.Fatal(err)
}
}
w := &pausedDownload{ResponseRecorder: httptest.NewRecorder(), started: make(chan struct{}), release: make(chan struct{})}
done := make(chan struct{})
go func() {
defer close(done)
a.workspace(w, runtimeRequest("GET", "/api/ws/download?instance=one&path=file.txt", "", sid))
}()
defer func() { close(w.release); <-done }()
select {
case <-w.started:
case <-time.After(3 * time.Second):
t.Fatal("download not started")
}
if a.instanceLocks.held() {
t.Fatal("slow network holds instance lifecycle lock")
}
a.readersMu.Lock()
active := a.workspaceReaders["one"]
a.readersMu.Unlock()
if active != 1 {
t.Fatalf("lease count %d", active)
}
other := httptest.NewRecorder()
otherDone := make(chan struct{})
go func() {
defer close(otherDone)
a.workspace(other, runtimeRequest("GET", "/api/ws/tree?instance=two", "", sid))
}()
select {
case <-otherDone:
case <-time.After(3 * time.Second):
t.Fatal("other instance blocked")
}
if other.Code != 200 {
t.Fatalf("other instance: %d", other.Code)
}
proxyDone := make(chan struct{})
proxyResponse := httptest.NewRecorder()
go func() {
defer close(proxyDone)
a.proxy(proxyResponse, runtimeRequest("GET", "/dsh/missing/", "", sid))
}()
select {
case <-proxyDone:
case <-time.After(3 * time.Second):
t.Fatal("proxy entry blocked by download")
}
if proxyResponse.Code != 404 {
t.Fatalf("proxy status %d", proxyResponse.Code)
}
for _, action := range []string{"restore", "upgrade", "stop", ""} {
method := "POST"
if action == "" {
method = "DELETE"
}
response := httptest.NewRecorder()
url := "/api/instances/one"
if action != "" {
url += "/" + action
}
body := `{}`
if action == "upgrade" {
body = `{"image_id":"sha256:` + strings.Repeat("a", 64) + `"}`
}
a.instance(response, runtimeRequest(method, url, body, sid))
if response.Code != 409 {
t.Errorf("%s during download: %d %s", action, response.Code, response.Body.String())
}
}
}
func TestDirectoryEntryLimit(t *testing.T) {
dir := t.TempDir()
for i := 0; i <= maxDirectoryEntries; i++ {
f, err := os.Create(filepath.Join(dir, fmt.Sprint(i)))
if err != nil {
t.Fatal(err)
}
f.Close()
}
root, err := os.OpenRoot(dir)
if err != nil {
t.Fatal(err)
}
defer root.Close()
if _, err = readRootDir(root, "."); !errors.Is(err, errDirectoryLimit) {
t.Fatalf("limit: %v", err)
}
if err = os.Remove(filepath.Join(dir, fmt.Sprint(maxDirectoryEntries))); err != nil {
t.Fatal(err)
}
entries, err := readRootDir(root, ".")
if err != nil || len(entries) != maxDirectoryEntries {
t.Fatalf("boundary: %d %v", len(entries), err)
}
}
type pausedUpload struct {
started chan struct{}
release chan struct{}
once sync.Once
}
func (r *pausedUpload) Read(p []byte) (int, error) {
r.once.Do(func() { close(r.started) })
<-r.release
return 0, io.EOF
}
func (r *pausedUpload) Close() error { return nil }
func TestSlowUploadDoesNotHoldLifecycleLock(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker call") })
if err := a.s.AddInstance(store.Instance{ID: "one", UID: "owner"}); err != nil {
t.Fatal(err)
}
if _, err := a.s.DB.Exec("UPDATE users SET quota=? WHERE id='owner'", 1<<30); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(a.root, "workspaces/owner/instances/one/home"), 0750); err != nil {
t.Fatal(err)
}
body := &pausedUpload{started: make(chan struct{}), release: make(chan struct{})}
r := runtimeRequest("POST", "/api/ws/upload?instance=one", "", sid)
r.Body = body
r.Header.Set("Content-Type", "multipart/form-data; boundary=test")
done := make(chan struct{})
go func() { defer close(done); a.workspace(httptest.NewRecorder(), r) }()
defer func() { close(body.release); <-done }()
select {
case <-body.started:
case <-time.After(3 * time.Second):
t.Fatal("upload not reading body")
}
if a.instanceLocks.held() {
t.Fatal("slow upload holds instance lock")
}
if a.userLocks.held() {
t.Fatal("slow upload holds user write lock")
}
}
func TestWorkspaceErrorResponseReleasesLocks(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker call") })
if err := a.s.AddInstance(store.Instance{ID: "one", UID: "owner"}); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(filepath.Join(a.root, "workspaces/owner/instances/one/home"), 0750); err != nil {
t.Fatal(err)
}
w := &pausedDownload{ResponseRecorder: httptest.NewRecorder(), started: make(chan struct{}), release: make(chan struct{})}
done := make(chan struct{})
go func() {
defer close(done)
a.workspace(w, runtimeRequest("GET", "/api/ws/tree?instance=one&path=missing", "", sid))
}()
defer func() { close(w.release); <-done }()
select {
case <-w.started:
case <-time.After(3 * time.Second):
t.Fatal("response not started")
}
if a.userLocks.held() {
t.Fatal("error response holds user write lock")
}
if a.instanceLocks.held() {
t.Fatal("error response holds instance lock")
}
a.readersMu.Lock()
leases := len(a.workspaceReaders)
a.readersMu.Unlock()
if leases != 0 {
t.Fatal("response retains lease")
}
}
// A restore or upgrade holds its instance and user locks for minutes; that
// must never block file operations or lifecycle actions of other users.
func TestLongMutationDoesNotBlockOtherUsers(t *testing.T) {
a, _ := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) {
switch {
case r.Method == "POST" && strings.HasSuffix(r.URL.Path, "/stop"):
w.WriteHeader(http.StatusNoContent)
case r.Method == "GET" && strings.HasSuffix(r.URL.Path, "/json"):
writeJSON(w, map[string]any{"Id": "cid-two", "State": map[string]any{"Status": "exited", "Running": false}})
default:
t.Errorf("unexpected Docker call %s %s", r.Method, r.URL.Path)
}
})
if err := a.s.CreateUser(store.User{ID: "other", Username: "other"}, "unused"); err != nil {
t.Fatal(err)
}
otherSid, _, err := a.s.Session("other")
if err != nil {
t.Fatal(err)
}
registerTestAccess(t, a, otherSid)
if err := a.s.AddInstance(store.Instance{ID: "one", UID: "owner"}); err != nil {
t.Fatal(err)
}
if err := a.s.AddInstance(store.Instance{ID: "two", UID: "other", ContainerID: "cid-two", Status: "running", Created: time.Now()}); err != nil {
t.Fatal(err)
}
for _, dir := range []string{"workspaces/owner/instances/one/home", "workspaces/other/instances/two/home"} {
if err := os.MkdirAll(filepath.Join(a.root, dir), 0750); err != nil {
t.Fatal(err)
}
}
// Simulate a mutation paused mid-flight (restore/upgrade state).
inst := a.instanceLocks.Acquire("one")
defer inst.Release()
user := a.userLocks.Acquire("owner")
defer user.Release()
treeDone := make(chan *httptest.ResponseRecorder, 1)
go func() {
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest("GET", "/api/ws/tree?instance=two", "", otherSid))
treeDone <- w
}()
select {
case w := <-treeDone:
if w.Code != 200 {
t.Fatalf("other user tree: %d %s", w.Code, w.Body.String())
}
case <-time.After(3 * time.Second):
t.Fatal("other user file operation blocked by long mutation")
}
stopDone := make(chan *httptest.ResponseRecorder, 1)
go func() {
w := httptest.NewRecorder()
a.instance(w, runtimeRequest("POST", "/api/instances/two/stop", "", otherSid))
stopDone <- w
}()
select {
case w := <-stopDone:
if w.Code != 200 {
t.Fatalf("other instance stop: %d %s", w.Code, w.Body.String())
}
case <-time.After(3 * time.Second):
t.Fatal("other instance lifecycle blocked by long mutation")
}
// The same user's mutation is still serialized by the user lock.
blocked := make(chan struct{})
go func() {
user := a.userLocks.Acquire("owner")
user.Release()
close(blocked)
}()
select {
case <-blocked:
t.Fatal("owner lock not held during mutation")
case <-time.After(300 * time.Millisecond):
}
}
internal/api/workspace_image.go
-117
View File
@@ -1,117 +0,0 @@
package api
import (
"bytes"
"encoding/xml"
"io"
"mime"
"net/http"
"os"
"path"
"strconv"
"strings"
"syscall"
)
const maxImageBytes = 16 << 20
func (a *API) wsImage(w http.ResponseWriter, r *http.Request, root *os.Root) {
if r.Method != http.MethodGet {
fail(w, 405, "method not allowed")
return
}
p := r.URL.Query().Get("path")
if p == "" || path.IsAbs(p) || p == "." || p == ".." || strings.HasPrefix(p, "../") || path.Clean(p) != p || strings.ContainsRune(p, 0) {
fail(w, 400, "invalid path")
return
}
anchor, err := root.Open(".")
if err != nil {
fail(w, 500, "storage unavailable")
return
}
defer anchor.Close()
f, err := openDownload(anchor, p)
if err != nil {
fail(w, 404, "image unavailable")
return
}
defer f.Close()
st, err := f.Stat()
if err != nil || !st.Mode().IsRegular() {
fail(w, 415, "unsupported image")
return
}
if st.Size() > maxImageBytes {
fail(w, 413, "image too large")
return
}
b, err := io.ReadAll(io.LimitReader(f, maxImageBytes+1))
if err != nil {
fail(w, 500, "image read failed")
return
}
if len(b) > maxImageBytes {
fail(w, 413, "image too large")
return
}
after, err := f.Stat()
if err != nil || int64(len(b)) != st.Size() || st.Size() != after.Size() || !st.ModTime().Equal(after.ModTime()) || st.Sys().(*syscall.Stat_t).Ctim != after.Sys().(*syscall.Stat_t).Ctim {
fail(w, 409, "image changed; reload")
return
}
mimeType := workspaceImageType(p, b)
if mimeType == "" {
fail(w, 415, "unsupported image")
return
}
w.Header().Set("Content-Type", mimeType)
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Content-Security-Policy", "sandbox; default-src 'none'; style-src 'unsafe-inline'")
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": path.Base(p)}))
w.Header().Set("Cache-Control", "no-store")
w.Header().Set("Content-Length", strconv.Itoa(len(b)))
w.Write(b)
}
func workspaceImageType(name string, b []byte) string {
switch mimeType := http.DetectContentType(b); mimeType {
case "image/png", "image/jpeg", "image/gif", "image/webp":
return mimeType
}
if !strings.EqualFold(path.Ext(name), ".svg") {
return ""
}
d := xml.NewDecoder(bytes.NewReader(bytes.TrimPrefix(b, []byte{0xef, 0xbb, 0xbf})))
depth, roots := 0, 0
for {
token, err := d.Token()
if err == io.EOF {
if roots == 1 && depth == 0 {
return "image/svg+xml"
}
return ""
}
if err != nil {
return ""
}
switch t := token.(type) {
case xml.StartElement:
if depth == 0 {
roots++
if roots != 1 || t.Name.Local != "svg" || t.Name.Space != "http://www.w3.org/2000/svg" {
return ""
}
}
depth++
case xml.EndElement:
depth--
case xml.CharData:
if depth == 0 && len(bytes.TrimSpace(t)) != 0 {
return ""
}
case xml.Directive:
return ""
}
}
}
internal/api/workspace_image_test.go
-129
View File
@@ -1,129 +0,0 @@
package api
import (
"bytes"
"mime"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path/filepath"
"strconv"
"syscall"
"testing"
"git.misaka.ren/m1saka/dsh-web/internal/store"
)
func TestWorkspaceImage(t *testing.T) {
a, sid := runtimeTestAPI(t, func(http.ResponseWriter, *http.Request) { t.Error("unexpected Docker call") })
for _, i := range []store.Instance{{ID: "one", UID: "owner"}, {ID: "foreign", UID: "other"}} {
if err := a.s.AddInstance(i); err != nil {
t.Fatal(err)
}
}
dir := filepath.Join(a.root, "workspaces/owner/instances/one/home")
if err := os.MkdirAll(dir, 0750); err != nil {
t.Fatal(err)
}
put := func(name string, b []byte) {
t.Helper()
if err := os.WriteFile(filepath.Join(dir, name), b, 0600); err != nil {
t.Fatal(err)
}
}
request := func(method, query, session string, want int) *httptest.ResponseRecorder {
t.Helper()
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest(method, "/api/ws/image?"+query, "", session))
if w.Code != want {
t.Fatalf("%s %s: got %d want %d: %s", method, query, w.Code, want, w.Body.String())
}
if len(a.workspaceReaders) != 0 {
t.Fatal("workspace lease leaked")
}
return w
}
get := func(name string, want int) *httptest.ResponseRecorder {
t.Helper()
return request("GET", "instance=one&path="+url.QueryEscape(name), sid, want)
}
request("GET", "instance=one&path=file", "", 401)
request("GET", "instance=foreign&path=file", sid, 404)
request("GET", "instance=missing&path=file", sid, 404)
request("GET", "path=file", sid, 410)
for _, method := range []string{"HEAD", "POST", "PUT", "DELETE", "PATCH"} {
request(method, "instance=one&path=file", sid, 405)
}
for _, p := range []string{"", ".", "..", "../file", "/etc/passwd", "a/../file", "./file", "a//file", "file/", "a/./file", "a\x00b"} {
get(p, 400)
}
get("missing", 404)
for _, tc := range []struct{ name, content, mime string }{
{"png.txt", "\x89PNG\r\n\x1a\n", "image/png"},
{"jpeg", "\xff\xd8\xff\xe0", "image/jpeg"},
{"gif87", "GIF87a", "image/gif"},
{"gif89", "GIF89a", "image/gif"},
{"webp", "RIFF\x00\x00\x00\x00WEBPVP8 ", "image/webp"},
{"active.svg", `<svg xmlns="http://www.w3.org/2000/svg" onload="alert(1)"><style>rect{fill:red}</style><script>alert(1)</script><foreignObject><html xmlns="http://www.w3.org/1999/xhtml"/></foreignObject></svg>`, "image/svg+xml"},
{"prefix.SVG", `<?xml version="1.0"?><!--ok--><s:svg xmlns:s="http://www.w3.org/2000/svg"/>`, "image/svg+xml"},
{" space ", "GIF89a exact", "image/gif"},
{`a\b`, "GIF89a backslash", "image/gif"},
} {
t.Run(tc.name, func(t *testing.T) {
put(tc.name, []byte(tc.content))
w := get(tc.name, 200)
if w.Body.String() != tc.content {
t.Fatal("raw image bytes changed")
}
disposition, params, err := mime.ParseMediaType(w.Header().Get("Content-Disposition"))
if err != nil || disposition != "attachment" || params["filename"] != tc.name {
t.Fatalf("invalid download filename: %s", w.Header().Get("Content-Disposition"))
}
for key, want := range map[string]string{"Content-Type": tc.mime, "Content-Length": strconv.Itoa(len(tc.content)), "Cache-Control": "no-store", "X-Content-Type-Options": "nosniff", "Content-Security-Policy": "sandbox; default-src 'none'; style-src 'unsafe-inline'"} {
if got := w.Header().Get(key); got != want {
t.Errorf("%s = %q, want %q", key, got, want)
}
}
})
}
for _, content := range []string{"", "plain text", "<html><svg/></html>", `<svg/>`, `<svg xmlns="wrong"/>`, `<svg xmlns="http://www.w3.org/2000/svg">`, `<svg xmlns="http://www.w3.org/2000/svg"/><html/>`, `<svg xmlns="http://www.w3.org/2000/svg"/>junk`, `<!DOCTYPE svg [<!ENTITY x SYSTEM "file:///etc/passwd">]><svg xmlns="http://www.w3.org/2000/svg">&x;</svg>`, `<svg xmlns="http://www.w3.org/2000/svg">&unknown;</svg>`} {
put("bad.svg", []byte(content))
get("bad.svg", 415)
}
put("wrong.png", []byte(`<svg xmlns="http://www.w3.org/2000/svg"/>`))
get("wrong.png", 415)
put("fake.png", []byte("<html>not an image</html>"))
get("fake.png", 415)
if err := os.Mkdir(filepath.Join(dir, "directory"), 0700); err != nil {
t.Fatal(err)
}
get("directory", 415)
if err := os.Symlink("png.txt", filepath.Join(dir, "link")); err != nil {
t.Fatal(err)
}
get("link", 404)
if err := os.Symlink(dir, filepath.Join(dir, "parent")); err != nil {
t.Fatal(err)
}
get("parent/png.txt", 404)
if err := os.Link(filepath.Join(dir, "png.txt"), filepath.Join(dir, "hard")); err != nil {
t.Fatal(err)
}
get("hard", 404)
get("png.txt", 404)
if err := syscall.Mkfifo(filepath.Join(dir, "fifo"), 0600); err != nil {
t.Fatal(err)
}
get("fifo", 404)
large := bytes.Repeat([]byte{0}, maxImageBytes)
copy(large, "GIF89a")
put("limit", large)
get("limit", 200)
put("oversize", append(large, 0))
get("oversize", 413)
if _, err := a.s.DB.Exec("UPDATE instances SET status='error' WHERE id='one'"); err != nil {
t.Fatal(err)
}
get("gif89", 200)
}
internal/api/workspace_instance_test.go
-70
View File
@@ -1,70 +0,0 @@
package api
import (
"git.misaka.ren/m1saka/dsh-web/internal/store"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
)
func TestWorkspaceInstanceScope(t *testing.T) {
a, sid := runtimeTestAPI(t, func(w http.ResponseWriter, r *http.Request) { t.Error("unexpected Docker call") })
for _, x := range []struct{ id, uid string }{{"one", "owner"}, {"two", "owner"}, {"foreign", "someone"}} {
if err := a.s.AddInstance(store.Instance{ID: x.id, UID: x.uid}); err != nil {
t.Fatal(err)
}
dir := filepath.Join(a.root, "workspaces", x.uid, "instances", x.id, "home")
if err := os.MkdirAll(dir, 0750); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, x.id+".txt"), []byte(x.id), 0600); err != nil {
t.Fatal(err)
}
}
for _, tc := range []struct {
query string
status int
contains string
}{
{"instance=one", 200, "one.txt"}, {"instance=two", 200, "two.txt"},
{"instance=foreign", 404, ""}, {"instance=missing", 404, ""},
{"instance=one&path=../two", 400, ""},
} {
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest("GET", "/api/ws/tree?"+tc.query, "", sid))
if w.Code != tc.status || tc.contains != "" && !strings.Contains(w.Body.String(), tc.contains) {
t.Fatalf("%s: %d %s", tc.query, w.Code, w.Body.String())
}
}
w := httptest.NewRecorder()
a.workspace(w, runtimeRequest("POST", "/api/ws/mkdir?instance=two", `{"path":"created"}`, sid))
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
if _, err := os.Stat(filepath.Join(a.root, "workspaces/owner/instances/two/home/created")); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(filepath.Join(a.root, "workspaces/owner/instances/one/home/created")); !os.IsNotExist(err) {
t.Fatal("cross instance write")
}
dir := filepath.Join(a.root, "workspaces/owner/instances/one/home")
if err := os.Symlink(t.TempDir(), filepath.Join(dir, "escape")); err != nil {
t.Fatal(err)
}
w = httptest.NewRecorder()
a.workspace(w, runtimeRequest("GET", "/api/ws/tree?instance=one&path=escape", "", sid))
if w.Code != 404 {
t.Fatal("symlink traversal allowed", w.Code)
}
if _, err := a.s.DB.Exec("UPDATE instances SET status='error' WHERE id='two'"); err != nil {
t.Fatal(err)
}
w = httptest.NewRecorder()
a.workspace(w, runtimeRequest("POST", "/api/ws/mkdir?instance=two", "{\"path\":\"blocked\"}", sid))
if w.Code != 409 {
t.Fatal("recovery write permitted", w.Code)
}
}
internal/backup/backup.go
-515
View File
@@ -1,515 +0,0 @@
// Package backup handles configuration archives. Callers must stop containers,
// exclude concurrent filesystem writers, and manage rollback across files.
package backup
import (
"archive/tar"
"bytes"
"compress/gzip"
"crypto/aes"
"crypto/cipher"
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path"
"path/filepath"
"sort"
"strings"
"time"
)
const (
MaxTotalBytes = 32 << 20
MaxFiles = 4096
maxArchiveBytes = 4*((MaxTotalBytes+2)/3) + 4096
magic = "DSHBACKUP\x01"
magicV2 = "DSHBACKUP\x02"
SchemaVersion = 2
maxManifestBytes = 4096
maxTarBytes = MaxTotalBytes + MaxFiles*4096 + maxManifestBytes
)
var allowed = [...]string{".credentials.yaml", "cordis.patch.yml", "profiles/web/package.json", "profiles/web/cordis.patch.yml", "profiles/web/pnpm-lock.yaml", "dsh-ssh.json"}
var resources = [...]string{"skins", "pets", "agent-presets", ".agent-presets"}
type Metadata struct {
DSHVersion string `json:"dsh_version"`
CreatedAt time.Time `json:"created_at"`
}
type manifest struct {
SchemaVersion int `json:"schema_version"`
Metadata
}
func allowedPath(name string, recursive bool) bool {
if name == "." || len(name) > 1024 || path.Clean(name) != name || strings.ContainsAny(name, "\\\x00:") || path.IsAbs(name) {
return false
}
for _, valid := range allowed {
if name == valid {
return true
}
}
if recursive {
for _, dir := range resources {
if strings.HasPrefix(name, dir+"/") {
return true
}
}
}
return false
}
type payload struct {
Version int `json:"version"`
Files map[string][]byte `json:"files"`
}
func newGCM(key []byte) (cipher.AEAD, error) {
b, err := aes.NewCipher(key)
if err != nil {
return nil, err
}
return cipher.NewGCM(b)
}
func openHome(home string) (*os.Root, error) {
info, err := os.Lstat(home)
if err != nil {
return nil, err
}
if info.Mode()&os.ModeSymlink != 0 {
return nil, errors.New("home must not be a symlink")
}
return os.OpenRoot(home)
}
// os.Root permits in-root symlinks, so reject them explicitly in each component.
func checkPath(root *os.Root, name string) error {
parts := strings.Split(name, "/")
for i := range parts {
info, err := root.Lstat(strings.Join(parts[:i+1], "/"))
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return err
}
if info.Mode()&os.ModeSymlink != 0 {
return fmt.Errorf("symlink rejected: %s", name)
}
if i < len(parts)-1 && !info.IsDir() {
return fmt.Errorf("not a directory: %s", name)
}
if i == len(parts)-1 && !info.Mode().IsRegular() {
return fmt.Errorf("not a regular file: %s", name)
}
}
return nil
}
func readBounded(f *os.File, limit int64) ([]byte, error) {
info, err := f.Stat()
if err != nil {
return nil, err
}
if !info.Mode().IsRegular() {
return nil, errors.New("not a regular file")
}
if info.Size() > limit {
return nil, errors.New("backup size limit exceeded")
}
b, err := io.ReadAll(io.LimitReader(f, limit+1))
if err != nil {
return nil, err
}
if int64(len(b)) > limit {
return nil, errors.New("backup size limit exceeded")
}
return b, nil
}
// Create writes a 0600 archive using a raw 16, 24, or 32 byte AES key.
// Missing allowlisted files are omitted; destination's parent must exist.
func Create(home, destination string, key []byte) error {
return CreateWithMetadata(home, destination, key, Metadata{})
}
// CreateWithMetadata writes an AES-GCM encrypted tar.gz with an authenticated
// manifest. Empty resource directories are omitted; callers must exclude writers.
func CreateWithMetadata(home, destination string, key []byte, metadata Metadata) error {
gcm, err := newGCM(key)
if err != nil {
return err
}
root, err := openHome(home)
if err != nil {
return err
}
defer root.Close()
p := payload{Version: SchemaVersion, Files: make(map[string][]byte)}
remaining := int64(MaxTotalBytes)
addFile := func(name string) error {
if !allowedPath(name, true) {
return fmt.Errorf("path not allowed: %q", name)
}
if err := checkPath(root, name); err != nil {
return err
}
f, err := root.Open(name)
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return err
}
data, readErr := readBounded(f, remaining)
closeErr := f.Close()
if readErr != nil {
return fmt.Errorf("%s: %w", name, readErr)
}
if closeErr != nil {
return closeErr
}
remaining -= int64(len(data))
p.Files[name] = data
if len(p.Files) > MaxFiles {
return errors.New("backup file count limit exceeded")
}
return nil
}
for _, name := range allowed {
if err := addFile(name); err != nil {
return err
}
}
visited := 0
var walk func(string) error
walk = func(name string) error {
visited++
if visited > MaxFiles*2 || len(name) > 1024 {
return errors.New("resource traversal limit exceeded")
}
info, err := root.Lstat(name)
if err != nil {
return err
}
if !info.IsDir() {
return addFile(name)
}
dir, err := root.Open(name)
if err != nil {
return err
}
defer dir.Close()
for {
entries, err := dir.ReadDir(1)
if err == io.EOF {
return nil
}
if err != nil {
return err
}
for _, entry := range entries {
if err := walk(name + "/" + entry.Name()); err != nil {
return err
}
}
}
}
for _, name := range resources {
info, err := root.Lstat(name)
if errors.Is(err, os.ErrNotExist) {
continue
}
if err != nil {
return err
}
if !info.IsDir() {
return fmt.Errorf("resource root is not a directory: %s", name)
}
if err := walk(name); err != nil {
return err
}
}
plain, err := encodeV2(p.Files, metadata)
if err != nil {
return err
}
nonce := make([]byte, gcm.NonceSize())
if _, err := rand.Read(nonce); err != nil {
return err
}
archive := append([]byte(magicV2), nonce...)
archive = gcm.Seal(archive, nonce, plain, []byte(magicV2))
if len(archive) > maxArchiveBytes {
return errors.New("backup archive size limit exceeded")
}
destRoot, err := os.OpenRoot(filepath.Dir(destination))
if err != nil {
return err
}
defer destRoot.Close()
return replace(destRoot, filepath.Base(destination), archive)
}
// Restore authenticates and validates everything before changing home. Files
// absent from the archive remain untouched. Replacement is atomic per file only.
func Restore(archive, home string, key []byte) error {
gcm, err := newGCM(key)
if err != nil {
return err
}
aroot, err := os.OpenRoot(filepath.Dir(archive))
if err != nil {
return err
}
defer aroot.Close()
if err := checkPath(aroot, filepath.Base(archive)); err != nil {
return err
}
f, err := aroot.Open(filepath.Base(archive))
if err != nil {
return err
}
data, readErr := readBounded(f, maxArchiveBytes)
closeErr := f.Close()
if readErr != nil {
return readErr
}
if closeErr != nil {
return closeErr
}
if len(data) < len(magic)+gcm.NonceSize()+gcm.Overhead() || (!bytes.HasPrefix(data, []byte(magic)) && !bytes.HasPrefix(data, []byte(magicV2))) {
return errors.New("invalid backup format")
}
nonce := data[len(magic) : len(magic)+gcm.NonceSize()]
header := string(data[:len(magic)])
plain, err := gcm.Open(nil, nonce, data[len(magic)+gcm.NonceSize():], []byte(header))
if err != nil {
return fmt.Errorf("authenticate backup: %w", err)
}
var p payload
if header == magicV2 {
p.Files, err = decodeV2(plain)
if err != nil {
return err
}
} else {
if err := decodeJSON(plain, &p); err != nil {
return err
}
if p.Version != 1 || p.Files == nil {
return errors.New("unsupported backup payload")
}
}
if len(p.Files) > MaxFiles {
return errors.New("backup file count limit exceeded")
}
total := 0
for name, data := range p.Files {
if !allowedPath(name, header == magicV2) {
return fmt.Errorf("path not allowed: %q", name)
}
for parent := path.Dir(name); parent != "."; parent = path.Dir(parent) {
if _, exists := p.Files[parent]; exists {
return errors.New("conflicting backup paths")
}
}
total += len(data)
if total > MaxTotalBytes {
return errors.New("backup size limit exceeded")
}
}
root, err := openHome(home)
if err != nil {
return err
}
defer root.Close()
for name := range p.Files {
if err := checkPath(root, name); err != nil {
return err
}
}
for name, data := range p.Files {
if err := root.MkdirAll(filepath.Dir(name), 0700); err != nil {
return err
}
if err := replace(root, name, data); err != nil {
return err
}
}
return nil
}
func decodeJSON(data []byte, value any) error {
d := json.NewDecoder(bytes.NewReader(data))
d.DisallowUnknownFields()
if err := d.Decode(value); err != nil {
return err
}
if err := d.Decode(new(any)); err != io.EOF {
return errors.New("trailing backup data")
}
return nil
}
func encodeV2(files map[string][]byte, metadata Metadata) ([]byte, error) {
if metadata.CreatedAt.IsZero() {
metadata.CreatedAt = time.Now().UTC()
}
if metadata.DSHVersion == "" {
metadata.DSHVersion = "unknown"
}
data, err := json.Marshal(manifest{SchemaVersion, metadata})
if err != nil {
return nil, err
}
if len(data) > maxManifestBytes {
return nil, errors.New("manifest size limit exceeded")
}
var out bytes.Buffer
gz := gzip.NewWriter(&out)
tw := tar.NewWriter(gz)
write := func(name string, data []byte) error {
if err := tw.WriteHeader(&tar.Header{Name: name, Mode: 0600, Size: int64(len(data)), Typeflag: tar.TypeReg, Format: tar.FormatPAX}); err != nil {
return err
}
_, err := tw.Write(data)
return err
}
if err := write("manifest.json", data); err != nil {
return nil, err
}
names := make([]string, 0, len(files))
for name := range files {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if err := write(name, files[name]); err != nil {
return nil, err
}
}
if err := tw.Close(); err != nil {
return nil, err
}
if err := gz.Close(); err != nil {
return nil, err
}
return out.Bytes(), nil
}
func decodeV2(plain []byte) (map[string][]byte, error) {
source := bytes.NewReader(plain)
gz, err := gzip.NewReader(source)
if err != nil {
return nil, err
}
defer gz.Close()
gz.Multistream(false)
unpacked, err := io.ReadAll(io.LimitReader(gz, maxTarBytes+1))
if err != nil {
return nil, err
}
if len(unpacked) > maxTarBytes {
return nil, errors.New("expanded archive size limit exceeded")
}
if source.Len() != 0 {
return nil, errors.New("trailing gzip data")
}
input := bytes.NewReader(unpacked)
tr := tar.NewReader(input)
files := make(map[string][]byte)
seenManifest := false
remaining := int64(MaxTotalBytes)
for {
h, err := tr.Next()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
if h.Typeflag != tar.TypeReg || h.Linkname != "" {
return nil, errors.New("non-regular archive entry")
}
limit := remaining
if h.Name == "manifest.json" {
if seenManifest {
return nil, errors.New("duplicate manifest")
}
limit = maxManifestBytes
} else {
if !allowedPath(h.Name, true) {
return nil, fmt.Errorf("path not allowed: %q", h.Name)
}
if _, exists := files[h.Name]; exists {
return nil, errors.New("duplicate archive entry")
}
if len(files) >= MaxFiles {
return nil, errors.New("backup file count limit exceeded")
}
}
if h.Size < 0 || h.Size > limit {
return nil, errors.New("backup size limit exceeded")
}
data, err := io.ReadAll(tr)
if err != nil {
return nil, err
}
if h.Name == "manifest.json" {
var m manifest
if err := decodeJSON(data, &m); err != nil {
return nil, err
}
if m.SchemaVersion != SchemaVersion || m.CreatedAt.IsZero() || m.DSHVersion == "" {
return nil, errors.New("invalid manifest")
}
seenManifest = true
} else {
files[h.Name] = data
remaining -= int64(len(data))
}
}
if !seenManifest || input.Len() != 0 {
return nil, errors.New("missing manifest or trailing tar data")
}
return files, nil
}
func replace(root *os.Root, name string, data []byte) error {
if err := checkPath(root, name); err != nil {
return err
}
var random [16]byte
if _, err := rand.Read(random[:]); err != nil {
return err
}
tmp := filepath.Join(filepath.Dir(name), ".backup-"+hex.EncodeToString(random[:]))
f, err := root.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600)
if err != nil {
return err
}
defer root.Remove(tmp)
_, writeErr := f.Write(data)
if writeErr == nil {
writeErr = f.Sync()
}
closeErr := f.Close()
if writeErr != nil {
return writeErr
}
if closeErr != nil {
return closeErr
}
if err := checkPath(root, name); err != nil {
return err
}
return root.Rename(tmp, name)
}
internal/backup/backup_test.go
-521
View File
@@ -1,521 +0,0 @@
package backup
import (
"archive/tar"
"bytes"
"compress/gzip"
"encoding/json"
"fmt"
"io"
"net"
"os"
"path/filepath"
"testing"
"time"
)
var testKey = bytes.Repeat([]byte{42}, 32)
func put(t *testing.T, name string, data []byte) {
t.Helper()
if err := os.MkdirAll(filepath.Dir(name), 0700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(name, data, 0600); err != nil {
t.Fatal(err)
}
}
func sealed(t *testing.T, p payload) string {
t.Helper()
plain, err := json.Marshal(p)
if err != nil {
t.Fatal(err)
}
return sealPlain(t, magic, plain)
}
func sealPlain(t *testing.T, header string, plain []byte) string {
t.Helper()
gcm, err := newGCM(testKey)
if err != nil {
t.Fatal(err)
}
nonce := make([]byte, gcm.NonceSize())
data := append([]byte(header), nonce...)
data = gcm.Seal(data, nonce, plain, []byte(header))
name := filepath.Join(t.TempDir(), "archive")
put(t, name, data)
return name
}
func TestResourcesAndManifest(t *testing.T) {
home, dest := t.TempDir(), t.TempDir()
for _, dir := range resources {
put(t, filepath.Join(home, dir, "nested/资源.bin"), []byte{0, 1, 255})
}
for _, name := range []string{"node_modules/pkg", "worktrees/x", "sessions/x", "logs/x", "manifest.json"} {
put(t, filepath.Join(home, name), []byte("excluded"))
}
metadata := Metadata{DSHVersion: "1.2.3", CreatedAt: time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)}
archive := filepath.Join(t.TempDir(), "archive")
if err := CreateWithMetadata(home, archive, testKey, metadata); err != nil {
t.Fatal(err)
}
data, err := os.ReadFile(archive)
if err != nil {
t.Fatal(err)
}
if !bytes.HasPrefix(data, []byte(magicV2)) {
t.Fatal("not v2")
}
gcm, _ := newGCM(testKey)
nonce := data[len(magicV2) : len(magicV2)+gcm.NonceSize()]
plain, err := gcm.Open(nil, nonce, data[len(magicV2)+gcm.NonceSize():], []byte(magicV2))
if err != nil {
t.Fatal(err)
}
gz, err := gzip.NewReader(bytes.NewReader(plain))
if err != nil {
t.Fatal(err)
}
defer gz.Close()
tr := tar.NewReader(gz)
h, err := tr.Next()
if err != nil || h.Name != "manifest.json" {
t.Fatal("missing manifest", err)
}
b, err := io.ReadAll(tr)
if err != nil {
t.Fatal(err)
}
var m manifest
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
if m.SchemaVersion != SchemaVersion || m.Metadata != metadata {
t.Fatalf("manifest: %+v", m)
}
if err := Restore(archive, dest, testKey); err != nil {
t.Fatal(err)
}
for _, dir := range resources {
b, err := os.ReadFile(filepath.Join(dest, dir, "nested/资源.bin"))
if err != nil || !bytes.Equal(b, []byte{0, 1, 255}) {
t.Fatal(dir, err, b)
}
}
if _, err := os.Stat(filepath.Join(dest, "manifest.json")); !os.IsNotExist(err) {
t.Fatal("manifest restored into home")
}
}
func TestLegacyRestore(t *testing.T) {
home := t.TempDir()
if err := Restore(sealed(t, payload{1, map[string][]byte{allowed[0]: []byte("legacy")}}), home, testKey); err != nil {
t.Fatal(err)
}
b, err := os.ReadFile(filepath.Join(home, allowed[0]))
if err != nil || string(b) != "legacy" {
t.Fatal(err, string(b))
}
}
type tarEntry struct {
name string
kind byte
data []byte
}
func makeTar(t *testing.T, entries []tarEntry) []byte {
t.Helper()
var b bytes.Buffer
gz := gzip.NewWriter(&b)
tw := tar.NewWriter(gz)
for _, e := range entries {
h := &tar.Header{Name: e.name, Typeflag: e.kind, Mode: 0600, Size: int64(len(e.data))}
if e.kind == tar.TypeSymlink || e.kind == tar.TypeLink {
h.Linkname = "target"
}
if err := tw.WriteHeader(h); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(e.data); err != nil {
t.Fatal(err)
}
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
if err := gz.Close(); err != nil {
t.Fatal(err)
}
return b.Bytes()
}
func TestV2RejectsInvalidArchivesBeforeWrites(t *testing.T) {
m, _ := json.Marshal(manifest{SchemaVersion, Metadata{"test", time.Now().UTC()}})
base := []tarEntry{{"manifest.json", tar.TypeReg, m}, {allowed[0], tar.TypeReg, []byte("new")}}
cases := map[string][]tarEntry{
"missing manifest": base[1:],
"duplicate manifest": append(append([]tarEntry{}, base...), base[0]),
"duplicate file": append(append([]tarEntry{}, base...), base[1]),
"invalid manifest": {{"manifest.json", tar.TypeReg, []byte(`{"schema_version":3}`)}},
"conflict": append(append([]tarEntry{}, base...), tarEntry{"pets/a", tar.TypeReg, nil}, tarEntry{"pets/a/b", tar.TypeReg, nil}),
}
for _, name := range []string{"pets/../escape", "pets//a", "pets/./a", "/pets/a", "pets/a/../../escape", "pets/a\\b", "pets/C:bad", "pets", "other/a"} {
cases[name] = append(append([]tarEntry{}, base...), tarEntry{name, tar.TypeReg, nil})
}
for _, kind := range []byte{tar.TypeSymlink, tar.TypeLink, tar.TypeFifo, tar.TypeChar, tar.TypeBlock, tar.TypeDir} {
cases[fmt.Sprint(kind)] = append(append([]tarEntry{}, base...), tarEntry{"pets/unsafe", kind, nil})
}
for name, entries := range cases {
t.Run(name, func(t *testing.T) {
home := t.TempDir()
put(t, filepath.Join(home, allowed[0]), []byte("old"))
if err := Restore(sealPlain(t, magicV2, makeTar(t, entries)), home, testKey); err == nil {
t.Fatal("accepted invalid archive")
}
b, _ := os.ReadFile(filepath.Join(home, allowed[0]))
if string(b) != "old" {
t.Fatal("partial write")
}
})
}
}
func TestResourceFilesystemRejections(t *testing.T) {
for _, name := range []string{"pets", "pets/nested", "pets/nested/file"} {
t.Run(name, func(t *testing.T) {
home := t.TempDir()
if err := os.MkdirAll(filepath.Dir(filepath.Join(home, name)), 0700); err != nil {
t.Fatal(err)
}
if err := os.Symlink(t.TempDir(), filepath.Join(home, name)); err != nil {
t.Fatal(err)
}
if err := Create(home, filepath.Join(t.TempDir(), "archive"), testKey); err == nil {
t.Fatal("followed resource symlink")
}
p, err := encodeV2(map[string][]byte{"pets/nested/file": []byte("bad")}, Metadata{})
if err != nil {
t.Fatal(err)
}
if err := Restore(sealPlain(t, magicV2, p), home, testKey); err == nil {
t.Fatal("restore followed symlink")
}
})
}
home := t.TempDir()
if err := os.Mkdir(filepath.Join(home, "pets"), 0700); err != nil {
t.Fatal(err)
}
l, err := net.Listen("unix", filepath.Join(home, "pets/socket"))
if err != nil {
t.Fatal(err)
}
defer l.Close()
if err := Create(home, filepath.Join(t.TempDir(), "archive"), testKey); err == nil {
t.Fatal("special file accepted")
}
}
func TestV2LimitsAndCorruption(t *testing.T) {
files := make(map[string][]byte)
home := t.TempDir()
for i := 0; i < MaxFiles; i++ {
name := fmt.Sprintf("pets/%d", i)
files[name] = nil
put(t, filepath.Join(home, name), nil)
}
archive := filepath.Join(t.TempDir(), "archive")
if err := Create(home, archive, testKey); err != nil {
t.Fatal(err)
}
if err := Restore(archive, t.TempDir(), testKey); err != nil {
t.Fatal(err)
}
put(t, filepath.Join(home, "pets/extra"), nil)
if err := Create(home, archive, testKey); err == nil {
t.Fatal("file count accepted")
}
files["pets/extra"] = nil
p, err := encodeV2(files, Metadata{})
if err != nil {
t.Fatal(err)
}
if _, err := decodeV2(p); err == nil {
t.Fatal("archive file count accepted")
}
p, err = encodeV2(map[string][]byte{"pets/large": make([]byte, MaxTotalBytes+1)}, Metadata{})
if err != nil {
t.Fatal(err)
}
if _, err := decodeV2(p); err == nil {
t.Fatal("expanded file limit accepted")
}
p, err = encodeV2(nil, Metadata{})
if err != nil {
t.Fatal(err)
}
badChecksum := bytes.Clone(p)
badChecksum[len(badChecksum)-8] ^= 1
for _, bad := range [][]byte{p[:len(p)-1], append(bytes.Clone(p), 0), append(bytes.Clone(p), p...), badChecksum} {
if _, err := decodeV2(bad); err == nil {
t.Fatal("corrupt gzip accepted")
}
}
if _, err := encodeV2(nil, Metadata{DSHVersion: string(make([]byte, maxManifestBytes))}); err == nil {
t.Fatal("oversized manifest accepted")
}
}
func TestRoundTrip(t *testing.T) {
home, dest := t.TempDir(), t.TempDir()
for _, name := range allowed {
put(t, filepath.Join(home, name), []byte("secret:"+name))
}
put(t, filepath.Join(home, "not-allowed"), []byte("ignore"))
archive := filepath.Join(t.TempDir(), "archive")
if err := Create(home, archive, testKey); err != nil {
t.Fatal(err)
}
first, err := os.ReadFile(archive)
if err != nil {
t.Fatal(err)
}
if bytes.Contains(first, []byte("secret:")) {
t.Fatal("plaintext leaked")
}
if err := Create(home, archive, testKey); err != nil {
t.Fatal(err)
}
second, err := os.ReadFile(archive)
if err != nil {
t.Fatal(err)
}
if bytes.Equal(first, second) {
t.Fatal("nonce reused")
}
put(t, filepath.Join(dest, allowed[0]), []byte("old"))
if err := os.Chmod(filepath.Join(dest, allowed[0]), 0644); err != nil {
t.Fatal(err)
}
if err := Restore(archive, dest, testKey); err != nil {
t.Fatal(err)
}
for _, name := range allowed {
data, err := os.ReadFile(filepath.Join(dest, name))
if err != nil {
t.Fatal(err)
}
if string(data) != "secret:"+name {
t.Fatalf("wrong data: %s", name)
}
info, err := os.Stat(filepath.Join(dest, name))
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm() != 0600 {
t.Fatalf("mode: %v", info.Mode())
}
}
if _, err := os.Stat(filepath.Join(dest, "not-allowed")); !os.IsNotExist(err) {
t.Fatal("unlisted file restored")
}
info, err := os.Stat(archive)
if err != nil {
t.Fatal(err)
}
if info.Mode().Perm() != 0600 {
t.Fatal("archive permissions")
}
}
func TestAuthentication(t *testing.T) {
archive := sealed(t, payload{1, map[string][]byte{allowed[0]: []byte("new")}})
original, err := os.ReadFile(archive)
if err != nil {
t.Fatal(err)
}
for _, offset := range []int{0, len(magic), len(original) - 1} {
t.Run(fmt.Sprint(offset), func(t *testing.T) {
data := bytes.Clone(original)
data[offset] ^= 1
put(t, archive, data)
home := t.TempDir()
put(t, filepath.Join(home, allowed[0]), []byte("old"))
if err := Restore(archive, home, testKey); err == nil {
t.Fatal("tampering accepted")
}
got, _ := os.ReadFile(filepath.Join(home, allowed[0]))
if string(got) != "old" {
t.Fatal("modified on failure")
}
})
}
put(t, archive, original)
if err := Restore(archive, t.TempDir(), bytes.Repeat([]byte{1}, 32)); err == nil {
t.Fatal("wrong key accepted")
}
put(t, archive, original[:len(original)-1])
if err := Restore(archive, t.TempDir(), testKey); err == nil {
t.Fatal("truncation accepted")
}
}
func TestInvalidPathsAndVersion(t *testing.T) {
for _, name := range []string{"../escape", "/tmp/escape", "profiles/web/../../../escape", "profiles//web/package.json", "other", "profiles\\web\\package.json"} {
t.Run(name, func(t *testing.T) {
home := t.TempDir()
put(t, filepath.Join(home, allowed[0]), []byte("old"))
archive := sealed(t, payload{1, map[string][]byte{allowed[0]: []byte("new"), name: []byte("bad")}})
if err := Restore(archive, home, testKey); err == nil {
t.Fatal("invalid path accepted")
}
got, _ := os.ReadFile(filepath.Join(home, allowed[0]))
if string(got) != "old" {
t.Fatal("partial restore before validation")
}
})
}
if err := Restore(sealed(t, payload{2, map[string][]byte{}}), t.TempDir(), testKey); err == nil {
t.Fatal("version accepted")
}
files := map[string][]byte{}
for i := 0; i < MaxFiles+1; i++ {
files[fmt.Sprint(i)] = nil
}
if err := Restore(sealed(t, payload{1, files}), t.TempDir(), testKey); err == nil {
t.Fatal("file count accepted")
}
}
func TestSymlinks(t *testing.T) {
for _, outside := range []bool{false, true} {
for _, parent := range []bool{false, true} {
t.Run(fmt.Sprintf("outside=%v/parent=%v", outside, parent), func(t *testing.T) {
home := t.TempDir()
target := filepath.Join(home, "target")
linkTarget := "target"
if outside {
target = filepath.Join(t.TempDir(), "target")
linkTarget = target
}
name := allowed[0]
link := filepath.Join(home, name)
if parent {
name = "profiles/web/package.json"
link = filepath.Join(home, "profiles")
put(t, filepath.Join(target, "web/package.json"), []byte("safe"))
} else {
put(t, target, []byte("safe"))
}
if err := os.Symlink(linkTarget, link); err != nil {
t.Fatal(err)
}
if err := Create(home, filepath.Join(t.TempDir(), "archive"), testKey); err == nil {
t.Fatal("create followed symlink")
}
archive := sealed(t, payload{1, map[string][]byte{name: []byte("bad")}})
if err := Restore(archive, home, testKey); err == nil {
t.Fatal("restore followed symlink")
}
})
}
}
}
func TestSymlinkPreflightAndArchive(t *testing.T) {
home := t.TempDir()
put(t, filepath.Join(home, allowed[0]), []byte("old"))
target := filepath.Join(t.TempDir(), "target")
put(t, target, []byte("safe"))
if err := os.Symlink(target, filepath.Join(home, "dsh-ssh.json")); err != nil {
t.Fatal(err)
}
archive := sealed(t, payload{1, map[string][]byte{allowed[0]: []byte("new"), "dsh-ssh.json": []byte("bad")}})
if err := Restore(archive, home, testKey); err == nil {
t.Fatal("symlink accepted")
}
got, err := os.ReadFile(filepath.Join(home, allowed[0]))
if err != nil {
t.Fatal(err)
}
if string(got) != "old" {
t.Fatal("preflight modified earlier file")
}
got, err = os.ReadFile(target)
if err != nil {
t.Fatal(err)
}
if string(got) != "safe" {
t.Fatal("outside target modified")
}
link := filepath.Join(t.TempDir(), "archive-link")
if err := os.Symlink(archive, link); err != nil {
t.Fatal(err)
}
if err := Restore(link, t.TempDir(), testKey); err == nil {
t.Fatal("archive symlink accepted")
}
if err := Create(t.TempDir(), link, testKey); err == nil {
t.Fatal("destination symlink accepted")
}
}
func TestSizeLimits(t *testing.T) {
home := t.TempDir()
name := filepath.Join(home, allowed[0])
put(t, name, nil)
if err := os.Truncate(name, MaxTotalBytes+1); err != nil {
t.Fatal(err)
}
archive := filepath.Join(t.TempDir(), "archive")
if err := Create(home, archive, testKey); err == nil {
t.Fatal("oversized file accepted")
}
if err := os.Truncate(name, MaxTotalBytes); err != nil {
t.Fatal(err)
}
if err := Create(home, archive, testKey); err != nil {
t.Fatal("exact limit:", err)
}
if err := Restore(archive, t.TempDir(), testKey); err != nil {
t.Fatal("exact limit restore:", err)
}
put(t, filepath.Join(home, allowed[1]), []byte{1})
if err := Create(home, archive, testKey); err == nil {
t.Fatal("aggregate limit accepted")
}
archive = sealed(t, payload{1, map[string][]byte{allowed[0]: make([]byte, MaxTotalBytes), allowed[1]: {1}}})
if err := Restore(archive, t.TempDir(), testKey); err == nil {
t.Fatal("restore aggregate limit accepted")
}
put(t, archive, nil)
if err := os.Truncate(archive, maxArchiveBytes+1); err != nil {
t.Fatal(err)
}
if err := Restore(archive, t.TempDir(), testKey); err == nil {
t.Fatal("oversized archive accepted")
}
}
func TestMissingFilesAndInvalidKey(t *testing.T) {
archive := filepath.Join(t.TempDir(), "archive")
if err := Create(t.TempDir(), archive, nil); err == nil {
t.Fatal("invalid key accepted")
}
if err := Create(t.TempDir(), archive, testKey); err != nil {
t.Fatal(err)
}
home := t.TempDir()
put(t, filepath.Join(home, allowed[0]), []byte("keep"))
if err := Restore(archive, home, testKey); err != nil {
t.Fatal(err)
}
got, _ := os.ReadFile(filepath.Join(home, allowed[0]))
if string(got) != "keep" {
t.Fatal("missing file removed")
}
}
internal/backup/password.go
-102
View File
@@ -1,102 +0,0 @@
package backup
import (
"bytes"
"crypto/rand"
"errors"
"os"
"path/filepath"
"syscall"
"golang.org/x/crypto/argon2"
)
const passwordMagic = "DSHPASS\x01"
const MaxEncryptedBytes int64 = int64(maxArchiveBytes + len(passwordMagic) + 16)
// The format fixes KDF costs; untrusted archives cannot request arbitrary memory.
func passwordKey(password string, salt []byte) ([]byte, error) {
if len(password) == 0 || len(password) > 1024 {
return nil, errors.New("backup password required (maximum 1024 bytes)")
}
return argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32), nil
}
func CreateWithPassword(home, destination, password string, metadata Metadata) error {
salt := make([]byte, 16)
if _, err := rand.Read(salt); err != nil {
return err
}
key, err := passwordKey(password, salt)
if err != nil {
return err
}
defer clear(key)
dir, err := os.MkdirTemp(filepath.Dir(destination), ".password-backup-")
if err != nil {
return err
}
defer os.RemoveAll(dir)
inner := filepath.Join(dir, "archive")
if err = CreateWithMetadata(home, inner, key, metadata); err != nil {
return err
}
data, err := os.ReadFile(inner)
if err != nil {
return err
}
data = append(append([]byte(passwordMagic), salt...), data...)
root, err := os.OpenRoot(filepath.Dir(destination))
if err != nil {
return err
}
defer root.Close()
return replace(root, filepath.Base(destination), data)
}
// Prepare authenticates and validates into a private staging home before the caller
// stops a container or snapshots its live home. Only ciphertext is retained.
func Prepare(archive, password string, legacyKey []byte) (restore func(string) error, cleanup func(), err error) {
f, err := os.OpenFile(archive, os.O_RDONLY|syscall.O_NOFOLLOW|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, nil, err
}
data, err := readBounded(f, MaxEncryptedBytes)
f.Close()
if err != nil {
return nil, nil, err
}
key := append([]byte(nil), legacyKey...)
if bytes.HasPrefix(data, []byte(passwordMagic)) {
if len(data) < len(passwordMagic)+16 {
return nil, nil, errors.New("truncated backup")
}
key, err = passwordKey(password, data[len(passwordMagic):len(passwordMagic)+16])
if err != nil {
return nil, nil, err
}
data = data[len(passwordMagic)+16:]
}
dir, err := os.MkdirTemp("", "dsh-backup-validate-")
if err != nil {
clear(key)
return nil, nil, err
}
cleanup = func() { clear(key); os.RemoveAll(dir) }
inner := filepath.Join(dir, "archive")
if err = os.WriteFile(inner, data, 0600); err == nil {
err = os.Mkdir(filepath.Join(dir, "home"), 0700)
}
if err == nil {
err = Restore(inner, filepath.Join(dir, "home"), key)
}
if err != nil {
cleanup()
return nil, nil, err
}
if err = os.RemoveAll(filepath.Join(dir, "home")); err != nil {
cleanup()
return nil, nil, err
}
return func(home string) error { return Restore(inner, home, key) }, cleanup, nil
}
internal/backup/password_test.go
-77
View File
@@ -1,77 +0,0 @@
package backup
import (
"bytes"
"os"
"path/filepath"
"testing"
)
func TestPasswordArchive(t *testing.T) {
home := t.TempDir()
dir := t.TempDir()
if err := os.WriteFile(filepath.Join(home, ".credentials.yaml"), []byte("secret"), 0600); err != nil {
t.Fatal(err)
}
first, second := filepath.Join(dir, "first"), filepath.Join(dir, "second")
for _, path := range []string{first, second} {
if err := CreateWithPassword(home, path, "old-password", Metadata{DSHVersion: "test"}); err != nil {
t.Fatal(err)
}
}
a, _ := os.ReadFile(first)
b, _ := os.ReadFile(second)
if bytes.Equal(a[:len(passwordMagic)+16], b[:len(passwordMagic)+16]) {
t.Fatal("reused salt")
}
if bytes.Contains(a, []byte("secret")) {
t.Fatal("plaintext")
}
if _, _, err := Prepare(first, "wrong-password", nil); err == nil {
t.Fatal("accepted wrong password")
}
restore, cleanup, err := Prepare(first, "old-password", nil)
if err != nil {
t.Fatal(err)
}
defer cleanup()
destination := t.TempDir()
if err := restore(destination); err != nil {
t.Fatal(err)
}
restored, _ := os.ReadFile(filepath.Join(destination, ".credentials.yaml"))
if string(restored) != "secret" {
t.Fatal(string(restored))
}
a[len(passwordMagic)] ^= 1
os.WriteFile(second, a, 0600)
if _, _, err := Prepare(second, "old-password", nil); err == nil {
t.Fatal("accepted altered salt")
}
legacy := filepath.Join(dir, "legacy")
key := bytes.Repeat([]byte{9}, 32)
if err := Create(home, legacy, key); err != nil {
t.Fatal(err)
}
restore, legacyCleanup, err := Prepare(legacy, "", key)
if err != nil {
t.Fatal(err)
}
defer legacyCleanup()
if err := restore(t.TempDir()); err != nil {
t.Fatal(err)
}
os.WriteFile(second, []byte(passwordMagic), 0600)
if _, _, err := Prepare(second, "old-password", nil); err == nil {
t.Fatal("accepted truncated header")
}
f, err := os.Create(second)
if err != nil {
t.Fatal(err)
}
f.Truncate(MaxEncryptedBytes + 1)
f.Close()
if _, _, err := Prepare(second, "old-password", nil); err == nil {
t.Fatal("accepted oversized file")
}
}
internal/docker/client.go
-247
View File
@@ -1,247 +0,0 @@
package docker
import (
"bufio"
"bytes"
"context"
"encoding/binary"
"encoding/json"
"fmt"
"io"
"net"
"net/http"
"net/url"
"strings"
)
const (
apiVersion = "/v1.47"
maxErrorBytes = 64 << 10
maxLogBytes = 1 << 20
)
type Client struct {
baseURL string
http *http.Client
}
type APIError struct {
StatusCode int
Message string
}
func (e *APIError) Error() string {
return fmt.Sprintf("docker API: HTTP %d: %s", e.StatusCode, e.Message)
}
func New(host string) *Client {
if host == "" {
host = "unix:///var/run/docker.sock"
}
transport := http.DefaultTransport.(*http.Transport).Clone()
if strings.HasPrefix(host, "unix://") {
socket := strings.TrimPrefix(host, "unix://")
transport.Proxy = nil
transport.DialContext = func(ctx context.Context, _, _ string) (net.Conn, error) {
return (&net.Dialer{}).DialContext(ctx, "unix", socket)
}
host = "http://docker"
} else if strings.HasPrefix(host, "tcp://") {
host = "http://" + strings.TrimPrefix(host, "tcp://")
}
return &Client{baseURL: strings.TrimRight(host, "/"), http: &http.Client{
Transport: transport,
CheckRedirect: func(_ *http.Request, _ []*http.Request) error { return http.ErrUseLastResponse },
}}
}
// Stream transfers ownership of a successful response body to the caller.
func (c *Client) Stream(ctx context.Context, method, path string, body io.Reader, contentType string) (*http.Response, error) {
if !strings.HasPrefix(path, "/") {
path = "/" + path
}
if path != apiVersion && !strings.HasPrefix(path, apiVersion+"/") && !strings.HasPrefix(path, apiVersion+"?") {
path = apiVersion + path
}
req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, body)
if err != nil {
return nil, err
}
if contentType != "" {
req.Header.Set("Content-Type", contentType)
}
resp, err := c.http.Do(req)
if err != nil {
return nil, err
}
if resp.StatusCode >= 200 && resp.StatusCode < 300 {
return resp, nil
}
defer resp.Body.Close()
data, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorBytes))
message := strings.TrimSpace(string(data))
var detail struct {
Message string `json:"message"`
}
if json.Unmarshal(data, &detail) == nil && detail.Message != "" {
message = detail.Message
}
if message == "" {
message = resp.Status
}
return nil, &APIError{StatusCode: resp.StatusCode, Message: message}
}
func (c *Client) Do(ctx context.Context, method, path string, body any, out any) error {
var input io.Reader
if body != nil {
data, err := json.Marshal(body)
if err != nil {
return err
}
input = bytes.NewReader(data)
}
resp, err := c.Stream(ctx, method, path, input, "application/json")
if err != nil {
return err
}
defer resp.Body.Close()
if out == nil || resp.StatusCode == http.StatusNoContent {
return nil
}
return json.NewDecoder(resp.Body).Decode(out)
}
type Container struct {
ID string `json:"Id"`
Name string
Image string
Config struct {
Image string
Tty bool
Labels map[string]string
Env []string
}
State struct {
Status string
Running bool
Paused bool
Restarting bool
ExitCode int
Error string
Health *struct{ Status string }
}
NetworkSettings struct {
IPAddress string
Networks map[string]struct {
IPAddress string
Gateway string
NetworkID string
}
}
}
func (c *Client) Inspect(ctx context.Context, id string) (*Container, error) {
var container Container
if err := c.Do(ctx, http.MethodGet, "/containers/"+url.PathEscape(id)+"/json", nil, &container); err != nil {
return nil, err
}
return &container, nil
}
func (c *Client) Logs(ctx context.Context, id string) (string, error) {
resp, err := c.Stream(ctx, http.MethodGet, "/containers/"+url.PathEscape(id)+"/logs?stdout=1&stderr=1&follow=0", nil, "")
if err != nil {
return "", err
}
defer resp.Body.Close()
reader := bufio.NewReader(resp.Body)
header, err := reader.Peek(8)
if err != nil && err != io.EOF {
return "", err
}
if len(header) == 0 || header[0] > 2 {
data, err := io.ReadAll(io.LimitReader(reader, maxLogBytes))
return string(data), err
}
var output bytes.Buffer
for output.Len() < maxLogBytes {
var frame [8]byte
_, err := io.ReadFull(reader, frame[:])
if err == io.EOF {
break
}
if err != nil {
return output.String(), fmt.Errorf("docker logs header: %w", err)
}
if frame[0] > 2 || frame[1] != 0 || frame[2] != 0 || frame[3] != 0 {
return output.String(), fmt.Errorf("docker logs: invalid stream header")
}
size := int64(binary.BigEndian.Uint32(frame[4:]))
remaining := int64(maxLogBytes - output.Len())
if size > remaining {
size = remaining
}
if _, err := io.CopyN(&output, reader, size); err != nil {
return output.String(), fmt.Errorf("docker logs payload: %w", err)
}
}
return output.String(), nil
}
func (c *Client) BuildRelease(ctx context.Context, tar io.Reader, version, tag, releaseID, repository, commit string, onLog func(string)) (string, error) {
labels := map[string]string{"dsh-mgr.release": releaseID, "dsh-mgr.source": repository, "dsh-mgr.commit": commit}
args, err := json.Marshal(map[string]string{"DSH_VERSION": version})
if err != nil {
return "", err
}
query := url.Values{"t": {tag}, "buildargs": {string(args)}, "rm": {"1"}}
data, _ := json.Marshal(labels)
query.Set("labels", string(data))
resp, err := c.Stream(ctx, http.MethodPost, "/build?"+query.Encode(), tar, "application/x-tar")
if err != nil {
return "", err
}
defer resp.Body.Close()
scanner := bufio.NewScanner(resp.Body)
scanner.Buffer(make([]byte, 4096), 1<<20)
var imageID string
for scanner.Scan() {
line := bytes.TrimSpace(scanner.Bytes())
if len(line) == 0 {
continue
}
var event struct {
Stream string `json:"stream"`
Error string `json:"error"`
ErrorDetail struct {
Message string `json:"message"`
} `json:"errorDetail"`
Aux struct {
ID string `json:"ID"`
} `json:"aux"`
}
if err := json.Unmarshal(line, &event); err != nil {
return "", fmt.Errorf("docker build event: %w", err)
}
if event.ErrorDetail.Message != "" {
return "", fmt.Errorf("docker build: %s", event.ErrorDetail.Message)
}
if event.Error != "" {
return "", fmt.Errorf("docker build: %s", event.Error)
}
if event.Stream != "" && onLog != nil {
onLog(event.Stream)
}
if event.Aux.ID != "" {
imageID = event.Aux.ID
}
}
if err := scanner.Err(); err != nil {
return "", fmt.Errorf("docker build stream: %w", err)
}
if imageID == "" {
return "", fmt.Errorf("docker build: response ended without image ID")
}
return imageID, nil
}
internal/docker/client_test.go
-242
View File
@@ -1,242 +0,0 @@
package docker
import (
"bytes"
"context"
"encoding/binary"
"encoding/json"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"path/filepath"
"strings"
"testing"
"time"
)
func testClient(t *testing.T, handler http.HandlerFunc) *Client {
t.Helper()
server := httptest.NewServer(handler)
t.Cleanup(server.Close)
client := New(strings.Replace(server.URL, "http://", "tcp://", 1))
t.Cleanup(client.http.CloseIdleConnections)
return client
}
func TestDo(t *testing.T) {
for _, path := range []string{"/containers/create?name=test", "containers/create?name=test", "/v1.47/containers/create?name=test"} {
t.Run(path, func(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != http.MethodPost || r.URL.RequestURI() != "/v1.47/containers/create?name=test" || r.Header.Get("Content-Type") != "application/json" {
t.Errorf("unexpected request: %s %s %v", r.Method, r.URL, r.Header)
}
var body map[string]string
if err := json.NewDecoder(r.Body).Decode(&body); err != nil || body["Image"] != "test" {
t.Errorf("body = %v, err = %v", body, err)
}
w.WriteHeader(http.StatusCreated)
io.WriteString(w, `{"Id":"container"}`)
})
var out struct{ ID string }
if err := client.Do(context.Background(), http.MethodPost, path, map[string]string{"Image": "test"}, &out); err != nil || out.ID != "container" {
t.Fatalf("out = %v, err = %v", out, err)
}
})
}
}
func TestDoNoContentAndInvalidJSON(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusNoContent) })
var out any
if err := client.Do(context.Background(), "POST", "/stop", nil, &out); err != nil {
t.Fatal(err)
}
if err := client.Do(context.Background(), "POST", "/stop", make(chan int), nil); err == nil {
t.Fatal("expected marshal error")
}
client = testClient(t, func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "invalid") })
if err := client.Do(context.Background(), "GET", "/test", nil, &out); err == nil {
t.Fatal("expected JSON decode error")
}
}
func TestAPIErrors(t *testing.T) {
for _, tc := range []struct {
name, body, want string
status int
}{
{"json", `{"message":"not found"}`, "not found", 404},
{"text", "unavailable", "unavailable", 503},
{"empty", "", "500 Internal Server Error", 500},
{"bounded", strings.Repeat("x", maxErrorBytes*2), strings.Repeat("x", maxErrorBytes), 500},
{"redirect", "no redirect", "no redirect", 302},
} {
t.Run(tc.name, func(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Location", "/elsewhere")
w.WriteHeader(tc.status)
io.WriteString(w, tc.body)
})
err := client.Do(context.Background(), "GET", "/test", nil, nil)
var apiErr *APIError
if !errors.As(err, &apiErr) || apiErr.StatusCode != tc.status || apiErr.Message != tc.want {
t.Fatalf("unexpected API error: %v", err)
}
if !strings.Contains(err.Error(), http.StatusText(tc.status)) && tc.name == "empty" {
t.Fatal(err)
}
})
}
}
func TestUnixAndCancellation(t *testing.T) {
socket := filepath.Join(t.TempDir(), "docker.sock")
listener, err := net.Listen("unix", socket)
if err != nil {
t.Fatal(err)
}
server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1.47/info" {
t.Error(r.URL.Path)
}
io.WriteString(w, `{"ok":true}`)
}))
server.Listener.Close()
server.Listener = listener
server.Start()
defer server.Close()
client := New("unix://" + socket)
defer client.http.CloseIdleConnections()
var out map[string]bool
if err := client.Do(context.Background(), "GET", "/info", nil, &out); err != nil || !out["ok"] {
t.Fatalf("%v %v", out, err)
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
transport := client.http.Transport.(*http.Transport)
if conn, err := transport.DialContext(ctx, "tcp", "ignored"); err == nil {
conn.Close()
t.Fatal("unix dial ignored cancellation")
}
if err := client.Do(ctx, "GET", "/info", nil, nil); !errors.Is(err, context.Canceled) {
t.Fatalf("canceled request: %v", err)
}
}
func TestInspect(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.EscapedPath() != "/v1.47/containers/a%2Fb/json" {
t.Error(r.URL.EscapedPath())
}
io.WriteString(w, `{"Id":"abc","Name":"/example","Config":{"Tty":true},"State":{"Running":true,"Status":"running","Health":{"Status":"healthy"}},"NetworkSettings":{"Networks":{"bridge":{"IPAddress":"172.18.0.2"}}}}`)
})
container, err := client.Inspect(context.Background(), "a/b")
if err != nil {
t.Fatal(err)
}
if container.ID != "abc" || !container.State.Running || !container.Config.Tty || container.State.Health.Status != "healthy" || container.NetworkSettings.Networks["bridge"].IPAddress != "172.18.0.2" {
t.Fatalf("%+v", container)
}
}
func frame(stream byte, text string) []byte {
data := make([]byte, 8, 8+len(text))
data[0] = stream
binary.BigEndian.PutUint32(data[4:], uint32(len(text)))
return append(data, text...)
}
func TestLogs(t *testing.T) {
for _, tc := range []struct {
name string
data []byte
want string
fail bool
}{
{"multiplexed", append(frame(1, "out\n"), frame(2, "err\n")...), "out\nerr\n", false},
{"tty", []byte("plain text\n"), "plain text\n", false},
{"short tty", []byte("ok"), "ok", false},
{"empty", nil, "", false},
{"empty frame", append(frame(1, ""), frame(2, "ok")...), "ok", false},
{"limited frame", frame(1, strings.Repeat("x", maxLogBytes+10)), strings.Repeat("x", maxLogBytes), false},
{"limited tty", bytes.Repeat([]byte("x"), maxLogBytes+10), strings.Repeat("x", maxLogBytes), false},
{"limited multiple frames", append(frame(1, strings.Repeat("x", maxLogBytes-1)), frame(2, "yz")...), strings.Repeat("x", maxLogBytes-1) + "y", false},
{"short header", []byte{1, 0}, "", true},
{"short payload", frame(1, "test")[:10], "te", true},
{"invalid header", []byte{1, 1, 0, 0, 0, 0, 0, 0}, "", true},
} {
t.Run(tc.name, func(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/v1.47/containers/example/logs" || r.URL.Query().Get("stdout") != "1" || r.URL.Query().Get("stderr") != "1" || r.URL.Query().Get("follow") != "0" {
t.Error(r.URL)
}
w.Write(tc.data)
})
got, err := client.Logs(context.Background(), "example")
if (err != nil) != tc.fail || got != tc.want {
t.Fatalf("got %d bytes want %d, err=%v", len(got), len(tc.want), err)
}
})
}
}
func TestBuild(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) {
if r.Method != "POST" || r.URL.Path != "/v1.47/build" || r.Header.Get("Content-Type") != "application/x-tar" || r.URL.Query().Get("t") != "test:1" {
t.Errorf("%s %s %v", r.Method, r.URL, r.Header)
}
var args map[string]string
if err := json.Unmarshal([]byte(r.URL.Query().Get("buildargs")), &args); err != nil || args["DSH_VERSION"] != "1.2.3" {
t.Errorf("%v %v", args, err)
}
body, _ := io.ReadAll(r.Body)
if string(body) != "caller tar bytes" {
t.Errorf("tar changed: %q", body)
}
for _, chunk := range []string{`{"stream":"step 1\n"}`, "\r\n\n", `{"au`, `x":{"ID":"sha256:image"}}`} {
io.WriteString(w, chunk)
w.(http.Flusher).Flush()
}
})
var logs []string
id, err := client.BuildRelease(context.Background(), strings.NewReader("caller tar bytes"), "1.2.3", "test:1", "release", "repository", "commit", func(s string) { logs = append(logs, s) })
if err != nil || id != "sha256:image" || len(logs) != 1 || logs[0] != "step 1\n" {
t.Fatalf("id=%q logs=%v err=%v", id, logs, err)
}
}
func TestBuildErrors(t *testing.T) {
for _, tc := range []struct{ name, data, want string }{
{"error", `{"error":"build failed"}`, "build failed"},
{"detail", `{"errorDetail":{"message":"detailed failure"}}`, "detailed failure"},
{"both", `{"error":"generic","errorDetail":{"message":"specific"}}`, "specific"},
{"late error", "{\"aux\":{\"ID\":\"partial\"}}\n{\"error\":\"late failure\"}", "late failure"},
{"malformed", `{"stream":`, "docker build event"},
{"too large", strings.Repeat("x", (1<<20)+1), "token too long"},
{"missing ID", `{"stream":"done"}`, "without image ID"},
} {
t.Run(tc.name, func(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, tc.data) })
id, err := client.BuildRelease(context.Background(), nil, "1", "test", "release", "repository", "commit", nil)
if err == nil || !strings.Contains(err.Error(), tc.want) || id != "" {
t.Fatalf("id=%q err=%v", id, err)
}
})
}
}
func TestBuildCancellation(t *testing.T) {
client := testClient(t, func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "{\"stream\":\"started\"}\n")
w.(http.Flusher).Flush()
<-r.Context().Done()
})
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
_, err := client.BuildRelease(ctx, nil, "1", "test", "release", "repository", "commit", func(string) { cancel() })
if !errors.Is(err, context.Canceled) {
t.Fatalf("err=%v", err)
}
}
internal/store/intents.go
-39
View File
@@ -1,39 +0,0 @@
package store
import (
"database/sql"
"errors"
"time"
)
var ErrInstanceLimit = errors.New("instance limit reached")
func reserveSlot(tx *sql.Tx, uid string) error {
var count int
if err := tx.QueryRow(`SELECT count(*) FROM instances WHERE uid=?`, uid).Scan(&count); err != nil {
return err
}
if count >= 2 {
return ErrInstanceLimit
}
return nil
}
func (s *Store) ReserveInstance(i Instance) error {
tx, err := s.DB.Begin()
if err != nil {
return err
}
defer tx.Rollback()
if err = reserveSlot(tx, i.UID); err != nil {
return err
}
if err = insertInstance(tx, i); err != nil {
return err
}
return tx.Commit()
}
func insertInstance(tx *sql.Tx, i Instance) error {
_, err := tx.Exec(`INSERT INTO instances VALUES(?,?,?,?,?,?,?,?)`, i.ID, i.UID, i.Name, i.Version, i.ImageID, i.Status, i.ContainerID, time.Now().Unix())
return err
}

Some files were not shown because too many files have changed in this diff Show More