add chunked upload to bypass 100MB proxy body limit
Resumable /api/uploads flow (init -> append -> complete -> abort) so a large recording uploads as many small slices (<= ASR_CHUNK_SIZE_MB, default 50MB) instead of one request a size-capping proxy (Cloudflare's 100MB) would reject. The web console auto-routes files larger than one slice through the chunked path with cross-slice progress and transient-error retry; smaller files keep the single-shot upload. - store: in-memory upload sessions + .part reassembly (offset writes, out-of-order/idempotent), reuse upload id as task id, orphan sweep on start, idle-session GC - fix limitedReader rejecting bodies exactly at the cap (broke full chunks) - fix drag-drop double-firing when a drop lands on the dropzone (its drop handler now stopPropagation so the window listener doesn't re-upload) - tests: store + HTTP flow, plus a 120MB reassembly test (ASR_TEST_LARGE)
This commit is contained in:
@@ -129,6 +129,7 @@ SenseVoice + ~2 MB silero VAD); subsequent starts load them from disk.
|
||||
| `ASR_USE_ITN` | `1` | Inverse text normalization (numbers, punctuation). |
|
||||
| `ASR_MAX_RETRIES` | `3` | Auto-retry attempts before a task is marked failed. |
|
||||
| `ASR_MAX_UPLOAD_MB` | `512` | Per-upload size cap. |
|
||||
| `ASR_CHUNK_SIZE_MB` | `50` | Slice size for resumable chunked uploads; kept under CDN body limits (e.g. Cloudflare's 100MB). |
|
||||
| `ASR_WORKERS` | `1` | Concurrent transcription workers. |
|
||||
| `ASR_VAD_THRESHOLD` | `0.5` | Silero speech probability threshold. |
|
||||
| `ASR_VAD_MIN_SPEECH` | `0.5` | Min speech segment (s). |
|
||||
@@ -194,6 +195,43 @@ curl -H "X-Api-Key: changeme" http://localhost:8080/api/tasks/<id>
|
||||
curl -H "X-Api-Key: changeme" http://localhost:8080/api/tasks/<id>/transcript
|
||||
```
|
||||
|
||||
### Chunked uploads (bypassing a 100MB proxy limit)
|
||||
|
||||
When the service sits behind a size-capping proxy — Cloudflare's free plan
|
||||
rejects request bodies over **100MB** — a single `POST /api/tasks` upload of a
|
||||
large recording never reaches the origin. The resumable chunked-upload API
|
||||
splits the file into slices no bigger than `ASR_CHUNK_SIZE_MB` (default **50MB**,
|
||||
well under the cap) and reassembles them server-side into one task:
|
||||
|
||||
| Method | Path | Description |
|
||||
|---|---|---|
|
||||
| `POST` | `/api/uploads` | Open a session: JSON `{filename,size}` → `{upload_id, chunk_size, total_chunks}`. |
|
||||
| `POST` | `/api/uploads/{id}?index=N` | Send slice `N` (0-based, raw body ≤ chunk_size). Order is free; resending is idempotent. |
|
||||
| `POST` | `/api/uploads/{id}/complete` | Once all slices arrive: merge → enqueue → `202` + task (same shape as `/api/tasks`). |
|
||||
| `DELETE` | `/api/uploads/{id}` | Abort the session and drop the partial file → `204`. |
|
||||
|
||||
```sh
|
||||
# Open a session for a 180MB file (4 slices of 50MB)
|
||||
UID=$(curl -sH "X-Api-Key: $PW" -H "Content-Type: application/json" \
|
||||
-d '{"filename":"big.mp4","size":188743680}' \
|
||||
http://localhost:8080/api/uploads | jq -r .upload_id)
|
||||
|
||||
# Send each slice (split with dd/split beforehand)
|
||||
for i in 0 1 2 3; do
|
||||
curl -sH "X-Api-Key: $PW" --data-binary @"slice_$i.bin" \
|
||||
"http://localhost:8080/api/uploads/$UID?index=$i" >/dev/null
|
||||
done
|
||||
|
||||
# Merge → enqueue (returns the task, just like /api/tasks)
|
||||
curl -H "X-Api-Key: $PW" -X POST http://localhost:8080/api/uploads/$UID/complete
|
||||
```
|
||||
|
||||
The web console handles this automatically: files larger than one chunk go
|
||||
through the chunked path (with a single progress bar across all slices);
|
||||
smaller files keep the simple single-request upload. Sessions live in memory,
|
||||
so an interrupted upload is retried from scratch on reload — and any orphaned
|
||||
partial files are swept on the next start.
|
||||
|
||||
## Task lifecycle & error handling
|
||||
|
||||
States: `pending` → `processing` → `completed` | `failed` | `skipped`.
|
||||
|
||||
Reference in New Issue
Block a user