StreamKeep

Storage efficiency

Broadcasts are multi-hour source-quality video. Retaining them on compute-attached disk would mean growing the instance for storage rather than for load, and paying for provisioned capacity whether or not it is used. Storage had to be billed by consumption and decoupled from the application host.

Delivery must not touch the application

If video is streamed through the API, every concurrent viewer consumes an application connection and host bandwidth. A single popular archive would then degrade sign-in, browsing and job submission for everyone. Read traffic had to be served by infrastructure that scales independently of the API.

Long-running work cannot run in a request

Fetching and storing a multi-hour broadcast takes far longer than any acceptable HTTP timeout, and it is subject to upstream failures outside my control. This work had to be asynchronous, durable across restarts, and safely retryable.

Cost has to stay predictable

The project is funded by its users on hours-based plans. A cost model that grows with viewership rather than with stored hours would break that relationship, so the unit customers pay for and the unit I am billed for had to match.

Component responsibilities

• Next.js frontend — the three public surfaces: stream library, live multiview and channel statistics. Bilingual, with the language cookie read server-side so the first paint already matches the reader.
• Go API — authentication through Kick OAuth 2.0 with PKCE, authorisation, archive submission, and the live recorder that polls the public directory every 60 seconds. It serves metadata only; no video passes through it.
• PostgreSQL — the system of record: archives and their state, chat transcripts, channel statistics, and the migration ledger. Job status and failure reasons live here, which makes a broken archive a row I can query rather than a log line I have to find.
• Redis — the job broker and coordination layer, running Asynq. It holds queued and in-flight tasks, live per-archive progress snapshots, and rate-limiting counters.
• Archive worker — a separate process that consumes the queue and performs the long-running work: fetching the source rendition with yt-dlp and ffmpeg, writing it to object storage, and capturing the chat transcript.
• Cloudflare R2 — object storage for the original video files. Chosen for consumption-based pricing and, critically, zero egress charges.
• Cloudflare CDN — delivery on a dedicated domain, in front of R2. Viewers stream HLS from here; the API is never in the path of a byte of video.

Why this shape

An orchestrated cluster was the obvious alternative and I decided against it. At this workload it would add a control plane to patch, upgrade and debug, and it would buy elasticity the traffic does not need. The relevant question was not which architecture is most scalable, but which one a single engineer can operate correctly at 3am.

• Operational simplicity — one host to patch, one file to read, no scheduler to reason about during an incident.
• Cost — a small VM plus consumption-priced storage, with no per-cluster or per-node overhead.
• Maintainability — the whole runtime is described in one Compose file that is versioned with the application.
• Sufficient capacity — the API is metadata-only and video is served by the CDN, so the host is not on the critical path for the traffic that actually scales.

The trade-off I accepted

One host and one database mean a host failure is downtime, not a failover, and a schema migration briefly pauses the worker. Both are acceptable at this stage and both are stated plainly rather than engineered around prematurely. The way out, once the workload needs it, is a second node under an orchestrator, which turns a host failure into a failover.

The pattern being avoided

The naive design routes playback through the application: viewer to API server to storage. It is simple to implement and it fails badly. Every concurrent viewer holds an application connection for the duration of a multi-hour video, host bandwidth becomes the ceiling on audience size, and one popular archive degrades sign-in and job submission for every other user. The blast radius of a traffic spike is the entire product.

The pattern used

Playback goes viewer to CDN to object storage, on a dedicated domain. The API issues metadata and never touches a byte of video. HLS segments are requested directly from the edge by the player.

• Application load is bounded — API concurrency tracks metadata requests, not viewer-hours.
• Bandwidth cost is bounded — R2 charges no egress, so a broadcast watched a thousand times costs the same to serve as one watched once.
• Delivery scales independently — audience growth is absorbed by the CDN, with no change to the host.
• Failure is isolated — a delivery problem does not take down archiving, and an application deploy does not interrupt playback.

One necessary exception

For live multiview, the master playlist is proxied, because that single file is the one whose CORS policy is restricted to the platform’s own origins. Variant playlists and video segments still stream directly from the upstream CDN. The proxy is scoped to the smallest object that requires it rather than to the whole stream.

Large media is a storage problem, not an application problem

The instinct is to treat video as data the application owns. Treating it as an object the application only references, and keeping it entirely off the request path, removed the majority of the scaling and cost questions before they became problems.

Reliable workers need durable state, not just a queue

A queue alone does not survive contact with retries. Keeping the authoritative status and attempt count in the database, scoping task identity by attempt, and guarding each sub-job with its own claim is what makes a re-queued job safe instead of merely possible.

Most incidents come from the deploy

Adding a verified backup, a strict rollout order and three independent health gates cost an afternoon and removed the failure mode I was most likely to cause myself. The gates that matter are the ones that check the real network path, not just the process.

Simplicity is a capacity decision, not a shortcut

Choosing Compose over an orchestrator was a judgement about what one engineer can operate, not an admission of missing skill. The corresponding obligation is to know exactly which signal would invalidate that choice, and to have the migration path ready before it does.

Design the cost model with the architecture

Selecting storage with no egress charge was an architectural decision as much as a financial one. It is what allows viewing to be free and unauthenticated without the economics inverting as the audience grows.