StreamKeep

Storage efficiency

Broadcasts are multi-hour source-quality video. Retaining them on compute-attached disk would mean growing the instance for storage rather than for load, and paying for provisioned capacity whether or not it is used. Storage had to be billed by consumption and decoupled from the application host.

Delivery must not touch the application

If video is streamed through the API, every concurrent viewer consumes an application connection and host bandwidth. A single popular archive would then degrade sign-in, browsing and job submission for everyone. Read traffic had to be served by infrastructure that scales independently of the API.

Long-running work cannot run in a request

Fetching and storing a multi-hour broadcast takes far longer than any acceptable HTTP timeout, and it is subject to upstream failures outside my control. This work had to be asynchronous, durable across restarts, and safely retryable.

Cost has to stay predictable

The project is funded by its users on hours-based plans. A cost model that grows with viewership rather than with stored hours would break that relationship, so the unit customers pay for and the unit I am billed for had to match.

Component responsibilities

• Next.js frontend — the three public surfaces: stream library, live multiview and channel statistics. Bilingual, with the language cookie read server-side so the first paint already matches the reader.
• Go API — authentication through Kick OAuth 2.0 with PKCE, authorisation, archive submission, and the live recorder that polls the public directory every 60 seconds. It serves metadata only; no video passes through it.
• PostgreSQL — the system of record: archives and their state, chat transcripts, channel statistics, and the migration ledger. Job status and failure reasons live here, which makes a broken archive a row I can query rather than a log line I have to find.
• Redis — the job broker and coordination layer, running Asynq. It holds queued and in-flight tasks, live per-archive progress snapshots, and rate-limiting counters.
• Archive worker — a separate process that consumes the queue and performs the long-running work: fetching the source rendition with yt-dlp and ffmpeg, writing it to object storage, and capturing the chat transcript.
• Cloudflare R2 — object storage for the original video files. Chosen for consumption-based pricing and, critically, zero egress charges.
• Cloudflare CDN — delivery on a dedicated domain, in front of R2. Viewers stream HLS from here; the API is never in the path of a byte of video.

Why this shape

An orchestrated cluster was the obvious alternative and I decided against it. At this workload it would add a control plane to patch, upgrade and debug, and it would buy elasticity the traffic does not need. The relevant question was not which architecture is most scalable, but which one a single engineer can operate correctly at 3am.

• Operational simplicity — one host to patch, one file to read, no scheduler to reason about during an incident.
• Cost — a small VM plus consumption-priced storage, with no per-cluster or per-node overhead.
• Maintainability — the whole runtime is described in one Compose file that is versioned with the application.
• Sufficient capacity — the API is metadata-only and video is served by the CDN, so the host is not on the critical path for the traffic that actually scales.

The trade-off I accepted

One host and one database mean a host failure is downtime, not a failover, and a schema migration briefly pauses the worker. Both are acceptable at this stage and both are stated plainly rather than engineered around prematurely. The migration path out is described in section 11.

Deployment safety

The pipeline runs only on the default branch, on a self-hosted runner on the production host. A resource group and a file lock together make it impossible for two deployments to overlap, and the job is marked non-interruptible so a newer pipeline cannot terminate a release mid-rollout.

• Backup first — every deployment takes a compressed pg_dump before it touches any running container.
• Validate the backup — the dump is checked for content and the deployment aborts if it comes back empty. An unverified backup is not a backup.
• Store it out of reach — dumps are written outside the CI checkout so the next pipeline’s clean checkout cannot delete them.
• Record the release — the commit and pipeline identity tag the images, and the last successful release is recorded on the host.

Migration safety

Schema migrations are embedded in the API binary, applied in filename order, and tracked in a migrations table so each runs exactly once. Sixteen have been applied so far.

The ordering of the rollout is the important part. The old worker is stopped before the new API starts, so no process is executing against the old schema while a migration is in flight. Only once the API has started and migrated do the worker and frontend come up. This removes the class of race where a worker writes a row shaped for a schema that no longer exists.

Health checks

A release passes three independent gates, in order, and failing any one of them fails the pipeline:

• Container health — the orchestrator waits on each service’s own health check with a bounded timeout before proceeding.
• Internal readiness — the API is probed on its readiness endpoint from inside the network, which confirms it reached a serving state rather than merely starting.
• External verification — the public API health URL and the site itself must both return HTTP 200 over the real network path, through DNS, TLS and the reverse proxy.

Background processing

Archive jobs run in a separate process from the API for two reasons: their duration has no relationship to a request lifecycle, and their failure modes are dominated by an upstream I do not control. Isolating them means a stalled download cannot consume an API worker or affect page latency.

• Durable queue — jobs are held in Redis through Asynq, so a worker restart does not lose queued work.
• State in Postgres — every archive carries a status and an attempt counter in the database, incremented atomically. Redis holds the transient progress snapshot; the durable truth is relational.
• Retry without duplication — the task identifier is scoped by attempt number, so a retry is never mistaken for a duplicate of the original request, and a re-queued job cannot create a second copy of the same object.
• Independent sub-jobs — chat sync, metadata enrichment and storyboard generation are queued separately from the video fetch, each with its own claim guarded by attempt number, so partial failure degrades one asset instead of the whole archive.
• Upstream backoff — chat pagination applies exponential backoff when the platform throttles, rather than discarding pages already fetched.

The pattern being avoided

The naive design routes playback through the application: viewer to API server to storage. It is simple to implement and it fails badly. Every concurrent viewer holds an application connection for the duration of a multi-hour video, host bandwidth becomes the ceiling on audience size, and one popular archive degrades sign-in and job submission for every other user. The blast radius of a traffic spike is the entire product.

The pattern used

Playback goes viewer to CDN to object storage, on a dedicated domain. The API issues metadata and never touches a byte of video. HLS segments are requested directly from the edge by the player.

• Application load is bounded — API concurrency tracks metadata requests, not viewer-hours.
• Bandwidth cost is bounded — R2 charges no egress, so a broadcast watched a thousand times costs the same to serve as one watched once.
• Delivery scales independently — audience growth is absorbed by the CDN, with no change to the host.
• Failure is isolated — a delivery problem does not take down archiving, and an application deploy does not interrupt playback.

One necessary exception

For live multiview, the master playlist is proxied, because that single file is the one whose CORS policy is restricted to the platform’s own origins. Variant playlists and video segments still stream directly from the upstream CDN. The proxy is scoped to the smallest object that requires it rather than to the whole stream.

Why object storage rather than instance disk

Block storage attached to a VM is provisioned and billed whether or not it is used, and growing it means resizing a host that is not otherwise under pressure. Object storage is billed by what is actually stored, needs no capacity planning, and separates the storage lifecycle from the compute lifecycle: I can rebuild the host without touching a single archived file.

Why a CDN, and why this one

Serving video from the origin makes bandwidth the dominant and least predictable line item, because it scales with popularity rather than with the catalogue. R2 charges no egress, so the marginal cost of an additional viewer is effectively zero and the bill is a function of hours retained.

That is what makes hours-based pricing honest: customers pay for retained video, which is exactly the axis on which my cost grows. Viewership, the axis I cannot predict, does not appear on either side.

Why not scale the compute tier instead

Serving video from compute couples an unbounded, bursty workload to the tier that also handles authentication and job submission. It would require over-provisioning for peak, paying for that headroom continuously, and accepting that a traffic spike degrades the product rather than merely costing more. Moving the volume to managed infrastructure removed the need to buy elasticity at all.

Large media is a storage problem, not an application problem

The instinct is to treat video as data the application owns. Treating it as an object the application only references, and keeping it entirely off the request path, removed the majority of the scaling and cost questions before they became problems.

Reliable workers need durable state, not just a queue

A queue alone does not survive contact with retries. Keeping the authoritative status and attempt count in the database, scoping task identity by attempt, and guarding each sub-job with its own claim is what makes a re-queued job safe instead of merely possible.

Most incidents come from the deploy

Adding a verified backup, a strict rollout order and three independent health gates cost an afternoon and removed the failure mode I was most likely to cause myself. The gates that matter are the ones that check the real network path, not just the process.

Simplicity is a capacity decision, not a shortcut

Choosing Compose over an orchestrator was a judgement about what one engineer can operate, not an admission of missing skill. The corresponding obligation is to know exactly which signal would invalidate that choice, and to have the migration path ready before it does.

Design the cost model with the architecture

Selecting storage with no egress charge was an architectural decision as much as a financial one. It is what allows viewing to be free and unauthenticated without the economics inverting as the audience grows.