Tutorials Developer

Building Media Applications: A Developer Primer

Intermediate · ~30 min

Overview

Media applications break the assumptions ordinary web architecture is built on. Requests finish in milliseconds. Transcodes take minutes. Payloads are kilobytes. Media files are gigabytes. Business logic is deterministic. Codec behaviour is a thicket of container formats, profiles, and hardware quirks. This guide covers where those assumptions fail, the patterns that address each failure, and what the browser can and cannot do natively. It is aimed at developers who are competent generally but new to media specifically. The category most likely to be surprised.

What You Need

Working knowledge of a backend language and HTTP
FFmpeg installed locally. It is the substrate under most of this domain
Object storage you can issue presigned URLs against
A job queue or task runner of some kind
Containers, for reproducing codec and library versions reliably
Test media that is genuinely awkward: variable frame rate, odd aspect ratios, broken metadata

Steps

Never route media bytes through your application server

Uploading a multi-gigabyte file through your API server consumes a worker for minutes, exhausts memory, and fails at whatever proxy limit sits in front of it. Issue a presigned URL and have the client transfer directly to object storage, then notify your backend with the resulting object key. The application handles metadata and coordination. The storage layer handles bytes.

Make uploads resumable and chunked

A large upload over a mobile connection will be interrupted, and restarting from zero is both a poor experience and a large bandwidth cost. Chunked, resumable protocols let a transfer continue from where it stopped and allow parallel chunk upload for throughput. Validate checksums per chunk rather than trusting a completed transfer.

Process asynchronously and model the job explicitly

Transcoding cannot be done inside a request. Accept the work, persist a job record, return an identifier immediately, and process on a worker. The job needs a real state machine (queued, running, succeeded, failed, cancelled) with progress, attempt count, and an error field that survives the process that produced it. Clients poll or subscribe rather than waiting.

Make jobs idempotent and safe to retry

Workers get killed mid-transcode by deployments, spot-instance reclamation, and memory limits. Any job may therefore run more than once. Derive output paths deterministically from the input and job parameters, write to a temporary location and atomically move on success, and make completion a compare-and-set rather than a blind write. Retrying should be boring.

Pin your media toolchain in a container

FFmpeg behaviour varies meaningfully between versions and builds, which encoders are compiled in, which flags exist, which defaults changed. A pipeline that works on a developer laptop and fails in production is usually a build difference. Pin the exact image, treat upgrades as deliberate migrations with regression tests, and record the toolchain version on each job for later diagnosis.

Probe input before you trust it

User-supplied media is reliably malformed. Variable frame rate, rotation stored only in metadata, audio streams with unexpected channel layouts, durations that disagree between container and stream, and files that are not what their extension claims are all routine. Probe first, validate against what your pipeline supports, and reject clearly with a useful message rather than failing deep inside a transcode.

Generate derivatives rather than serving originals

Serving a source file to a browser is slow and often unplayable. Produce web-friendly renditions. A compatible codec and container, faststart metadata at the front so playback can begin before download completes, plus thumbnails and preview proxies. For anything at scale, produce multiple renditions and serve adaptively so playback matches the connection.

Budget storage and egress as first-class costs

Media systems are usually dominated by storage and bandwidth rather than compute. Decide retention explicitly, tier cold material to cheaper storage, and put a CDN in front of anything served repeatedly. Egress in particular tends to be the line item that surprises teams, because it grows with success rather than with data volume.

Pro Tips

Keep the original master untouched forever and derive everything else. Regenerating derivatives is cheap. Re-acquiring a source is not.
Log the exact command line and toolchain version for every job. It converts "the transcode looks wrong" into a reproducible case.
Set explicit resource limits on media workers. FFmpeg will happily consume every core and all available memory.
Test with deliberately hostile files, zero-length, truncated, wrong extension, enormous resolution, unusual frame rates.
Prefer streaming through pipes over writing intermediate files when chaining operations. Disk I/O dominates surprisingly often.
Treat progress reporting as a product feature. Long jobs without progress feel broken even when they are working.

Knowledge Base

What You'll Learn

This is the entry point for the engineering material on this site. Below: why media breaks standard assumptions, what the browser can do natively, how to think about codecs without becoming a codec expert, and where the deeper guides go.

The four assumptions media breaks

Requests are short. Web frameworks, load balancers, and proxies all assume a request completes quickly. A transcode takes minutes to hours. Everything about timeouts, worker pools, and connection handling has to change, which is why asynchronous job processing is not an optimisation here but the baseline architecture.

Payloads are small. Standard request handling buffers bodies in memory. A gigabyte upload through that path exhausts memory or hits a proxy limit. Media systems move bytes around the application rather than through it.

Work is deterministic. Ordinary business logic produces the same output for the same input. Media processing depends on library versions, hardware acceleration availability, and encoder defaults that change between releases, so reproducibility must be engineered through pinned toolchains rather than assumed.

Failures are exceptional. In media pipelines, malformed input is the normal case. A meaningful share of user-supplied files will have something wrong with them, and the pipeline's behaviour on bad input is a primary design concern rather than an edge case.

What the browser can actually do

Browser media capability is much stronger than it was and still has hard edges worth knowing before you plan around it.

Web Audio is genuinely powerful: a full node-graph synthesis and processing system with real filters, analysis, and offline rendering faster than real time. Substantial audio tools run entirely client-side with no server involvement.

Canvas and WebGL/WebGPU handle image and frame manipulation well, including per-frame video processing at moderate resolutions.

WebCodecs exposes low-level encode and decode, which makes real client-side transcoding viable in supporting browsers. A significant change from the era when everything had to round-trip to a server.

The constraints are memory and consistency. Mobile browsers terminate tabs that allocate too much, and the limits are neither published nor uniform. A design that works on a desktop can crash a phone. Codec support also varies by browser and platform for licensing reasons, so capability detection is mandatory rather than optional.

Codecs, containers, and the distinction that causes most confusion

A container: MP4, MOV, MKV, WebM: is a wrapper describing how streams, timing, and metadata are organised. A codec. H.264, HEVC, AV1, AAC, Opus, is the compression method for the actual stream inside it. A file extension names the container and tells you little about whether anything can play it.

This is the source of the most common support complaint in media applications: a file that plays in one place and not another, with the same extension. The container is fine. The codec inside is unsupported.

For broad compatibility, H.264 video with AAC audio in an MP4 container remains the safest combination by a wide margin, at the cost of efficiency. HEVC and AV1 compress substantially better, AV1 particularly, but support is uneven and hardware decode availability matters for battery and performance on mobile.

One detail worth knowing specifically: MP4 files store an index that browsers need before playback can start, and by default it is written at the end of the file. Moving it to the front, faststart, is what allows playback to begin before the whole file has downloaded, and forgetting it is a very common cause of "the video takes forever to start".

Designing the job model

The job record is the core abstraction in a media backend, and it is worth designing carefully because everything else depends on it.

At minimum it needs a stable identifier, an explicit state, the input reference, the parameters that determine output, a progress indicator, an attempt count, a structured error, and the toolchain version used. Storing parameters explicitly rather than reconstructing them lets you re-run a job identically months later, which matters for debugging and for regenerating derivatives after a pipeline change.

Progress deserves specific attention. FFmpeg emits progress on a parseable stream, and surfacing it turns an opaque several-minute wait into something users tolerate. Where true progress is not available, a staged indicator naming the current phase is still far better than a spinner.

Cancellation should be real. A cancelled job needs to kill the underlying process and clean up partial output, not merely mark a row. Orphaned processes holding CPU and disk are a common operational problem in systems that treated cancellation as a status change.

Where to go deeper on this site

The large file uploads guide covers chunked and resumable transfer in practical detail, and the API design for async media jobs guide covers the job endpoint and state machine at professional depth. Containerizing media processing tools covers toolchain pinning.

For operational concerns, automated testing for media applications covers how to test pipelines whose output is a file rather than a value, and CI/CD for cross-platform builds covers shipping media tooling. CDN architecture for video at scale covers delivery once volume is real.

For domain context, media asset management explains how media organisations think about assets and metadata, and the behind-the-tools explainers on video codecs and adaptive bitrate streaming cover the compression and delivery mechanics that this page only summarises.

FAQ

Q: Should I transcode on the server or in the browser?
A: Server for anything that must be reliable, consistent, or large. You control the toolchain and the resources. Browser via WebCodecs for quick client-side operations where avoiding a round trip is the point, accepting that capability varies by browser and that mobile memory limits are real and unforgiving. Many systems do both, with the browser handling previews and the server producing masters.

Q: Why does my video play in one browser but not another?
A: Almost always a codec support difference rather than a container problem. The file extension describes the container while the codec inside determines playability, and support varies by browser and platform for licensing reasons. Producing an H.264/AAC MP4 rendition alongside more efficient formats is the standard fix.

Q: How do I show accurate progress for a transcode?
A: FFmpeg can emit machine-readable progress on a separate stream, which you parse in the worker and write to the job record for clients to poll or subscribe to. Where the total duration is unknown, report the current stage by name instead. A named phase is far more reassuring than an indeterminate spinner.

Q: What is the biggest architectural mistake in media applications?
A: Routing media bytes through the application server: as uploads, as downloads, or both. It exhausts memory, occupies workers for minutes, and collides with proxy limits. Presigned direct-to-storage transfer with the application handling only coordination and metadata avoids an entire category of scaling problem.

Q: Do I need to understand codecs in depth?
A: No, but you need the container-versus-codec distinction, why faststart matters for streaming playback, and roughly how the common codecs trade compression against compatibility and decode cost. That covers most day-to-day decisions. Deep codec knowledge only becomes necessary when you are optimising encoding ladders or debugging quality at the bitstream level.

Q: How should I test a media pipeline?
A: With a fixture set of deliberately awkward real files, variable frame rate, rotation metadata, unusual channel layouts, truncated files, wrong extensions, and assertions on probed properties of the output rather than on exact bytes, since encoders are not bit-reproducible across versions. Checking duration, resolution, codec, channel count, and stream presence catches the overwhelming majority of regressions.