Tutorials Developer

Your First Media Feature: What Breaks and Why

By Infinite Creation · Beginner · ~18 min

The Problem

The first time a developer adds video or audio to an application, several things that have always worked stop working. The upload times out. The conversion blocks a request until it fails. The file plays on the machine it was made on and nowhere else. None of this is because media programming is hard. It is because media breaks four assumptions that are safe everywhere else in web development, and knowing which four is most of the difficulty.

Method

Assumption one: a request holds the whole thing

A form post assumes the payload fits in memory, finishes inside a timeout, and can be retried from the start. A two-gigabyte video satisfies none of those. The answer is chunked or resumable uploads, where the file is cut into pieces that are uploaded independently and reassembled, so a failure costs one piece rather than an hour.

Assumption two: work finishes inside a request

Transcoding a video takes minutes and sometimes longer than the video itself. Doing it inside the request that uploaded it holds a connection open, occupies a worker, and loses everything on a restart. Media work belongs in a job queue, with the request returning an identifier and the client asking about progress separately.

Assumption three: the file format is the format

An MP4 is a container, not a codec, and what is inside it decides whether a given browser can play it. Two files with the same extension can behave completely differently. Inspect what is actually in a file rather than trusting its name, which ffprobe does in one command.

Assumption four: you can trust the input

Uploaded media arrives with wrong rotation flags, variable frame rates, audio at an unexpected sample rate, and occasionally a file that is not media at all. Normalising on ingest, to one known output format, saves far more time than handling every variation downstream.

Learn ffmpeg before reaching for a library

Most media libraries in most languages are wrappers around ffmpeg. Learning the underlying commands means you can debug what the wrapper produced, and you will need to, because the wrapper's error messages are usually ffmpeg's error messages with less context.

Generate a thumbnail and a duration on ingest

Almost every media feature needs both, and computing them later means reading the file again. Extracting a frame and reading the metadata at ingest time is cheap and removes a whole class of later work.

Store the original and derive everything else

Keep exactly what was uploaded, untouched, and treat every converted version as disposable. When a decision about output format turns out to be wrong, and it will, having the original means regenerating rather than asking users to upload again.

Measure before optimising anything

Media workloads are dominated by one or two operations, and they are usually not the ones that feel slow. Time the pipeline stage by stage before rewriting anything, because intuition about where the time goes is unreliable here.

What the Method Assumes

Any web application you can add a file upload to
A command line with ffmpeg installed, which is how nearly all of this work is actually done
A video file large enough to be inconvenient, because small files hide every problem here

Gotchas

ffprobe answers most questions faster than reading documentation about a format.
A file that plays in one browser and not another is nearly always a codec question, not a corruption question.
Media storage costs grow faster than anyone plans for, because every derived version is another copy. Decide early which ones are worth keeping.
Test with a large file from the beginning. Every problem on this page is invisible on a ten-second clip.

Knowledge Base

Reference Notes

Two distinctions clear up most of the early confusion: containers against codecs, and why so much of this work is asynchronous.

Containers and codecs

A container is a box: MP4, MOV, WebM, MKV. It holds streams and describes where they are. A codec is how each stream is compressed: H.264, HEVC, AV1, AAC, Opus. The extension tells you the box, and the box tells you very little about whether something will play.

This is why a file that works on a developer's machine fails on a user's phone. Both can open the container and only one can decode what is inside. When something will not play, the first question is always which codecs are in the file and which the target supports, and both halves are answerable in a few seconds.

Why media work is asynchronous

The cost of processing media scales with duration and resolution rather than with request volume, which is the opposite of most web work. A single upload can occupy a core for minutes, and ten simultaneous uploads can occupy the machine.

Once processing is in a queue, several things become possible that were not: retrying a failed job without the user re-uploading, running heavy work on different hardware from the web tier, and applying back pressure when demand spikes. The queue is not an optimisation added later. It is the shape the problem has.

Where This Fits

This page names what breaks, and Building Media Applications lays out the architecture that handles it.

FAQ

Q: Do I need to learn ffmpeg?
A: Effectively yes. Nearly every tool and library in this space is built on it, and understanding the commands is what lets you debug the layers above.

Q: Should I transcode on my own servers or use a service?
A: A service is usually cheaper until volume is high and predictable. The reason to run it yourself is control over formats and cost at scale, not simplicity.

Q: Why does my uploaded video appear sideways?
A: Phones record in one orientation and set a rotation flag rather than rotating the pixels. Some players honour it and some do not, which is why normalising orientation on ingest is standard practice.

Q: What format should I convert everything to?
A: For broad browser playback, H.264 video with AAC audio in an MP4 remains the safest single answer. Newer codecs compress better and are worth adding as alternatives rather than as the only option.

Vocabulary

Container The file format that holds media streams and describes where they are, such as MP4 or WebM. It is what the file extension names, and it tells you very little on its own about whether a given player can decode the contents.

Codec The method used to compress a single stream, such as H.264 for video or AAC for audio. Playback failures are nearly always a codec question rather than a container question, because a player can open a box it cannot read.

Transcode Converting media from one codec or container to another. It costs time proportional to the duration of the media rather than to request volume, which is why it belongs in a job queue and not in a request.

Resumable upload An upload cut into independently transferred pieces that are reassembled at the destination. It exists because a single large transfer that fails at ninety per cent otherwise has to start again from nothing.