Tutorials Developer

How Video Codecs Actually Decide What to Compress

Advanced · ~14 min

Overview

A video codec isn't simply 'squeezing' a file smaller uniformly. It's making specific, deliberate decisions about which visual information human perception is least likely to notice missing, based on a handful of well-understood techniques that this explainer walks through simply.

What You Need

No special equipment required. This is a technical explainer, not a hands-on tutorial

Steps

The simple version: codecs throw away what you likely won't notice

At the simplest level, video compression works by identifying visual information that has a smaller impact on perceived quality and reducing or removing it, rather than uniformly shrinking every piece of data by the same amount. The goal is maximizing perceived quality at a given file size, not literally preserving every original pixel value.

What's really happening: motion prediction between frames

Rather than storing every frame as a completely independent image, most modern codecs store only the first frame in a sequence fully, then store subsequent frames as descriptions of what changed relative to nearby frames (since consecutive video frames are usually very similar). A technique called motion prediction or inter-frame compression, which is responsible for most of a video codec's size reduction.

What's really happening: chroma subsampling

Human vision is more sensitive to changes in brightness (luminance) than to changes in color (chrominance), so most codecs store color information at a lower effective resolution than brightness information. A technique called chroma subsampling that reduces file size with minimal perceptible quality loss for most typical viewing, though it can become visible in specific situations like green screen keying, which is why some professional workflows request less aggressive subsampling.

Why this matters practically

Understanding motion prediction explains why fast motion and scene cuts compress worse than static shots (there's more change between frames to describe), and understanding chroma subsampling explains why green screen keying sometimes shows ragged edges even on otherwise clean-looking footage (the color information has less effective resolution than the brightness information the eye reads edges from). Both are directly practical, not just academic.

Pro Tips

Expect higher-motion footage (action, fast pans, scene cuts) to need a higher bitrate for the same visual quality as static shots. This follows directly from how motion prediction compression works.
If you're keying green screen footage, request less aggressive chroma subsampling in your recording settings if your camera and workflow support it. This directly addresses the edge-quality issue subsampling can cause.
Don't assume all codecs handle these techniques identically, different codecs make different specific tradeoffs, which is part of why some compress more efficiently than others for the same visual quality.

Knowledge Base

What You'll Learn

Video compression is a set of deliberate, perception-based decisions, motion prediction between frames and chroma subsampling being two of the biggest, rather than a uniform shrinking process, and understanding these techniques explains several practical video quality behaviors.

The Plain-English Version

A codec looks at what changes between frames and stores mostly the changes, and it stores color at a lower effective resolution than brightness because your eyes care less about exact color precision than exact brightness precision. Both save file size with minimal noticeable quality loss for most viewing.

The Misunderstanding This Clears Up

People sometimes assume a bigger file size or higher resolution alone guarantees better quality, but if the codec's motion prediction or chroma subsampling is fighting against your specific content (very high motion, or green screen keying), a technically larger file can still look or perform worse than expected for that specific use.

Where This Fits

This guide covers one specific part of building media applications. The wider picture, why media workloads break ordinary web architecture, and the upload, job, and toolchain patterns that handle them, is in Building Media Applications: A Developer Primer, which frames the discipline as a whole and links out to the detailed guides underneath it, including this one. If you are starting from scratch rather than solving a specific problem, read that first and come back here.

FAQ

Q: Why does fast-motion footage need a higher bitrate to look good?
A: Because most codecs compress by describing what changes between frames rather than storing every frame fully, fast motion or scene cuts create more change to describe, requiring more data (a higher bitrate) to maintain the same visual quality as a mostly static shot.

Q: Why does green screen footage sometimes show ragged or fringed edges even when it looks clean otherwise?
A: This is often related to chroma subsampling. Most codecs store color information at a lower effective resolution than brightness information, and a keyer relies heavily on precise color boundaries, so the reduced color resolution can show up as imprecise, ragged edges even when overall picture quality looks fine to the eye.