Tutorials Developer

How Video Codecs Actually Decide What to Compress

By Infinite Creation · Advanced · ~14 min

A 1960s mid-century comic book style editorial illustration showing two engineers at terminals in a machine room, with a colour grading panel with trackballs visible in the frame.

Overview

A video codec isn't simply 'squeezing' a file smaller uniformly. It's making specific, deliberate decisions about which visual information human perception is least likely to notice missing, based on a handful of well-understood techniques that this explainer walks through simply.

Steps

The simple version: codecs throw away what you likely won't notice

At the simplest level, video compression works by identifying visual information that has a smaller impact on perceived quality and reducing or removing it, rather than uniformly shrinking every piece of data by the same amount. The goal is maximizing perceived quality at a given file size, not literally preserving every original pixel value.

What's really happening: motion prediction between frames

Rather than storing every frame as a completely independent image, most modern codecs store only the first frame in a sequence fully, then store subsequent frames as descriptions of what changed relative to nearby frames (since consecutive video frames are usually very similar). A technique called motion prediction or inter-frame compression, which is responsible for most of a video codec's size reduction.

What's really happening: chroma subsampling

Human vision is more sensitive to changes in brightness (luminance) than to changes in color (chrominance), so most codecs store color information at a lower effective resolution than brightness information. A technique called chroma subsampling that reduces file size with minimal perceptible quality loss for most typical viewing, though it can become visible in specific situations like green screen keying, which is why some professional workflows request less aggressive subsampling.

Why this matters practically

Understanding motion prediction explains why fast motion and scene cuts compress worse than static shots (there's more change between frames to describe), and understanding chroma subsampling explains why green screen keying sometimes shows ragged edges even on otherwise clean-looking footage (the color information has less effective resolution than the brightness information the eye reads edges from). Both are directly practical, not just academic.

Tested: what motion prediction and chroma subsampling do

We tested steps 2 to 4 on 2026-09-25 with FFmpeg 8.1.2 and its x264 encoder. We made six five-second 720p clips from a computer-generated picture with the same spread of fine and coarse detail as a photograph: a static shot, three pans, a cut every second, and a completely new picture every frame. We encoded each twice at the same quality setting, once normally and once with every frame stored whole, as if the codec had no motion prediction. The last column is the quality of the normal encode, and the whole-frame encode matched it within 0.6 dB every time.

Clip	Normal encode	Every frame whole	Saved by prediction	Quality (PSNR)
Static shot	289 kB	35,059 kB	121.4 times	36.5 dB
Pan, 2 pixels a frame	353 kB	35,060 kB	99.2 times	36.5 dB
Pan, 8 pixels a frame	519 kB	35,047 kB	67.5 times	36.5 dB
Pan, 32 pixels a frame	1,179 kB	35,053 kB	29.7 times	36.5 dB
A cut every second	1,413 kB	35,061 kB	24.8 times	36.5 dB
A new picture every frame	37,207 kB	37,204 kB	1.0 times	36.6 dB

Step 2 holds, and the table shows where the saving comes from. The static shot came out 121 times smaller with prediction, because each frame after the first was almost entirely "same as before". With a new, unrelated picture every frame, prediction saved nothing (1.00 times). The codec can only describe change cheaply when most of the picture was already there.

Step 4 and the first question below hold, in order: the static shot, then the slow, medium and fast pans, then the cuts, then new pictures, each bigger than the last. The pans came to 353 kB, 519 kB and 1,179 kB. Most of the extra is the strip of new picture that slides in at the edge of every frame, which has nothing earlier to be predicted from. Priced at what a whole frame cost, that strip accounted for each pan's extra size to within 19 percent. Our pans had no motion blur, no grain and nothing moving within the picture, which camera footage has and which costs more again. Treat these as the best case for motion prediction.

We held the quality fixed with a constant quantizer, because the usual CRF setting would not have compared like with like. At CRF 23 the static shot's normal encode measured 42.7 dB and its whole-frame encode 27.8 dB. At CRF, x264 gives extra quality to the parts of a frame that later frames copy from, so prediction bought quality as well as size.

For step 3 we built a green-screen frame with a round subject and hair-like strands 1, 2 and 3 pixels wide, stored it losslessly as 4:4:4, 4:2:2 and 4:2:0, and keyed it pixel by pixel on colour alone. At 4:4:4 every strand survived. At 4:2:2, which halves colour detail across the frame only, 1-pixel vertical strands vanished and horizontal ones survived. At 4:2:0 all 1-pixel strands vanished. Strands 2 and 3 pixels wide survived at every setting, however they lined up with the pixel grid.

The subject's round edge shows the ragged-edge effect the FAQ describes. Pixels partly between the screen colour and the subject's made a band 0.03 pixels wide along the edge at 4:4:4, 1.57 at 4:2:2 and 2.08 at 4:2:0. That band is where a soft key shows green fringe. FFmpeg's own chromakey filter lost the 1-pixel strands even from the 4:4:4 file, so the keyer's design can matter as much as the subsampling.

Validated. Measured 2026-09-25 with FFmpeg 8.1.2 (x264, QP 26 with every frame type at the same quantizer and adaptive quantization off) and numpy on macOS 26.6.2. The pictures are computer-generated: rigid pans with no blur, grain or moving subject, and a flat subject colour on an even green. Sizes and edges from camera footage will differ. The key is our own pixel-by-pixel colour key and FFmpeg's chromakey filter, not a commercial keyer.

Pro Tips

Expect higher-motion footage (action, fast pans, scene cuts) to need a higher bitrate for the same visual quality as static shots. This follows directly from how motion prediction compression works.
If you're keying green screen footage, request less aggressive chroma subsampling in your recording settings if your camera and workflow support it. This directly addresses the edge-quality issue subsampling can cause.
Don't assume all codecs handle these techniques identically, different codecs make different specific tradeoffs, which is part of why some compress more efficiently than others for the same visual quality.

Knowledge Base

What You'll Learn

Video compression is a set of deliberate, perception-based decisions, motion prediction between frames and chroma subsampling being two of the biggest, rather than a uniform shrinking process, and understanding these techniques explains several practical video quality behaviors.

The Plain-English Version

A codec looks at what changes between frames and stores mostly the changes, and it stores color at a lower effective resolution than brightness because your eyes care less about exact color precision than exact brightness precision. Both save file size with minimal noticeable quality loss for most viewing.

The Misunderstanding This Clears Up

People sometimes assume a bigger file size or higher resolution alone guarantees better quality, but if the codec's motion prediction or chroma subsampling is fighting against your specific content (very high motion, or green screen keying), a technically larger file can still look or perform worse than expected for that specific use.

Where This Fits

Codec behaviour is why media breaks ordinary web architecture, the problem Building Media Applications is built around.

FAQ

Q: Why does fast-motion footage need a higher bitrate to look good?
A: Because most codecs compress by describing what changes between frames rather than storing every frame fully, fast motion or scene cuts create more change to describe, requiring more data (a higher bitrate) to maintain the same visual quality as a mostly static shot.

Q: Why does green screen footage sometimes show ragged or fringed edges even when it looks clean otherwise?
A: This is often related to chroma subsampling. Most codecs store color information at a lower effective resolution than brightness information, and a keyer relies heavily on precise color boundaries, so the reduced color resolution can show up as imprecise, ragged edges even when overall picture quality looks fine to the eye.