Tutorials Developer

WebCodecs: Encoding and Decoding in the Browser

Professional · ~20 min

Overview

WebCodecs gives browsers low-level access to the encoders and decoders already present for playback, which makes genuine client-side media processing possible: trimming, transcoding, frame extraction, and effects without uploading anything. That is a meaningful change for privacy, latency, and server cost. It also has hard constraints that server-side processing does not, particularly around memory on mobile. This guide covers where it wins and where it does not.

What You Need

A browser that supports the API, capability detection is mandatory, not optional
An understanding of frames, chunks, and the demux/mux boundary
A muxer and demuxer, since WebCodecs handles codecs and not containers
Web Workers, because this work does not belong on the main thread
Real mobile devices for testing memory behaviour
A server-side fallback path for unsupported browsers

Steps

Detect support and plan the fallback first

Support varies by browser and platform, and the available codecs vary within supported browsers. Query configuration support before assuming anything, and design the server-side path as the default rather than the exception. A client-side feature that silently fails on a third of your users is worse than not having it.

Understand that WebCodecs does codecs, not containers

The API encodes and decodes compressed chunks. It does not read MP4 files or write them, demuxing an input and muxing an output are your responsibility, usually via a library. This surprises people who expect a file-in, file-out interface, and it is the first thing to plan for architecturally.

Do the work in a worker

Decoding, processing, and encoding frames on the main thread will make the interface unresponsive. Move the pipeline into a Web Worker and transfer frames rather than copying them. This is not an optimisation to add later. It changes the structure enough that retrofitting is painful.

Close every frame explicitly

Video frames hold substantial memory outside the normal garbage-collected heap, and they are not reclaimed automatically when a reference goes out of scope. Failing to close each frame after use leaks quickly and crashes the tab, particularly on mobile. This is the single most common WebCodecs bug.

Apply backpressure rather than queueing everything

Decoding faster than you process produces a growing queue of frames in memory, which is exactly how a working desktop implementation crashes a phone. Watch the queue size and pause decoding when it grows. Treat memory as the constraint you design around rather than discover.

Choose client-side for the right reasons

It wins where the round trip is the cost: instant preview, trimming before upload, privacy-sensitive material that should not leave the device, and reducing server bills for simple operations. It loses where consistency and scale matter, because you control neither the hardware nor the codec implementation.

Pro Tips

Close frames as soon as you are done with them. Leaked frames crash mobile tabs fast.
Test memory behaviour on a real mid-range phone, not a flagship and not a desktop.
Feature-detect specific codec configurations, not just the presence of the API.
Keep the server path as the default. Client-side is an optimisation, not a foundation.
Hardware and software encoders produce different output. Do not assume client-side results are reproducible.

Knowledge Base

What You'll Learn

WebCodecs changes what a browser can do with media and does not remove the constraints of running on someone else's device. Below: where it genuinely wins, and the memory model that catches people out.

Where Client-Side Processing Genuinely Wins

The decision is not about capability, plenty is possible both ways, but about what the round trip costs.

Latency-sensitive interaction. Trimming a clip, scrubbing a preview, or applying an effect and seeing it immediately are far better client-side. Uploading a gigabyte to find out where to cut is a poor experience regardless of how fast the server is.

Privacy. Material that should not leave the device (medical, legal, personal) can be processed without ever being transmitted. This is a genuine architectural advantage rather than a marketing point.

Cost at the margins. Trimming before upload reduces bandwidth and storage meaningfully. Simple operations done client-side remove server compute entirely.

Where it loses: anything requiring consistent, reproducible output, because you control neither the hardware encoder nor its version. Anything long-running, because the tab may be backgrounded or closed. Anything on unpredictable hardware where a mid-range phone must complete the same work as a workstation.

The Memory Model That Catches People Out

WebCodecs failures on mobile are almost always memory, and the reason is a mismatch between how JavaScript usually behaves and how frames actually work.

A decoded video frame is a large buffer held outside the normal garbage-collected heap. Unlike ordinary objects, it is not reclaimed when the last reference disappears. It must be explicitly released. A pipeline that decodes frames and lets them fall out of scope will consume memory until the tab is terminated.

The second trap is queue growth. Decoders can typically produce frames faster than downstream code consumes them, so without backpressure an entire video accumulates in memory as decoded frames, vastly larger than the compressed file. This works on a desktop with abundant memory and fails on the phone where most people will run it.

Mobile browsers also terminate tabs that allocate aggressively, with limits that are neither published nor consistent. The practical discipline is to close frames immediately, cap the number in flight, monitor queue depth, and test on a mid-range device rather than the best one you own.

Where This Fits

This guide covers one specific part of building media applications. The wider picture, why media workloads break ordinary web architecture, and the upload, job, and toolchain patterns that handle them, is in Building Media Applications: A Developer Primer, which frames the discipline as a whole and links out to the detailed guides underneath it, including this one. If you are starting from scratch rather than solving a specific problem, read that first and come back here.

FAQ

Q: Can WebCodecs read and write MP4 files?
A: No. It encodes and decodes compressed chunks. Containers are your responsibility. You need a demuxer to get chunks out of an input file and a muxer to write output, usually from a library. This is the first architectural surprise for people expecting a file-in, file-out interface.

Q: Why does my WebCodecs pipeline crash on mobile?
A: Almost certainly memory. Video frames hold large buffers outside the garbage-collected heap and must be closed explicitly, and decoders produce frames faster than most pipelines consume them. Close every frame immediately after use and apply backpressure so the queue cannot grow unbounded.

Q: Should I process media client-side or on the server?
A: Client-side where the round trip is the cost: instant preview, trimming before upload, or privacy-sensitive material. Server-side where consistency, reproducibility, or scale matter, since you control neither the client hardware nor its encoder. Many applications sensibly do both.

Q: Is output from client-side encoding reproducible?
A: No. Hardware and software encoders differ, and implementations vary across browsers, platforms, and devices. Do not build anything that assumes byte-identical output or consistent quality characteristics from client-side encoding. Verify by probing properties rather than comparing files.