Tutorials Developer

Monitoring a Media Processing Pipeline

Professional · ~20 min

Overview

A media pipeline fails differently from a web service. Jobs run for minutes or hours, a single bad input can wedge a worker, and the most damaging failure, output that completes successfully but is subtly wrong, produces no error at all. Standard infrastructure monitoring catches almost none of that. This guide covers the metrics that actually indicate health in a media pipeline, what deserves an alert, and how to catch silent-corruption failures that look like successes.

What You Need

A pipeline that already records job state, which is the prerequisite for everything here
A metrics system and somewhere to graph it
Log aggregation, since worker logs are where the causes live
An agreed definition of what "healthy" means for your queue
A small set of canary files whose correct output you know
Someone who will actually receive the alerts

Steps

Measure queue depth and age, not just throughput

Throughput tells you what happened. Queue depth and the age of the oldest waiting job tell you what is about to happen. A queue that is growing steadily is a capacity problem that will become an incident on a predictable schedule. Oldest-job-age is the single most useful number on a media pipeline dashboard.

Track job duration as a distribution

Averages hide everything that matters. Watch percentiles: a stable median with a rising ninety-ninth percentile means a subset of inputs is pathological, which is a completely different problem from everything being uniformly slower. Break the distribution down by job type and input characteristics.

Alert on failure rate and retry rate separately

A rising failure rate means jobs are dying. A rising retry rate with a stable failure rate means jobs are dying and recovering, which masks a real problem while consuming capacity invisibly. Teams that only alert on final failure routinely run for weeks at double the necessary compute cost without noticing.

Watch worker health, not just worker count

Media workers get killed by memory limits, wedged by malformed input, and left holding orphaned processes after cancellation. Track memory high-water marks, processes that outlive their job, and workers that have been running the same job far longer than the distribution suggests they should.

Catch silent corruption with canaries

The worst failure produces a file that exists, has plausible size, and is wrong. Run a small set of known inputs through the live pipeline on a schedule and assert on probed properties of the output, duration, resolution, codec, channel count, stream presence. This is the only monitoring that catches a bad encoder upgrade before your users do.

Alert on symptoms, page on impact

Not everything measured deserves to wake someone. Page when work is not getting done or output is wrong, queue age past your service target, failure rate above threshold, canary assertions failing. Everything else belongs on a dashboard or a daily digest. Alert fatigue is how real incidents get missed.

Pro Tips

Record the toolchain version on every job. When output changes character, that field is the first thing you will want.
Log the exact command line for each job. It converts "the transcode looks wrong" into a reproducible case in seconds.
Storage capacity deserves a real alert with plenty of headroom. A full disk mid-transcode fails in confusing ways.
Track cost per job alongside duration. Media compute bills grow quietly and are hard to attribute afterwards.
Instrument cancellation. Orphaned processes holding CPU after a cancelled job are a common and invisible capacity leak.

Knowledge Base

What You'll Learn

Media pipelines have failure modes that generic monitoring does not look for. Below: why queue metrics lead, and how to detect the failures that report success.

Why Queue Metrics Lead and Resource Metrics Lag

Infrastructure monitoring conventionally centres on CPU, memory, and disk. Those matter, and on a media pipeline they are lagging indicators of a problem the queue already showed you.

Media workers are frequently designed to consume all available CPU. That is what a transcoder is supposed to do. So high CPU is the normal state and tells you nothing about whether the system is keeping up. A pipeline can sit at full CPU utilisation while comfortably meeting its targets, or while falling hours behind.

Queue depth and oldest-job age answer the actual question: is work being completed as fast as it arrives, and how long is the worst-affected user waiting? Those numbers degrade before anything else, and they map directly onto user experience.

The useful dashboard therefore leads with queue age against your service target, then job duration percentiles, then failure and retry rates, with resource metrics available for diagnosis rather than as the primary signal.

The Failures That Report Success

The hardest media pipeline failures are the ones where every job completes, no error is logged, and the output is wrong. Standard monitoring is structurally blind to these because it watches for errors, and there are none.

Typical causes: an encoder upgrade changing a default so output is subtly different. A filter chain silently skipping a step for inputs with an unexpected property. Audio dropped because a stream mapping assumed a channel layout. Rotation metadata ignored so video is delivered sideways. Colour space misinterpreted so everything is washed out.

All of these produce valid files of plausible size, and all of them are typically discovered by a viewer rather than by the system.

Canary jobs are the answer. Push a handful of known inputs through the real pipeline on a schedule and assert on measurable properties of the result rather than on exact bytes, since encoders are not bit-reproducible across versions. Checking duration, resolution, codec, stream count, and channel layout catches the overwhelming majority of these failures within one cycle.

Where This Fits

This guide covers one specific part of media asset management. The wider picture, metadata schemas, naming conventions, proxies and storage tiers, governance, and avoiding vendor lock-in, is in Media Asset Management (MAM) Explained, which frames the discipline as a whole and links out to the detailed guides underneath it, including this one. If you are starting from scratch rather than solving a specific problem, read that first and come back here.

FAQ

Q: What is the most useful single metric for a media pipeline?
A: The age of the oldest job still waiting in the queue. It maps directly onto the worst user experience currently happening, degrades before resource metrics do, and is meaningful whether your pipeline is busy or idle. Throughput and CPU both look healthy while a queue quietly grows.

Q: Why not just alert on CPU like any other service?
A: Because transcoders are designed to consume all available CPU, so high utilisation is the normal state rather than a warning. A pipeline can run at full CPU while comfortably on target or while hours behind. Queue depth and job age distinguish those two cases. CPU does not.

Q: How do I catch a transcode that succeeded but produced wrong output?
A: Canary jobs. Run a small set of known inputs through the live pipeline on a schedule and assert on probed properties of the output (duration, resolution, codec, stream count, channel layout) rather than on exact bytes, since encoders are not bit-reproducible. This is the only monitoring that catches a bad encoder upgrade before users do.

Q: Should I alert on retries?
A: Yes, separately from failures. A stable failure rate with a rising retry rate means jobs are dying and recovering, which hides a real problem while silently consuming extra capacity and cost. Teams that only watch final outcomes often run for weeks at inflated compute spend without noticing.