Tutorials Developer

API Design for Async Media Processing Jobs

Professional · ~20 min

Overview

A typical REST API returns a result in the same request that asked for it. Media processing (transcoding, rendering, analysis) can take anywhere from seconds to hours, which breaks that assumption completely. This guide covers the job-queue pattern media processing APIs actually need (accepting a job, tracking its status, and delivering the result asynchronously) independent of any specific framework.

What You Need

A queue or task-scheduling system (even a simple one) to hold accepted jobs
Persistent storage for job status and results, separate from the queue itself

Steps

Accept the job, don't try to process it synchronously

The initial API call should do the minimum necessary: validate the request, enqueue the job, and immediately return a job ID. Processing happens separately, decoupled from the request that triggered it.

Design an explicit, finite set of job states

Something like queued, processing, completed, and failed, with no ambiguous in-between states. Clients build logic around these states directly. An inconsistent or growing state list breaks client integrations that assumed a fixed set.

Offer both polling and webhook delivery

A status endpoint clients can poll works everywhere, including environments that can't receive inbound webhooks. A webhook callback on completion is more efficient for clients that can use it. Supporting both covers the broadest range of integration environments.

Make webhook delivery idempotent and retryable

Networks fail. A webhook delivery can be attempted more than once, and the receiving client needs to handle a duplicate delivery safely (via a job ID or delivery ID) rather than assuming exactly-once delivery, which no reliable webhook system actually guarantees.

Return actionable errors, not just a failed state

A job that fails should report why (an unsupported codec, a corrupt input file, a resource limit hit) not just a generic failure flag. This is the single biggest factor in how much support burden a media processing API generates.

Version your job payload schema from day one

Job request and result payloads inevitably need new fields over time. Version the schema explicitly from the start rather than retrofitting versioning after you already have integrations depending on an unversioned shape.

Pro Tips

Include estimated processing time or queue position in the initial response where possible. It meaningfully improves the integrating developer's experience even though it's optional.
Log every state transition with a timestamp, when a client asks "why did my job take 20 minutes," a state-transition log answers it immediately instead of requiring investigation.
Set a job expiration/cleanup policy from the start, completed job results and their associated files accumulate quickly at any real volume.

Knowledge Base

Media Processing Breaks the Synchronous Request Model

Most API design guidance assumes a request completes in milliseconds. Media processing timescales range from seconds to hours depending on file size and complexity, trying to force that into a synchronous request/response either times out unpredictably or ties up server resources holding connections open for no good reason. The job-queue pattern isn't a stylistic choice here. It's a structural necessity.

Webhooks and Polling Solve Different Constraints

Webhooks are more efficient (no wasted polling requests) but require the client to have a reachable public endpoint, which isn't always true. Some integration environments run behind firewalls or in contexts that can't receive inbound calls. Polling is less efficient but universally compatible. Offering both, rather than picking one, covers the realistic range of client environments.

Idempotency Is Not Optional at Any Real Scale

Any sufficiently large system will eventually retry a webhook delivery, whether due to a timeout, a transient network failure, or an at-least-once delivery guarantee by design. An API (and its clients) that assumes exactly-once delivery will eventually process a job's result twice, building idempotency in from the start avoids a class of bugs that's otherwise painful to retrofit.

Where This Fits

This guide covers one specific part of media asset management. The wider picture, metadata schemas, naming conventions, proxies and storage tiers, governance, and avoiding vendor lock-in, is in Media Asset Management (MAM) Explained, which frames the discipline as a whole and links out to the detailed guides underneath it, including this one. If you are starting from scratch rather than solving a specific problem, read that first and come back here.

FAQ

Q: Why not just make the client wait on a long HTTP request until processing finishes?
A: Media processing times are too variable and often too long for a held-open HTTP connection to be reliable, proxies, load balancers, and client timeouts all tend to have limits well under realistic processing times for larger files. A job-queue pattern with a separate status check avoids depending on a single fragile long-lived connection.

Q: Should I offer webhooks, polling, or both?
A: Both, if you can, webhooks are more efficient for clients that can receive them, but not every integration environment can accept inbound webhooks (some are behind firewalls or lack a public endpoint). Polling is a reliable fallback that works everywhere, even if it's less efficient.