Think of a factory assembly line for bottling juice: one station washes the bottles, the next fills them, another caps them, a fourth slaps on a label. No single worker does everything — each handles one step and slides the bottle along to the next. New bottles keep entering while finished ones roll off the end.
Pipes and Filters is that assembly line for data. A big processing job is split into a series of small stages (the filters), connected by channels (the pipes) that carry the output of one stage into the input of the next.
The problem
When all the steps of a job live inside one big function or service, they fuse into a tangle. Take an image service that, for every upload, decodes the file, resizes it, stamps a watermark and compresses the result. All four steps share one codebase, one deploy and one machine. You can't change one step without retesting the others, and you can't reuse a step elsewhere because it's welded to its neighbors.
Scaling gets awkward too. Resizing takes 400 of every 600 milliseconds, while the other steps are quick. Because they're bundled together, the only way to give resize more room is to run more copies of the whole service, each one carrying steps that were never the problem.
Step through it below. When uploads jump, predict how many copies of the whole service it takes to keep up.
How it works
You pull each step out into its own self-contained filter. A filter receives data on its input pipe, performs exactly one transformation, and writes the result to its output pipe — and that's all it knows about. It doesn't know who fed it or who consumes it, only the shape of the data flowing through.
That independence is the whole payoff. You can reorder stages, drop a stage in or out, reuse a stage in another pipeline, and scale each stage on its own — give the slow one more workers while the fast ones run lean. And because every stage processes a stream, they all run at once: while stage three works on item one, stage one is already pulling in item three. This is map/filter/reduce thinking stretched into a distributed pipeline.
Below, the same four steps run as four filters joined by queues. When rush hour hits again, watch which queue fills up, and predict how many workers the slow filter needs. Then see what the split costs: a hop for every pipe, and a debugging question you want answered before your first incident. At the end, flip the switch to trace one image with and without a correlation ID.
Make filters idempotent and let pipes buffer. If a stage crashes partway through, the item should be safe to reprocess without corrupting anything — so design each filter to be idempotent. Using a durable queue as the pipe between stages also lets you fan a busy stage out to multiple competing consumers, absorbing bursts and recovering cleanly from failures.
Your four-filter pipeline receives 8 images/s, but only 3 come out the end. The queue in front of the second filter keeps growing; the other queues are nearly empty. What should you do first?
Every pipe has a cost. Each hop serializes an item, writes it to a queue and reads it back, and big payloads like images usually travel as a reference to blob storage rather than inside the message (a claim check). If each filter does only a millisecond of work, the hops become most of the cost. Debugging gets harder too: one item's story is now spread across several machines' logs, so stamp a correlation ID on every item at the start and have every filter log it.
When to use it
Pipes and filters fits naturally when a task is a clear sequence of distinct steps that operate on a stream of data — ETL jobs, image and video processing, log enrichment, or any workflow where stages have different resource appetites and you want to scale them independently.
It's overkill for a quick task that runs in a few milliseconds inside one process; the pipes themselves add latency and operational overhead. It's also a poor fit when the steps are tightly interdependent and need to share lots of state, since the whole point is that filters stay isolated. And you'll need to think hard about failures and ordering up front — a stage dying mid-stream is a question the pattern asks you to answer deliberately, not by accident.
A teammate proposes splitting a request handler into five filters joined by queues. Each step takes about 1 ms, and the steps read and write a lot of shared state. What's the strongest objection?