Parallel Computing and Pancakes: Part I

At Ampersand, we needed to run 200,000 Bayesian models in production to power our viewership forecasts: two hundred thousand models, each needing 5-10 minutes of compute time. If we ran these one at a time, back-to-back, we’d be waiting 4 years for the job to finish.

We didn’t have 4 years to wait. Who does? Our machines had multiple cores¹, but even my 8-core Mac Mini was hopelessly outmatched. In 2026, on AWS, you can rent a computer with 192 cores. We needed to go much bigger than that.

A good way to understand parallel computing is pancakes².

Say you need to feed an entire army. If you’re making pancakes one at a time in a single pan, you’ll be cooking for years. So you upgrade to a griddle, and now you can make eight pancakes at once. That’s faster, but still nowhere near enough to feed an army. So you get more griddles. A hundred chefs at a hundred griddles, each making eight pancakes simultaneously. Suddenly you’re producing food at scale. We needed the computational equivalent of a thousand griddles all fired up at once.

But a thousand griddles brings its own problems. You need enough gas to keep them all hot. Tracking gas for one griddle is easy; tracking it for a thousand is a logistics operation. You need butter for each griddle to keep the pancakes from sticking, but at scale, sometimes you’re going to get a bad batch of rancid butter. And griddles break. Maybe a single griddle only fails once every thousand hours, but when you have a thousand griddles, that means one is breaking every hour. Suddenly you’re spending as much time managing griddles as making pancakes.

The same problems show up in parallel computing. Computers break, networks go down, disks fill up, memory runs out. There’s an old joke in software: once you decide to use parallel computing, you have two problems. Your original problem, and parallel computing.

Of course, these are problems you can only have if you have the computers in the first place. We didn’t have a thousand computers. Most organizations don’t. Fortunately, we had access to AWS. Instead of buying a thousand computers, we could borrow them. They have tens of thousands of servers available, and you can rent them for exactly as long as you need.

Borrowing computers from AWS introduces its own complexity, though. You don’t always rent a whole computer. Often you’re renting half of one, or a quarter, and your job needs to handle running on any of those fractions. It gets worse: not all the computers have the same specifications. Some are recent; some are a year or two old; some are older still. Within each size category, you’ll find different core counts, different clock speeds, different amounts of memory. Every job needs to handle this variability, or you’re constantly fighting your own infrastructure.

Given all this complexity, we needed a framework to manage the work for us. The first tool we reached for was Dask, and it seemed like the right choice. It’s a proper distributed computing framework, battle-tested on large workloads, seemingly designed for our kind of problem. We’d been wrestling with it for weeks, though, trying to get it to scale beyond a few thousand concurrent jobs. The returns were diminishing. So I started asking a different question: not “how do we make Dask work?” but “why isn’t it working?”

The answer, once I looked for it, was straightforward: Dask was too smart for what we needed.

Remember the pancakes? Dask assumes your thousand chefs need to coordinate. Maybe one is making eggs while another makes toast, and they all need to hit the table at the same time. The scheduler tracks dependencies, manages timing, shuttles ingredients between stations. That kind of coordination is genuinely hard, and Dask is built for it.

We didn’t need that. Every chef was just making pancakes, independently, forever. You make the batter. You cook the pancake. You serve it. A thousand chefs in a thousand kitchens could each make pancakes all day, and they’d never need to talk to each other.

Our 200,000 models worked the same way. Computer scientists call this “embarrassingly parallel.” Each model was independent. We just needed to run them all and collect the results. But Dask’s central scheduler and data transfer capabilities were designed for jobs that actually need coordination. For us, they became pure overhead. We couldn’t tell it to simply run everything independently and leave us alone.

This is a trap that’s easy to fall into. You pick the tool that’s designed for your general category of problem, and you assume the difficulty is in configuring it correctly. So you keep tuning and tweaking, convinced you’re one setting away from making it work. But sometimes the tool isn’t wrong because you’re using it incorrectly. It’s wrong because it was built to solve a different, more complex version of your problem, and that extra sophistication is what’s getting in your way.

We needed something with a simpler conceptual model. Not a tool that tried to orchestrate a thousand chefs in concert, but one that could hand each chef a recipe, make sure the kitchen was the right size, and start a new batch if the last one burned. Smart about reliability, but unburdened by coordination. In Part II, I’ll explain how we found it by integrating more closely with AWS itself, using a tool they’d built to work directly with their infrastructure, and how it took us from a workload we couldn’t run at all to one that finished in hours.