Kafka was built at LinkedIn to move activity streams. By the time they wrote it up, it was carrying 7 trillion messages a day across 4,000 brokers. Serious engineering for a serious problem. Their problem.
Here’s the reflex it created. You need background jobs: send the welcome email, retry the webhook, build the nightly report. The tutorial says install Redis, add a queueing library, and if you’re “serious,” RabbitMQ or Kafka. Three systems running before your first job does.
It’s a trap at your scale. And it’s not your fault: the books were written for Google scale, and nobody told you they weren’t talking to you.
Now count your jobs. Not the jobs you might have. The ones you have. A normal product runs a few thousand a day, with rush hours that might touch a few per second. You are about to deploy LinkedIn’s artillery against a queue that fits on a napkin.
The database you already run handles this without noticing. A jobs table and SELECT ... FOR UPDATE SKIP LOCKED is the whole trick, and every serious framework has it packaged. Rails 8 went further: the default queue is now the database. Not for toy apps. HEY, 37signals’ email service, pushes about 20 million jobs a day through it. That’s 230 jobs a second, on the technology the tutorials told you couldn’t handle your fifty.
Do the math. Four thousand jobs a day is one job every twenty seconds. The database queue, proven at 230 jobs a second, clears your whole day in 17 seconds. LinkedIn’s Kafka would clear your year in two hundredths of a second. One of these is a reference class. The other is a costume.
And the database queue gives you something the broker never will: one transaction. Save the order and enqueue its email in the same commit, and there is no crack between systems for a job to fall into. With a separate broker there is always a crack, and the industry’s fix for it is the outbox pattern, which is, read it slowly, a jobs table in your database. The cure for the second system is the first one.
Your jobs become rows. You inspect the queue with a SELECT. You count failures with a WHERE. Your backup backs it up. Nothing new to monitor. One less system that can break.
And half your queue doesn’t even need to be a queue. The welcome email can leave a minute late. The nightly report was never urgent. This is where cron stops being the boring old tool and becomes the discovery of your architecture: a fifty-year-old scheduler, zero dependencies, that has outlived every framework it ever served. One crontab line sweeping the table every minute is real-time enough for most of what a business does, and it breaks less often than a worker fleet. Batch was never the compromise. It was the right size all along.
You’re small. At your scale you get to do less. Take that: it’s a competitive advantage. Simple wins, and when you grow, you adapt.
The queue you’re shopping for is already running. It’s called your database.
You’re allowed to just use it.