Across my three products, background jobs come to a few thousand a day. Rush hour is a few per second. The queue that runs them is a table in the database each product already had.
That’s not what the tutorial says. You need background jobs, the welcome email, the webhook retry, the nightly report, and the tutorial says install Redis, add a queueing library, and if you’re serious, RabbitMQ or Kafka. Three systems running before your first job does.
The reflex has a source. Kafka was built at LinkedIn to move activity streams, and by the time they wrote it up it carried 7 trillion messages a day across 4,000 brokers. The books on queues were written from inside numbers like that, and no cover said who they were for.
So count the jobs you ran yesterday, not the ones you might run some day. If the number looks like mine, you are about to install LinkedIn’s queue for a few thousand rows a day.
The database you already run handles that without noticing. A jobs table and SELECT ... FOR UPDATE SKIP LOCKED is all the queue is, and it comes packaged. Solid Queue in Rails, the database driver in Laravel, Oban in Elixir, River in Go. Rails 8 ships it in the box, in place of Redis and Sidekiq for most people. HEY, 37signals’ email service, pushes about 20 million jobs a day through it. That’s 230 jobs a second, on the technology the tutorials told you couldn’t handle your fifty.
Do the math. Four thousand jobs a day is one job every twenty seconds. A database queue proven at 230 a second clears your whole day in 17 seconds. LinkedIn’s Kafka would clear your year in two hundredths of a second. One of these is a reference class. The other is a costume.
The database queue also gives you something the broker never will, one transaction. Save the order and enqueue its email in the same commit, and there is no crack between systems for a job to fall into. With a separate broker there is always a crack, and the industry’s fix is the outbox pattern, which is, read it slowly, a jobs table in your database.
Your jobs become rows. You inspect the queue with a SELECT and count failures with a WHERE. Your backup backs it up. Nothing new to monitor.
Half the queue doesn’t even need to be a queue. The welcome email can leave a minute late, and the nightly report was never urgent. One crontab line sweeping the table every minute is real-time enough for most of what a business does, and cron, a fifty-year-old scheduler with zero dependencies, breaks less often than a worker fleet.
There is a day the table is the wrong call. When the jobs table is the hottest table in the database and the workers polling it are most of the load, you have reached the broker’s problem, and you’ll know it from the query stats, not from a book. The other case is a consumer that isn’t yours, another team’s system that can’t share your database. Then the broker is the boundary, and it’s earning its keep.
Create the jobs table first. Move to a broker the day the table can’t keep up, and bring the query that proves it.