Kafka’s own introduction says it plainly. Producers and consumers are “fully decoupled and agnostic of each other”, and producers never need to wait for consumers. A producer writes an event and forgets about it. Consumers subscribe to what they need. No one waits, and teams ship without meetings.
That holds until the first time a topic has to change. Then the producer waits for every consumer it has. Big companies have a platform team for that. In a small or medium team, the platform team is you.
I’ve lived this at my day job, on a system I can’t describe in public, so the names here are generic and the sequence is exact. A topic’s shape needs to change. Adding an optional field is the one change that costs nothing. Everything else, a rename, a removed field, a type change, a new partition key, is a coordination mess, because every consumer is welded to the shape it reads. So you create a v2 topic.
You re-broadcast history into it, so the consumers that need the past can have it. Then every consumer switches topics, on its own team’s schedule, one sprint here and two there. The old topic stays up until the last one moves, and so does the old code that produces the old shape. Then you wait to delete both, and one day someone asks in a meeting whether anyone is still reading v1. The broker can list the consumer groups that commit offsets there, and nothing about the readers that don’t, so the answer is a survey of teams.
That is the meeting the architecture promised to delete. The producer team waits on every consumer team, the consumer teams wait on the re-broadcast, and a change that would have been one migration in one pull request becomes a project.
Decoupling was in the brochure. Eventual coupling is what runs in production. The teams are as coupled as they were with a shared function call, and now the coupling has a delay and no stack trace.
The missing stack trace is the second cost. When the output is wrong, the trail runs through a producer’s logs, a topic’s offsets, a consumer group’s lag, and a second consumer’s logs, owned by three teams in three dashboards. One bug becomes a distributed debugging session.
It doesn’t end at schema changes, because topics don’t move at the same pace either. One topic carried users. A second carried events about those users. One consumer needed both. Kafka keeps events in order inside a partition and promises nothing across topics, so the event about a user arrived before the user did. We parked the early events in a waiting table until their user showed up, with a timeout and a job that expired the orphans. Then the waiting table became a Flink job, a stream processor that can join topics, which means we ran a second distributed system to do what one JOIN does inside a database. It cost us an incident along the way, one I can’t describe here.
A database does all of this without a meeting. A schema change is a migration in one pull request, and every reader in the codebase moves with it. Two facts about one user are two tables and a join. A bug is one stack trace. Work to do is a jobs table, and the consumer that’s behind is where processed is false. Nothing rebalances, and there is no history to re-broadcast.
Before you put a topic between two teams, ask what happens the first time its shape has to change. If the answer includes the other team’s sprint, you haven’t decoupled anything. You’ve scheduled the coupling for later.