The problem events actually solve
Event-driven architecture is a solution to coupling, not to scale. When six systems need to know an order was placed, the synchronous version means the order service calls six things, knows about six things, and fails when any of them is down. The event version means the order service announces what happened and stops caring who listens.
That is a genuine and valuable inversion. The order service ships without knowing the analytics pipeline exists. A new consumer subscribes without a change upstream. If you find yourself adding another call to a service that already calls four things after finishing its real work, coupling is your problem and events are the right tool.
What Kafka gives you over a plain queue
The distinguishing feature is the retained, replayable log. A traditional queue deletes a message once consumed; Kafka keeps it for a configured retention, so a new consumer can read history from the beginning and a broken consumer can be fixed and replayed. That property is what makes it suitable as a system of record for events rather than merely transport.
If you do not need replay, multiple independent consumer groups over one stream, or ordering within a partition, you probably do not need Kafka. A managed queue is dramatically less operational work. Be honest about which of those three you actually require, because they are the entire justification for the added complexity.
The operational cost nobody budgets
Running Kafka well means understanding partitions, consumer group rebalancing, offset management, retention policy, and what happens when a consumer falls behind. Managed services remove server administration but not the conceptual load. Your team still has to reason about why a rebalance storm is stalling throughput at 3am.
Then there is schema management. Events are a contract between systems that deploy independently, so a producer adding a required field breaks every consumer. That needs a schema registry and a compatibility policy from day one. Teams that skip it discover the need during their first production incident, which is the expensive way to learn it.
Eventual consistency is a product decision
The moment you go event-driven you accept that different parts of the system disagree for a while. A user updates their profile and the search index catches up seconds later. That is usually fine — but it is a product decision, not an implementation detail, and it must be made with the people who will field the support tickets.
Some flows genuinely cannot tolerate it. If a user changes a setting and is immediately shown a screen rendered from a lagging read model, they will believe the save failed and do it again. Either keep those flows synchronous or design the UI to read its own writes. Pretending eventual consistency is invisible to users is how event-driven systems earn a reputation for flakiness.
Debugging distributed flows
In a synchronous system a stack trace tells you what happened. In an event-driven one a business process is a chain of independent handlers with no shared call stack. Without correlation IDs propagated through every event and logged by every consumer, answering what happened to order 12345 becomes archaeology across several services.
Budget for that up front: correlation IDs, structured logs, and a dead-letter queue with an actual process for inspecting and replaying it. A dead-letter queue nobody monitors is just a place where failed business transactions go to be forgotten, and I have seen one hold months of silently dropped work.
What I reach for first now
For most teams under real time pressure, the outbox pattern with a managed queue covers a large share of what they wanted from Kafka: write the event to a table in the same transaction as the business change, relay it to a queue, and get reliable at-least-once delivery without a distributed log to operate. Less powerful, far less work.
I reach for Kafka when there are genuinely many independent consumers of the same stream, when replay is a requirement rather than a nice-to-have, or when throughput is high enough that per-message queue overhead matters. Those cases are real. They are just less common than the number of Kafka clusters in the wild suggests.
Key takeaways
- Events solve coupling, not scale — reach for them when one service must notify many
- Kafka's justification is replay, multiple consumer groups, or partition ordering; without those a managed queue wins
- Budget for a schema registry and compatibility policy from day one, not after the first breaking change
- Eventual consistency is a product decision — design read-your-own-writes where users would notice
- Propagate correlation IDs and actively monitor the dead-letter queue, or failed transactions vanish silently
- The outbox pattern with a managed queue covers most use cases at a fraction of the operational cost
Conclusion
Kafka is excellent technology frequently adopted for the wrong reason — as a scale story when the real problem was coupling, or as a default because it is what serious companies use. Make the decision on replay, consumer count and ordering. If none apply, take the simpler path and spend the saved effort on the product.
Enjoyed this article?
Vivek Kumar Singh
Technical Expert · Full Stack Cloud Engineer · Tokyo, Japan
