Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations.
What alternatives would you be looking at?
Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.
* Just keep your architecture a monolith. You'll do fine the majority of the cases.
* Event-sourcing doesn't require Kafka clusters. Nor do event-driven setups. You don't need complex tooling to pass around strings/json-blurps. An S3 bucket or a Postgresql database storing "Events-as-json" is often fine.
* Postgres can do most of what you need (except for the "webscale" clustering etc)[0] in practice already.
* Redis[1]
My main point is that while Kafka is a fantastic tool, you don't need that tool to achieve what you want in many cases.
> It seems as good a way as any to decouple systems
IMO relying on a tool to achieve a good software design, rather than design-patterns, is a recipe for trouble. If anything, because it locks you in (do you suddenly get a tightly coupled system if you remove Kafka?) or because its details force you into directions that don't naturally fit your domain or problem.
--
[0] https://spin.atomicobject.com/2021/02/04/redis-postgresql/ [1] https://redis.com/redis-best-practices/communication-pattern... etc.
Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close).
Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.
MY problems are so special that my use of Kafka was perfect, but YOURS are trivial and you shouldn’t even consider Kafka.
I consider this a form of gatekeeping of advice on using Kafka.