I wonder if Kafka represents an existential angst in these Kubernetized Microservice times. Or is it more simply I am just too dumb to learn and use this shit correctly.
But there are times when you have a problem, and amongst the possible solutions is Kafka.
I've come across Kafkaesque problems only three times in the last seven years: a hosting platform that had to parse logs of over 700 WordPress sites for security and other businesslogic. Putting all events of a financial app backend into datalakes and filtering and parsing all openstreetmap changesets live.
Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations.
What alternatives would you be looking at?
* Just keep your architecture a monolith. You'll do fine the majority of the cases.
* Event-sourcing doesn't require Kafka clusters. Nor do event-driven setups. You don't need complex tooling to pass around strings/json-blurps. An S3 bucket or a Postgresql database storing "Events-as-json" is often fine.
* Postgres can do most of what you need (except for the "webscale" clustering etc)[0] in practice already.
* Redis[1]
My main point is that while Kafka is a fantastic tool, you don't need that tool to achieve what you want in many cases.
> It seems as good a way as any to decouple systems
IMO relying on a tool to achieve a good software design, rather than design-patterns, is a recipe for trouble. If anything, because it locks you in (do you suddenly get a tightly coupled system if you remove Kafka?) or because its details force you into directions that don't naturally fit your domain or problem.
--
[0] https://spin.atomicobject.com/2021/02/04/redis-postgresql/ [1] https://redis.com/redis-best-practices/communication-pattern... etc.