But there are times when you have a problem, and amongst the possible solutions is Kafka.
I've come across Kafkaesque problems only three times in the last seven years: a hosting platform that had to parse logs of over 700 WordPress sites for security and other businesslogic. Putting all events of a financial app backend into datalakes and filtering and parsing all openstreetmap changesets live.
Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations.
What alternatives would you be looking at?
Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.
No more daily SQL dumps from offshore to onshore and big batch procedures to genereate outdated events.
For me, Kafka sits in the same area of solutions as Kubernetes, Hadoop clusters, or anything "webscale": you don't need it. Untill you do, but by then you'll (i) have Serious Problems which such systems solve and (ii) the manpower and budgets to fix them.
With which I don't mean to avoid Kafka at all costs. By all means, play around with it: if anything, the event-driven will teach you things that make your a better Rails/Flask/WordPress developer if that is what you do.
* Just keep your architecture a monolith. You'll do fine the majority of the cases.
* Event-sourcing doesn't require Kafka clusters. Nor do event-driven setups. You don't need complex tooling to pass around strings/json-blurps. An S3 bucket or a Postgresql database storing "Events-as-json" is often fine.
* Postgres can do most of what you need (except for the "webscale" clustering etc)[0] in practice already.
* Redis[1]
My main point is that while Kafka is a fantastic tool, you don't need that tool to achieve what you want in many cases.
> It seems as good a way as any to decouple systems
IMO relying on a tool to achieve a good software design, rather than design-patterns, is a recipe for trouble. If anything, because it locks you in (do you suddenly get a tightly coupled system if you remove Kafka?) or because its details force you into directions that don't naturally fit your domain or problem.
--
[0] https://spin.atomicobject.com/2021/02/04/redis-postgresql/ [1] https://redis.com/redis-best-practices/communication-pattern... etc.
Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close).
Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.
MY problems are so special that my use of Kafka was perfect, but YOURS are trivial and you shouldn’t even consider Kafka.
I consider this a form of gatekeeping of advice on using Kafka.