zlacker

>>pul+(OP)
Must be something about Kafka to attract these kind of explanations. Another one few months back was a children's book on Kafka [1] . For me it just look like solution looking for actual problems.

I wonder if Kafka represents an existential angst in these Kubernetized Microservice times. Or is it more simply I am just too dumb to learn and use this shit correctly.

1. https://news.ycombinator.com/item?id=27541339

>>geodel+nkb
When you are wondering whether you might need Kafka, it is certain that you don't need it.

But there are times when you have a problem, and amongst the possible solutions is Kafka.

I've come across Kafkaesque problems only three times in the last seven years: a hosting platform that had to parse logs of over 700 WordPress sites for security and other businesslogic. Putting all events of a financial app backend into datalakes and filtering and parsing all openstreetmap changesets live.

>>berkes+yob
Not sure I agree. It seems as good a way as any to decouple systems, asynchronously exchange messages between services, get them into durable storage, exactly once processing, replay semantics etc. I think it should be on the table at least whenever we have two services needing to exchange data.

Maybe a few use cases could be switched out for direct API calls, but I think Kafka hits the sweet spot in many situations.

What alternatives would you be looking at?

>>benjam+eqb
It is as good as any if you have no financial constraints and no technical overhead and time constraints, but everyone does.

Kafka is one of those systems that needs to be justified by out-scaling other solutions that don't come wedded with all its baggage.

>>weego+Erb
What would you say is its baggage?

>>hansbo+jsb
For me, the baggage is mostly the complexity of the service. With that comes monitoring, maintenance, tuning, debugging and troubleshooting.

Lessened somewhat with SaaS products like Amazon Kinesis (technically not a Kafka, but close).

Another "baggage" is that an event-driven setup is eventual-consistent -and async- by nature. If your software already is eventual-consistent, this is not a problem. But it is a huge change if you come from a blocking/simple "crud" setup.

>>berkes+nUb
I second Kafka having massive operational overhead. It's a burden and is killing any support for it within our org.