zlacker

>>pul+(OP)
Must be something about Kafka to attract these kind of explanations. Another one few months back was a children's book on Kafka [1] . For me it just look like solution looking for actual problems.

I wonder if Kafka represents an existential angst in these Kubernetized Microservice times. Or is it more simply I am just too dumb to learn and use this shit correctly.

1. https://news.ycombinator.com/item?id=27541339

>>geodel+nkb
When you are wondering whether you might need Kafka, it is certain that you don't need it.

But there are times when you have a problem, and amongst the possible solutions is Kafka.

I've come across Kafkaesque problems only three times in the last seven years: a hosting platform that had to parse logs of over 700 WordPress sites for security and other businesslogic. Putting all events of a financial app backend into datalakes and filtering and parsing all openstreetmap changesets live.

>>berkes+yob
I work in the oil and gas industry where legacy systems runs on their last breath. Kafka is a fantastic tool and solves a shit ton of problem. We have millions of sensors on an offshore installation, these all send data into kafka, where we generate events on new topics from different timeseries. Other data services consume these topics and get data updated in near realtime.

No more daily SQL dumps from offshore to onshore and big batch procedures to genereate outdated events.

>>olavgg+ECb
Sounds like you have Serious Problems, for which Kafka is a very good solution.

For me, Kafka sits in the same area of solutions as Kubernetes, Hadoop clusters, or anything "webscale": you don't need it. Untill you do, but by then you'll (i) have Serious Problems which such systems solve and (ii) the manpower and budgets to fix them.

With which I don't mean to avoid Kafka at all costs. By all means, play around with it: if anything, the event-driven will teach you things that make your a better Rails/Flask/WordPress developer if that is what you do.