zlacker

>>pul+(OP)
Must be something about Kafka to attract these kind of explanations. Another one few months back was a children's book on Kafka [1] . For me it just look like solution looking for actual problems.

I wonder if Kafka represents an existential angst in these Kubernetized Microservice times. Or is it more simply I am just too dumb to learn and use this shit correctly.

1. https://news.ycombinator.com/item?id=27541339

>>geodel+nkb
Sometimes I wonder if I'm the crazy one. Kafka seems to me to be the only sensible foundational datastore out there: it can maintain and propagate a log with all the properties you would want a datastore to have. Relational database seem to be a crazily overengineered solution in search of a problem, with incredibly poor reliability properties (essentially none of them are true master-master HA out of the box, and they tend to require significant compromises to make them so) to boot.

>>lmm+ltb
> Relational database seem to be a crazily overengineered solution in search of a problem

I mean, whoever in their right mind would want to:

- have a snapshot of data

- query data, including ad-hoc querying

- query related data

- have trasactional updates to data

When all you need is an unbounded stream of data that you need to traverse in order to do all these things.

>>dmitri+UAb
> - have a snapshot of data

Being able to see a snapshot is good, and I would hope to see a higher-level abstraction that can offer that on top of something Kafka-like. But making the current state the primary thing is a huge step backwards, especially when you don't get a history at all by default.

> - query data, including ad-hoc querying

OK, fair, ad-hoc queries are one thing that relational databases are legitimately good at. Something that can maintain secondary indicies and do query planning based on them is definitely useful. But you're asking for trouble if you use them in your live dataflow or allow ad-hoc queries to write to your datastore.

> - have trasactional updates to data

I do think this one is genuinely a mistake. What do you do when a transaction fails? All of the answers I've heard imply that you didn't actually need transactions in the first place.

>>lmm+ACb
You think transactional updates are a mistake? I guess you’ve never had to worry about the integrity of your data?

Bank -> debit card purchase -> perform all required database work in a transaction -> transaction fails -> decline debit card purchase

Without transactions, in this scenario, maybe the debit card transaction fails but money is still taken out of your account? Doesn’t sound very pleasant.

>>irl_ch+qDb
Database transactions would be useless for banks - you'll notice that if you try to pay while not having enough money in your account, the attempt doesn't simply disappear. What banks actually do is something akin to what Kafka-based systems do - the attempt to charge is recorded in a leger, and then fulfilment of that happens asynchronously.

>>lmm+lFb
That's seriously one of the worst takes I've seen on this website.

>the attempt to charge is recorded in a leger

Hint: how do you think this attempt is recorded and fulfilled? Or, do you think "it's just appended" and bank recalculates your balance from scratch every time you spend 1$ on coke can?

Only bank I've heard of that's not using traditional relational database for ledger is Monzo [1] - but they still use Cassandra's transactions.

[1] https://www.scaleyourapp.com/an-insight-into-the-backend-inf...

>>KptMar+sKb
> Hint: how do you think this attempt is recorded and fulfilled? Or, do you think "it's just appended" and bank recalculates your balance from scratch every time you spend 1$ on coke can?

That's how the bank I worked with did it. Of course there was caching in place so we didn't actually recompute everything every time, but the implementation of that was a lot closer to "commit a kafka offset" than an RDBMS-style transaction. (E.g. we didn't overwrite the "current balance" in-place, we appended a new "current balance as of time x").

>>lmm+pLb
every large ingest app i have worked on had something akin to kafka, from raw 3d seismic broadcast via sat, to RF tower motion detection, to carrier grade cellular billing. common denominator was a replayable ingest queue. yes, kafka is a "great" idea. however, it is not a replacement for querying.