zlacker

Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

>>dekhn+(OP)
Kafka is also a system that can make pretty good general use of more CPUs and more storage, but doesn't have much need for RAM. Tying the CPU and RAM together whether by CPU model or cloud vendor offerings is annoying if you're trying to scale only vertically.

>>dekhn+(OP)
If they charge these big numbers more it's precisely because they're trying to capture some of the embarrassingly better value you get from vertical scaling. It's a testament to vertical scaling's effectiveness that they _can_ do so.

>>vegabo+2b
Sure, but by doing so they consume the effectiveness?

>>foota+ow
No, because you pay a fixed cost to get higher performance and then benefit through the whole lifetime of the product (I'm assuming you are purchasing rationally and keep your machines loaded at 75% or better, and your software is not egregiously wasteful).

>>moreli+M3
Kafka can keep a decent bit of data in RAM using file system pages. Often times you end up wasting CPUs on kafka nodes, not memory i think.

https://docs.confluent.io/platform/current/kafka/deployment....

>>dekhn+(OP)
With clouds this is not true anymore. They are exactly linear. If you ask for a smaller node they are simply propositioning a chunk of a larger machine anyway.

There is a point where the exponential pricing starts, but that point is way out there than most people expect. Probably ~100CPU, ~1TB RAM, >50Gbps network etc.

>>defend+sN
I find that if you are seeking lots of consumers around large topics no amount of RAM is really sufficient, and if you are mostly sticking to the tails like a regular Kafka user, even 64GB is usually way more than enough.

CPU isn't usually a problem until you start using very large compactions, and then suddenly it can be a massive bottleneck. (Actually I would love to abuse more RAM here but log.cleaner.dedupe.buffer.size has a tiny maximum value!)

Kafka Streams (specifically) is also configured to transact by default, even though most applications aren't written to be able to actually benefit from that. If you run lots of different consumer services this results in burning a lot of CPU on transactions in a "flat profile"-y way that's hard to illustrate to application developers since each consumer, individually, is relatively small - there's just thousands of them.

>>defend+KO
They're linear... because they're charging you rates based on the cost of the large server, divided down into whatever server you provisioned.

Amusingly, for $94K (probably more like $85K after negotiation) you can buy a white box server: Dual Epyc 9000, 96 core/192thread, 3.5GHz, w/ 3TB RAM, 240T of very fast SSD, and a 10G NIC. The minimum config, Dual Epyc 9124, 16core/32thread, 64GB RAM, and only 4TB of storage is $9K (more like $8K after negotiation). That's "only" a factor of 10 in price for 8X CPUs, 48X the RAM, and 60X the storage.

>>dekhn+Qa1
> They're linear... because they're charging you rates based on the cost of the large server, divided down into whatever server you provisioned.

And the reason they do it that way is because it's cheaper. Because the scaling is sublinear up to a good size.