zlacker

>>pul+(OP)
> Vertical scaling — a bigger, exponentially more expensive server

This is in practice not true at all. Vertical scaling is typically a sublinear cost increase (up to a point, but that point is a ridiculous beast of a machine), since you're (typically) upgrading just the CPU and/or just the RAM or just the storage; not all of them at once.

There are instances where you can get nearly 10x the machine for 2x the cost.

>>margin+lu
Disagree- typically vertical scaling is lumpy, and even worse- CPU and RAM upgrades are typically not linear, because you're limited by the number of slots/sockets and the manufacturers intentionally charge higher (expoentially) prices for the largest RAM and fastest CPUs.

>>dekhn+UG
Kafka is also a system that can make pretty good general use of more CPUs and more storage, but doesn't have much need for RAM. Tying the CPU and RAM together whether by CPU model or cloud vendor offerings is annoying if you're trying to scale only vertically.

>>moreli+GK
Kafka can keep a decent bit of data in RAM using file system pages. Often times you end up wasting CPUs on kafka nodes, not memory i think.

https://docs.confluent.io/platform/current/kafka/deployment....

>>defend+mu1
I find that if you are seeking lots of consumers around large topics no amount of RAM is really sufficient, and if you are mostly sticking to the tails like a regular Kafka user, even 64GB is usually way more than enough.

CPU isn't usually a problem until you start using very large compactions, and then suddenly it can be a massive bottleneck. (Actually I would love to abuse more RAM here but log.cleaner.dedupe.buffer.size has a tiny maximum value!)

Kafka Streams (specifically) is also configured to transact by default, even though most applications aren't written to be able to actually benefit from that. If you run lots of different consumer services this results in burning a lot of CPU on transactions in a "flat profile"-y way that's hard to illustrate to application developers since each consumer, individually, is relatively small - there's just thousands of them.