zlacker

Understanding Kafka with Factorio (2019)

submitted by pul+(OP) on 2023-07-13 15:00:16 | 196 points 104 comments
[view article] [source] [go to bottom]

NOTE: showing posts with links only show all posts
◧◩◪◨
14. moreli+aJ[view] [source] [discussion] 2023-07-13 17:40:28
>>Sohcah+bG
https://aws.amazon.com/msk/pricing/ prices scale linearly with CPU beginning with m5.large, and I wouldn't really want to run a production Kafka on anything less than m5.xlarge. (They do at least keep linearly scaling all the way up.) Speculating wildly, I could probably have run some of our real clusters on the equivalent of a 8xlarge, but of course 32 core systems were not widely available at that time. The cluster I run today, even a hypothetical 48xlarge would struggle.

YMMV for non-managed stuff, but really, you can only bump cores like 3 times realistically, 4 if you started really shitty, before you start getting into special pricing brackets.

◧
17. dang+CN[view] [source] 2023-07-13 17:55:29
>>pul+(OP)
Related:

Understanding Kafka with Factorio (2019) - >>29304414 - Nov 2021 (72 comments)

Understanding Kafka with Factorio - >>20362179 - July 2019 (84 comments)

(Reposts are fine after a year or so; links to past threads are just to satisfy extra-curious readers)

◧◩◪◨⬒
21. margin+MV[view] [source] [discussion] 2023-07-13 18:25:55
>>fluori+uR
I disagree with this definition of horizontal scaling. If you're moving to a bigger computer rather than more computers, then you're scaling vertically and not horizontally.

(and fwiw, wikipedia agrees with this definition: https://en.wikipedia.org/wiki/Scalability#Horizontal_(scale_... )

◧◩◪◨⬒
53. the_af+2s1[view] [source] [discussion] 2023-07-13 20:54:55
>>bee_ri+1c1
Your comment is on point. However...

... if you read Jay Krep's introduction to logs [1] (in the Kafka sense of logs) you can see that while he has a nice sense of humor [2], he felt pretty good about the Log abstraction and about Kafka. In no sense do I get the feeling he thought he was creating a kludge or something bad -- or "kafkaesque". Judging by the article, it might as well been named Apache Tolstoy!

[1] https://engineering.linkedin.com/distributed-systems/log-wha...

[2] "'Each working data pipeline is designed like a log; each broken data pipeline is broken in its own way.' — Count Leo Tolstoy (translation by the author)"

◧◩◪◨
56. defend+mu1[view] [source] [discussion] 2023-07-13 21:07:22
>>moreli+GK
Kafka can keep a decent bit of data in RAM using file system pages. Often times you end up wasting CPUs on kafka nodes, not memory i think.

https://docs.confluent.io/platform/current/kafka/deployment....

◧◩◪
104. dang+cR7[view] [source] [discussion] 2023-07-15 20:05:10
>>keithn+Kb2
After a year or so: https://news.ycombinator.com/newsfaq.html
[go to top]