zlacker

>>DonHop+(OP)
The analogy helped me understand, but in factorio belts are required primarily because everything takes physical space. What are real world use cases for kafka? Is it primarily if your data is too much to simply store in a db?

>>chrisc+GC
Push architectures put all of the control on the producer, but become quite painful in the face of an unreliable consumer. Pull architectures put all of the control on the consumer, but become quite painful in the face of an unreliable producer.

Message queues allow the producer to think it's participating in a push architecture, while the consumer believes it's a pull-based arch. Because the message queue itself does very little work, it gets to play a very well-behaved consumer insofar as the producer is concerned, while behaving like a well-behaved producer for the consumer. The queue's buffering then allows both the producer and the consumer to misbehave to some degree while the queue itself keeps the illusion of good behaviour.

Kafka specifically falls under the category of log-oriented message queues, which are eminently useful for distributing any workload that looks like "tail a log and process each line as it comes in" across a large number of nodes.