zlacker

Push architectures put all of the control on the producer, but become quite painful in the face of an unreliable consumer. Pull architectures put all of the control on the consumer, but become quite painful in the face of an unreliable producer.

Message queues allow the producer to think it's participating in a push architecture, while the consumer believes it's a pull-based arch. Because the message queue itself does very little work, it gets to play a very well-behaved consumer insofar as the producer is concerned, while behaving like a well-behaved producer for the consumer. The queue's buffering then allows both the producer and the consumer to misbehave to some degree while the queue itself keeps the illusion of good behaviour.

Kafka specifically falls under the category of log-oriented message queues, which are eminently useful for distributing any workload that looks like "tail a log and process each line as it comes in" across a large number of nodes.

>>pdpi+(OP)
This. With the slight nuance that it works better if you don't think of it as queues, since that suggests messages being removed when processed.

The big win when introducing Kafka is that (apart from the schema) it completely decouples producers and consumers of messages. That in turn reduces the required coordination between independent teams within an organisation.

>>pdpi+(OP)
It’s a good theory, and definitely something you should be aware of but... in my experience if the producer fails, you get nothing (of course), and if the consumer fails, the messages just back up until the queue overflows and of course if the queue itself fails - which can happen due to hardware or network as well as software issues - you have a new point of failure that’s unrelated to the producers or consumers. So the whole thing is only as good as the responsiveness of your monitoring team... which is effectively the case with synchronous message delivery, only with the added complexity of troubleshooting the queue in the middle.