zlacker

One important drawback of SQS is that it's eventually consistent, you can read the same message twice from different workers. Nevertheless we keep using it with additional checks when it's critical, it's still the cheapest solution by maintenance.

>>varela+(OP)
> you can read the same message twice from different workers

If you have a short visibility timeout that it appears back on the queue before you delete it... Sure.

>>philli+c1
No. By default SQS uses `at-least-once delivery` system. It's distributed and when you connect to API one SQS node that you working with could have different state comparing to another, so you may fetch the same message twice by 2 separate workers even with big visibility timeout.

>>varela+(OP)
Making the processing of a message idempotent is the ideal way to handle that limitation

>>lclark+z1
That's not always possible. Thus a big no-no for SQS if that's what you need. Concluding: SQS cannot replace RabbitMQ in all usecases.

>>cies+G5
If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer.

https://bravenewgeek.com/tag/amazon-sqs/

>>cies+G5
Exactly once delivery =/= exactly once processing.

>>tybit+h7
> Exactly once delivery is impossible other than with at least once delivery

Can you explain this? Don't many applications deliver once and only once via locking? It's obviously easier as an application developer to say "I will only get this once" and accept losing messages than dealing with idempotence particularly in distributed services.

>>nkozyr+Sd
This is what I mean.

>>nkozyr+Sd
What if your server dies right as it's about to release that lock and mark the job done? How will you ever know if it was completed or not?

>>nkozyr+Sd
Locking is no longer 100% reliable as soon as you have horizontal distribution of the same data over multiple nodes (for redundancy, so you can guarantee delivery) instead of sourcing from e.g. a monolithic rdbms. Eventual consistency is the model for a whole lot of distributed systems, e.g. S3 or Mongo. The CAP theorem applies to more than just databases, so MQs tend to use eventual consistency as well, which looks like at-least-once and not guaranteed exactly-once delivery.

While dealing with at-most-once delivery is easier in isolation as an application developer on the consumer side, dealing with lost messages is in practice MUCH MUCH harder on the producer side than idempotent handling where required on the consumer side. You end up building elaborate mechanisms for locking and retries and receipt validation which can all fail.

Just think about emails, there are a lot of situations where you can't be 100% certain whether the other side received the message or not. For something low-priority it may be better to ignore partial failures, but if it's important it may be better to send a second message to guarantee delivery. If you're modeled on never sending the same message twice AND the messages matter, you're in trouble.

>>varela+(OP)
FIFO queues should fix that issue...

FIFO (First-In-First-Out) queues are designed to enhance messaging between applications when the order of operations and events is critical, or where duplicates can't be tolerated

>>varela+(OP)
Amazon's hand an answer to that with FIFO queues for the better part of years, but even without a queue based system you probably still need some notion of idempotence in the actions your system takes.

Systems "without queues" have just eschewed one queue for N queues, where N is the number of clients buffering actions for retries at any given moment.

>>varela+(OP)
Implementing a queue that does exactly once delivery is a very hard problem and comes with its own trade-offs. Mostly around scalability and reliability.

>>akvadr+ey
And it comes with the trade off of 3000 (batches) messages per second (and I bet some reduced potential availability). Sure, it’s a lot of messages for most people but non-FIFO SQS is one of the truly limitless services offered by AWS. No limits to raise, no soft maximums AWS staff ask you not to exceed. Just limitless scaling.