zlacker

>>anonu+(OP)
Strong disagree on using a database as a message queue. This article[0] covers many of the reasons why. Summary: additional application complexity and doesn't scale well with workers.

0. https://www.cloudamqp.com/blog/why-is-a-database-not-the-rig...

EDIT>> I am not suggesting people build their own rabbitmq infrastructure. Use a cloud service. The article is informational only.

>>daenz+X9
I'm increasingly of the opinion that relational databases are absolutely the right way to build queue systems for most projects.

One of the biggest advantages comes when you start thinking about them in terms of transactions. Transactional guarantees are really useful here: guarantee that a message will be written to the queue if the transaction commits successfully, and guarantee that a message will NOT be written to the queue otherwise.

https://brandur.org/job-drain describes a great pattern for achieving that using PostgreSQL transactions.

>>simonw+jf
The transaction feature seems nice but how often is your application dropping queue messages because something happened between tx.commit() and queue.send(msg)? My experience has been that this is not an issue.

>>daenz+9h
If you're big enough to worry about the scalability of Postgres, you're big enough to experience this failure fairly often IMO.

>>peterh+cj
Scalability was the second of two concerns I listed. The first was additional application complexity that real message queues hide from you by virtue of being a system built for that usage pattern.

>you're big enough to experience this failure fairly often IMO

Please explain how? You would either have to suffer from frequent network connectivity issues that affects only your db and not your queue, or your process must be mysteriously dying in the microseconds between those 2 operations. Either of those cases are not something I would consider things that happen "fairly often," even if you were processing trillions of messages per day.

In my experience, the vast majority of message processing failures happen at the worker level.

>>daenz+Gj
If the queue goes down you end up updating the db without enqueuing a job and now an engineer needs to go in and re enqueue the missing jobs manually.