zlacker

>>bo0tzz+(OP)
Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world.

Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these same problems. I wrote a bit about them here in our docs: https://riverqueue.com/docs/transactional-enqueueing

Ultimately I want to help engineers be able to focus their time on building a reliable product, not chasing down distributed systems edge cases. I think most people underestimate just how far you can get with this model—most systems will never outgrow the scaling constraints and the rest are generally better off not worrying about these problems until they truly need to.

Please check out the website and docs for more info. We have a lot more coming but first we want to iron out the API design with the community and get some feedback on what features people are most excited for. https://riverqueue.com/

>>bgentr+9l
I don't know why people even use libraries in those languages - assuming you stick to a database engine you understand well then the (main)-database-as-queue pattern is trivial to implement. Any time spent writing code is quickly won back by not having to debug weird edge cases, and sometimes you can highly optimize what you're doing (for example it becomes easy to migrate jobs which are data-dominated to the DB server which can cut processing time by 2-3 orders of magnitude).

It's particularly suited to use cases such background jobs, workflows or other operations which occur within your application and scales well enough for what 99.9999% of us will be doing.

>>radica+891
I thought the first rule of queues was never use a database as a queue.

>>AtlasB+iS5
The "problem" is that a database as a queue doesn't infinitely scale, but nothing infinitely scales. If you're starting to reach the limits of Postgres-as-a-queue then you've either done something very wrong or very right to get to that point.

It's probably more correct to say that the engineering effort required to make a Postgres-as-a-queue scale horizontally is a lot more than the engineering effort required to make a dedicated queueing service scale horizontally. The trade-off is that you're typically going to have to start scaling horizontally much sooner with your dedicated queuing service than with a Postgres database.

The argument for Postgres-as-a-queue is that you might be able to get to market much quicker, which can be significantly more important than how well you can scale down the track.