zlacker

> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering.

Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

>>hipade+(OP)
as far as I'm aware the most popular job queue library in elixir depends on postgres and has performance characteristics that cover the vast majority of background processing needs I've come across.

I wonder maybe if you've limited yourself by assuming relational DBs only have features for relational data. That isn't the case now and really hasn't been the case for quite some time now.

>>hipade+(OP)
Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.

>>hipade+(OP)
> I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

I'm also very familiar with jobs and I have used the usual tools like Redis and RMQ, but I wouldn't make a blanket statement like that. There are people using RDBS as queues in prod so we have some counter-examples. I wouldn't mind at all to get rid of another system (not just one server but the cluster of RMQ/Redis you need for HA). If there's a big risk in using pg as backend for a task queue, I'm all ears.

>>qaq+k3
Probably order of magnitude more or perhaps a multiple of that depending on the hardware and design.

In theory an append-only and/or HOT strategy leaning on Postgres just ripping through moderate sized in-mem lists could be incredibly fast. Design would be more complicated and perhaps use case dependent but I bet could be done.

>>Rapzid+2A2
Yep that's why I specifically mentioned "fairly modest instance" on reasonably fast box you can get magnitude more. You can partition the tasks table to reduce the number of rows skip locked has to run through to grab next task.