zlacker

USE. ADVISORY. LOCKS.

Do not use SKIP LOCKED unless it is a toy/low throughout.

Row locks require transactions and disk writes.

Advisory locks require neither. (However, you do have to stay inside the configurable memory budget.)

>>pauldd+(OP)
Maybe it's changed in the last year or so, but from benchmarking and writing / running queue software for Postgres - SKIP LOCKED was/is significantly faster. Is that different for MySQL?

>>pauldd+(OP)
Pretty common advice for scaling Postgres is to deploy pgbouncer in transaction mode in front of it to handle connection pooling.

Advisory locks don’t work in this setup (and will start behaving in strange ways if you do try to use them.) Something to consider if you go this route.

>>mikeoc+i5
Transaction-scoped advisory locks are very much a thing too.

>>pauldd+(OP)
To do anything safe and interesting you’ll need transactions. Using SKIP LOCKED won’t be your bottleneck, your application will. Job queues are about side effects and the rest of your application needs to keep up.

Oban is able to run over 1m jobs a minute, and the ultimate bottleneck is throttling in application code to prevent thrashing the database: https://getoban.pro/articles/one-million-jobs-a-minute-with-...

>>pauldd+(OP)
Not all use cases are high throughput. That’s not what makes it a toy

>>klysm+Ub
"Toy/low throughput" = "Toy or low throughout"

>>sorent+G9
That's true.

However in the empty poll case, you can avoid a transaction.

>>pauldd+2d
Can you define "low throughput"? I think people have significantly different ideas of what that means.

>>mikeoc+i5
Depends. That has more to do with how your're scaling application servers.

>>djur+EF
Yeah I think people generally reach for “big” tools when this “toy” would work fine for the vast majority of projects.

>>djur+EF
> Can you define "low throughput"?

IDK maybe <1000 messages per minute

Not saying SKIP LOCKED can't work with that many. But you'll probably want to do something with lower overhead.

FWIW, Que uses advisory locks [1]

[1] https://github.com/que-rb/que