zlacker

Anybody using graphile-worker[1] in production/heavy load? It looks awesome, and I coded up some simple prototype tasks (email, sms, etc), but question how it truly scales. They claim horizontal scaling is trivial.

> graphile-worker is horizontally scalable. Each instance has a customisable worker pool, this pool defaults to size 1 (only one job at a time on this worker) but depending on the nature of your tasks (i.e. assuming they're not compute-heavy) you will likely want to set this higher to benefit from Node.js' concurrency.

[1] https://github.com/graphile/worker

>>nodeso+(OP)
Graphile Worker maintainer here; keep in mind that postgres is not the ideal location for a job queue, so you’re going to be limited ultimately by postgres’ capabilities. I’ve seen Worker max out at around 10k jobs/second but very much YMMV - you should benchmark it for your expected use case. Personally I’d move to a dedicated job queue if I started having an average of more than 1-2k jobs per second. The majority of systems never hit anywhere near this (we very much cater to the “long tail” of job queue needs).

Regarding the horizontal scalability; that relates to if you have heavy tasks (tasks that take a second or more to execute) - you can use more instances to get higher throughput.

Hope this helps!