zlacker

[return to "River: A fast, robust job queue for Go and Postgres"]
1. latchk+Gc[view] [source] 2023-11-20 16:58:22
>>bo0tzz+(OP)
If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0].

It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution.

Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not suggesting that they use the GCP tasks product. That said, I would have just used GCP tasks too (assuming the usecase dictated it, fantastic and rock solid product.)

[0] https://cloud.google.com/tasks

◧◩
2. brandu+8g[view] [source] 2023-11-20 17:10:41
>>latchk+Gc
There's a lot to be said about the correctness benefits of a transactional model.

The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for. In practice, this leads to a lot of uncertainty around the edges, and operators having to reconcile things manually.

There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.

◧◩◪
3. latchk+sg[view] [source] 2023-11-20 17:11:55
>>brandu+8g
> what if it fails, or what if you're not sure about whether it failed?

This is covered in the GCP Tasks documentation.

> There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.

My post was talking about what I would implement if I was doing my own queue, as the authors were. Not about using GCP Tasks.

◧◩◪◨
4. politi+si[view] [source] 2023-11-20 17:18:06
>>latchk+sg
Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack.

https://brandur.org/job-drain

>>15294722

[go to top]