zlacker

[return to "River: A fast, robust job queue for Go and Postgres"]
1. latchk+Gc[view] [source] 2023-11-20 16:58:22
>>bo0tzz+(OP)
If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0].

It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution.

Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not suggesting that they use the GCP tasks product. That said, I would have just used GCP tasks too (assuming the usecase dictated it, fantastic and rock solid product.)

[0] https://cloud.google.com/tasks

◧◩
2. brandu+8g[view] [source] 2023-11-20 17:10:41
>>latchk+Gc
There's a lot to be said about the correctness benefits of a transactional model.

The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for. In practice, this leads to a lot of uncertainty around the edges, and operators having to reconcile things manually.

There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.

[go to top]