zlacker

I like how GCP cloud tasks reverses the model. Instead of workers pinging the server asking for work, have the queue ping the worker and the worker is effectively a http endpoint. So you send a message to the server, it queues it and then pings a worker with the message.

https://cloud.google.com/tasks/docs/dual-overview

>>latchk+(OP)
Ooh, that's kind of interesting. Am I reading this right that it holds the HTTP connection open for up to thirty minutes waiting for the work to complete? That's kind of wild.

>>donatj+A
Indeed. If you're hitting AppEngine or GCP Functions, they auto scale workers up for you to manage long running tasks. Ideally though, you finish as quickly as possible by breaking the work down into more tasks. That way, you can parallelize as much as possible.

It is all configurable, but I've scaled up to hundreds of workers at a time to blast through tasks and it wasn't expensive at all.

Workers being an HTTP endpoint makes them super easy to implement and even better... write tests for.

>>latchk+(OP)
I love Task Queues. We are using them extensively. Also, they give you deduplication for free and a lot of other nice features like delayed tasks storing tasks for up to 30 days extremely detailed rate limits etc.

>>lysecr+a3
GCP is really under rated in this regard.

Are there any open source implementations of Task Queues? It feels like something that has been missing for years.

>>latchk+(OP)
Just like using JMS, MSMQ or similar queues, I fail to see what is so great about it.

>>latchk+A3
Yea, this is the only thing I don't like about them, that I can't test them locally.

More generally, is there something like a "on prem cloud" which just replicates say Cloud Tasks (but also other Cloud Apis) using local compute as well as say a local db. For testing / development this would be very cool.

>>lysecr+Nn1
I implemented my tasks as cloud functions, so I just test Tasks the same way I tested functions... by calling the handler function directly.

>>pjmlp+G7
JMS is a specification, not an implementation.

>>latchk+mv1
Playing pedantic?

>>pjmlp+XW2
Ok, if you can't see what is so great about it, then I'd suggest spending some time in the GCP documentation to figure it out.

>>latchk+043
Likewise there are plenty of JMS implementations to play around.

>>pjmlp+xU4
Sure. Name one implementation that does exactly what Cloud Tasks does.