zlacker

I'm super interested in a Postgres-only task queue, but I'm still unclear from your post whether the only broker dependency is PostgreSQL. You mention working towards getting rid of the RabbitMQ dependency but the existence of RabbitMQ in your stack is dissonant with the statement 'a conviction that PostgreSQL is the right choice for a task queue'. In my mind, if you are using Postgres as a queue, I'm not sure why you'd also have RabbitMQ.

>>dalber+(OP)
We're using RabbitMQ for pub/sub between different components of our engine. The actual task queue is entirely backed by Postgres, but things like streaming events between different workers are done through RabbitMQ at the moment, as well as sending a message from one component to another when you distribute the engine components. I've written a little more about this here: >>39643940 .

We're eventually going to support a lightweight Postgres-backed messaging table, but the number of pub/sub messages sent through RabbitMQ is typically an order of magnitude higher than the number of tasks sent.

>>abelan+91
Do you find it frustrating that what people basically want is:

(1) you, for free

(2) develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license

(3) in order to have it on RDS

(4) and never hear from you again?

This is a colorful exaggeration. But it’s true. It is playing out with the pgvecto-rs people too.

People don’t want Postgres because it is good. They want it because it is offered by RDS, which makes it good.

>>abelan+91
If it's feasible, having Postgres as the only dependency would greatly simplify deployment and management for smaller scale systems.

Great job so far- The flow-based UI with triggers is killer! AFAIK, this surpasses what Celery includes.

>>doctor+z4
So true.

The advice of "commoditize your complements" is working out great for amazon. Ironically, AWS is almost a commodity itself, and the OSS community could flip the table, but we haven't figured out how to do it.

>>doctor+z4
At least pgvector is financially supported by AWS.

>>fizx+t8
AWS is a commodity, albeit an expensive one. After all, it has competitors like GCP, which some people like me actually prefer.

>>doctor+z4
While I understand the sentiment, we see it very differently. We're interested in creating the best product possible, and being open source helps with that. The users who are self-hosting in our Discord give extremely high quality feedback and post feature ideas and discussions which shape the direction of the product. There's plenty of room for Hatchet the OSS repo and Hatchet the cloud version to coexist.

> develop all the functionality of RabbitMQ as a Postgres extension with the most permissive license

That's fair - we're not going to develop all the functionality of RabbitMQ on Postgres (if we were, we probably would have started with a amqp-compatible broker). We're building the orchestration layer that sits on top of the underlying message queue and database to manage the lifecycle of a remotely-invoked function.

>>abelan+91
That makes sense, though a bit disappointing. One hope of using Postgres as a task queue is simplifying your overall stack. Having to host RabbitMQ partially defeats that. I'll stay tuned for the Postgres-backed messaging!

>>dalber+Lh
Then maybe Procrastinate (https://procrastinate.readthedocs.io/en/main/) is something for you (I just contributed some features to it). It has very good documentation, MIT license, and also some nice features like job scheduling, priorities, cancellation, etc.

>>giovan+Ce
I would say the lock-in is considerable. To avoid the lock-in (maybe you go full Kubernetes on EC2) then it’s ton more work.

>>fizx+t8
Similar to Amazon's Retail fulfillment infrastructure, the AWS supply chain infrastructure is definitely not a commodity.