zlacker

[return to "Choose Postgres queue technology"]
1. mianos+It[view] [source] 2023-09-25 01:21:25
>>bo0tzz+(OP)
Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern.

Having transactions is quite handy.

https://wiki.postgresql.org/wiki/SkyTools

I did a few talks on this at Sydpy as I used it at work quite a bit. It's handy when you already have postgresql running well and supported.

This said, I'd use a dedicated queue these days. Anything but RabbitMQ.

◧◩
2. reuben+oz[view] [source] 2023-09-25 02:24:53
>>mianos+It
> Anything but RabbitMQ.

Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

◧◩◪
3. boyter+uO[view] [source] 2023-09-25 05:36:57
>>reuben+oz
I used RabbitMQ for a while and nothing but problems.

Admittedly I probably shouldn't have used it the way I did. I dumped many millions of tasks into it, then fanned out processes pulling from that queue that took a variable amount of time to run. Some ran in seconds, some hours.

I had picked RabbitMQ because I wanted that queue to be durable and resist workers dying, or being restarted. However long lived tasks like this is not really what it was designed for (in my opinion). I kept running into issues where it would take a long time to restart, and stop answering connections and need a restart to continue. I ended up having to write monitoring code to check for this and handle it to have it be slightly reliable.

Im sure it works well for smaller short lived messages, but considering the issues I bumped into I would be hesitant to try it. Id probably reach to redis first with wrappers allowing me to swap out to any other queue as required first.

[go to top]