zlacker

[return to "Choose Postgres queue technology"]
1. donatj+lG[view] [source] 2023-09-25 03:41:24
>>bo0tzz+(OP)
We have our own queue, because it was easy, fun and has been exceedingly reliable above all else. Far moreso than other things we had tried. Cough Gearman cough SQS cough

One endpoint accepts work to a named queue, writes it to a file in an XFS directory. Another locks a mutex, moves the file to an in progress directory and unlocks the mutex before passing the content to the reader. A third and final endpoint deletes the in progress job file. There is a configurable timeout, after which they end up at a dead letter box. I am simplifying only a little bit. It's a couple hundred lines of Go.

The way this is set up means a message will only ever be handed to one worker. That simplifies things a lot. The workers ask for work when they want it, rather than being constantly listening.

It took a little tuning but we process a couple billion events a day this way and it's been basically zero maintenance for almost 10 years. The wizards in devops even figured out a way to autoscale it.

◧◩
2. amerin+JH[view] [source] 2023-09-25 03:59:41
>>donatj+lG
Sounds neat. Do you have the go code anywhere for folks to poke at?
◧◩◪
3. donatj+4I[view] [source] 2023-09-25 04:04:34
>>amerin+JH
Nah, afraid not. I wish.

We always wanted to open source it, but we got bought out by a big and very IP protective company before we got the chance.

[go to top]