zlacker

[parent] [thread] 9 comments
1. jedber+(OP)[view] [source] 2022-09-06 03:56:47
Since Redis is the data store, if your Redis instance goes away, do all of your tasks go away too?
replies(2): >>welder+C5 >>belthe+b6
2. welder+C5[view] [source] 2022-09-06 04:52:38
>>jedber+(OP)
Goes away like you delete your Redis VM or restarting the Redis server? Redis persists to disk so your tasks aren't lost.
replies(1): >>jedber+U5
◧◩
3. jedber+U5[view] [source] [discussion] 2022-09-06 04:55:11
>>welder+C5
Like the VM/Instance/Machine dies with its disk.
replies(1): >>welder+78
4. belthe+b6[view] [source] 2022-09-06 04:56:50
>>jedber+(OP)
This is still a pretty common misconception (and one that I held until recently when looking at Redis configuration options again), but Redis does have durable operating modes to reload the datastore after going offline (which do significantly trade off speed for durability). If your task queue requires a high enough volume of workload that running Redis in a durable fashion is insufficient, then it's probably time to think about a different queuing system than WakaQ.
replies(1): >>jedber+s6
◧◩
5. jedber+s6[view] [source] [discussion] 2022-09-06 04:59:52
>>belthe+b6
I'm talking about if the VM/Instance/Disk goes away. It looks like Redis is a single point of failure unless you run Redis in a redundant way, but skimming the code it looks like that is at best left as an exercise to the reader. ie. Redundancy in the data store doesn't appear to be handled by this code.
replies(1): >>welder+Fg
◧◩◪
6. welder+78[view] [source] [discussion] 2022-09-06 05:14:31
>>jedber+U5
Yes, I don't know any distributed task queue that stays online or backs up tasks when the broker dies. Maybe one could be built on CockroachDB? My implementation is designed to process tasks fast and if the Redis server dies (hasn't happened in 10 years) you just boot up a new one. I try to design my tasks to expect some failures and be able to re-run safely.
replies(1): >>jedber+q8
◧◩◪◨
7. jedber+q8[view] [source] [discussion] 2022-09-06 05:18:56
>>welder+78
RabbitMQ and SQS both have redundancy built in.

There's nothing wrong with the trade off you made, but when I read distributed I assumed that you meant the task broker was distributed and therefore redundant, not the consumers.

replies(1): >>welder+tg
◧◩◪◨⬒
8. welder+tg[view] [source] [discussion] 2022-09-06 06:32:05
>>jedber+q8
Sorry about the confusion. Btw, we've met in person once in Mountain View some years back. Nice to see you here :)
◧◩◪
9. welder+Fg[view] [source] [discussion] 2022-09-06 06:33:24
>>jedber+s6
Correct, this setup wouldn't survive chaos monkey.
replies(1): >>BiteCo+HF
◧◩◪◨
10. BiteCo+HF[view] [source] [discussion] 2022-09-06 10:34:49
>>welder+Fg
Then I wouldn't call it distributed, since it's centralized on redis.

Otherwised good product.

[go to top]