zlacker

[parent] [thread] 3 comments
1. alecr9+(OP)[view] [source] 2022-09-06 12:22:53
Nice project!

I'm one of TaskTiger's current maintainers. Just curious if any specific features are missing (or how it compares in benchmarks)?

replies(1): >>welder+s4
2. welder+s4[view] [source] 2022-09-06 12:52:51
>>alecr9+(OP)
When workers claim tasks, TaskTiger moves tasks to a new queue instead of popping them like WakaQ. That means if your task fails with WakaQ then it's just gone.

> how it compares in benchmarks

WakaQ processes many tasks per fork, while I think TaskTiger only processes one? That should make WakaQ slightly faster, but it still depends on too many factors to know without benchmarks.

I actually benchmarked TaskTiger in production a long time ago against Celery. From memory, I think it was slower than Celery. I can't find those benchmarks and even if I could, WakaTime's scale is bigger now so not sure how useful they would be. If you create some benchmarks would love to see them.

replies(1): >>alecr9+Js5
◧◩
3. alecr9+Js5[view] [source] [discussion] 2022-09-07 21:24:30
>>welder+s4
> TaskTiger only processes one

Generally yes, though it also supports "batching" tasks where tasks are `.delay`'d normally but the task is given batches of N sets of args at a time and the task's code has to understand batching logic.

:+1: I'd definitely believe the task handling code is slower than celery. Honestly I haven't personally done too much optimization work on it (though perhaps previous maintainers have, I'm not sure). Performance of TaskTiger itself hasn't really been problematic at all for us in production with hundreds of workers. Usually it's been external things (like databases, third party API's, etc) that impact our task throughput the most. Or wanting to shift the architecture entirely to avoid using memory-bound Redis as a queue with an overflow risk.

replies(1): >>welder+Yv5
◧◩◪
4. welder+Yv5[view] [source] [discussion] 2022-09-07 21:42:32
>>alecr9+Js5
> Or wanting to shift the architecture entirely to avoid using memory-bound Redis as a queue with an overflow risk.

I wanted to use SSDB[1] instead of Redis for that reason, but it doesn't support the necessary data structures.

[1] https://github.com/ideawu/ssdb

[go to top]