zlacker

[parent] [thread] 2 comments
1. dig1+(OP)[view] [source] 2024-10-14 12:36:36
There is a library called tesser [1] (by Jepsen/Riemann author) that behaves like parallel transducers with more Clojure "native" syntax. With transducers, you have to use "comp," and with tesser, you use "->>" as you'd use with lazy functions.

Sadly, tesser is not advertised as it should; I find it much more flexible than transducers. E.g. you could parallelize tesser code over Spark/Hadoop cluster.

[1] https://github.com/aphyr/tesser

replies(2): >>aeonik+H2 >>thom+Qb
2. aeonik+H2[view] [source] 2024-10-14 13:01:04
>>dig1+(OP)
See the injest library as well, I've had pretty good experience with ita performance.

https://github.com/johnmn3/injest

I can squeeze more performance out of tesser, but injest gives me a surprising boost with very little ceremony most of the time.

3. thom+Qb[view] [source] 2024-10-14 14:07:43
>>dig1+(OP)
As I understand it Tesser doesn't support ordering which makes it tricky for many types of logic. I'd like to be able to control exactly which parts of a pipeline are parallelisable without too much ceremony, but I can't say I've used Rama in anger or that it makes this ergonomic.
[go to top]