zlacker

[parent] [thread] 3 comments
1. ramesh+(OP)[view] [source] 2023-09-12 19:55:38
Offloading 40% of layers to CPU, about 50t/s with 16 threads.
replies(2): >>pocket+H8 >>jpdus+521
◧
2. pocket+H8[view] [source] 2023-09-12 20:24:37
>>ramesh+(OP)
That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?
replies(1): >>ramesh+MD
◧◩
3. ramesh+MD[view] [source] [discussion] 2023-09-12 22:42:09
>>pocket+H8
>Is there a secret to what you're doing?

Core speed and memory bandwidth matter a lot. This is on a Ryzen 7950 with DDR5.

◧
4. jpdus+521[view] [source] 2023-09-13 01:36:16
>>ramesh+(OP)
Care to share your detailed stack and command to reach 50t/s? I also have a 7950 with DDR 5 and I don't even get 50 t/s on my two RTX 4090s....
[go to top]