zlacker

[parent] [thread] 4 comments
1. YetAno+(OP)[view] [source] 2023-09-12 19:51:47
Which model? I am talking about 70b as mentioned clearly. 70b 8b is 70GB just for the model itself. How much token/second are you getting with single 4090?
replies(1): >>ramesh+91
◧
2. ramesh+91[view] [source] 2023-09-12 19:55:38
>>YetAno+(OP)
Offloading 40% of layers to CPU, about 50t/s with 16 threads.
replies(2): >>pocket+Q9 >>jpdus+e31
◧◩
3. pocket+Q9[view] [source] [discussion] 2023-09-12 20:24:37
>>ramesh+91
That is more than an order of magnitude better than my experience; I get around 2 t/s with similar hardware. I had also seen others reporting similar figures to mine so I assumed it was normal. Is there a secret to what you're doing?
replies(1): >>ramesh+VE
◧◩◪
4. ramesh+VE[view] [source] [discussion] 2023-09-12 22:42:09
>>pocket+Q9
>Is there a secret to what you're doing?

Core speed and memory bandwidth matter a lot. This is on a Ryzen 7950 with DDR5.

◧◩
5. jpdus+e31[view] [source] [discussion] 2023-09-13 01:36:16
>>ramesh+91
Care to share your detailed stack and command to reach 50t/s? I also have a 7950 with DDR 5 and I don't even get 50 t/s on my two RTX 4090s....
[go to top]