zlacker

[parent] [thread] 2 comments
1. kcorbi+(OP)[view] [source] 2023-12-20 21:55:33
Hey, I'm the post author. This is a totally fair point! I do think though that depending on your specific requirements open-source models can be a 10x+ improvement. For example, we serve Mistral 7B for less than 1/10th the cost of GPT-4-Turbo, which is the model most of our users are comparing us to.
replies(2): >>xrd+X8 >>MacsHe+y21
◧
2. xrd+X8[view] [source] 2023-12-20 22:50:45
>>kcorbi+(OP)
This is the 10x I was looking for. Great post by the way!
◧
3. MacsHe+y21[view] [source] 2023-12-21 08:18:29
>>kcorbi+(OP)
I serve ~300tk/s of Mistral 7B for $0.60/hr by renting a cloud 3090. That's a lot cheaper than GPT-4-Turbo, though the quality is closer to GPT-3.5.

Mixtral 8x7b is closer to GPT-4 quality though and only 2x the compute requirement of Mistral 7B.

[go to top]