zlacker

But this doesn’t apply to self-hosted, o?

>>Comput+(OP)
It does. LLMs are most efficient when running large batches, so the gpu cost is super high if you’re underutilizing it. It will cost more than a cloud provider like open ai who has the volume to keep their GPUs saturated

>>lyjack+Le
Yup. It’s also important to mention that OpenAI enjoys the luxury of having large clusters of H100s (the last time I checked).