zlacker

[parent] [thread] 4 comments
1. thewat+(OP)[view] [source] 2023-09-12 20:16:10
What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?
replies(3): >>ramesh+12 >>mjirv+tn >>pdntsp+gL
◧
2. ramesh+12[view] [source] 2023-09-12 20:23:08
>>thewat+(OP)
>What's the best way to use LLama2-70b without existing infrastructure for orchestrating it?

That's an exercise left to the reader for now, and is where your value/moat lies.

replies(1): >>thewat+q5
◧◩
3. thewat+q5[view] [source] [discussion] 2023-09-12 20:37:21
>>ramesh+12
> That's an exercise left to the reader for now, and is where your value/moat lies.

Hopefully more on-demand services enter the space. Currently where I am we don't have the resources for any type of self orchestration and our use case is so low/sporadic that we can't simply have a dedicated instance.

Last I saw the current services were rather expensive but I should recheck.

◧
4. mjirv+tn[view] [source] 2023-09-12 21:50:45
>>thewat+(OP)
I stumbled upon OpenRouter[0] a few days ago. Easiest I’ve seen by far (if you want SaaS, not hosting it yourself).

[0] https://openrouter.ai

◧
5. pdntsp+gL[view] [source] 2023-09-13 00:13:32
>>thewat+(OP)
I bought an old server off ServerMonkey for like $700 with a stupid amount of RAM and CPUs and it runs Llama2-70b fine, if a little slowly. Good for experimenting
[go to top]