zlacker

[parent] [thread] 31 comments
1. gjsman+(OP)[view] [source] 2023-11-17 21:14:32
I wonder if the cost of running GPT-3.5 and GPT-4 models at scale turned out to have been astoundingly more expensive than anticipated.

Imagine if you were the CTO of a company, massively underestimated your AWS bill, and presented your board with something enormous. Maybe something like that happened?

Or, if I wanted to speculate to the extremely negative; what if the training and operating costs ballooned to such a degree, that the deal with Microsoft was an attempt to plug the cash hole without having to go to the board requesting an enormous loan? Because the fact that Copilot (edit: previously known as Bing Chat and Bing Image Creator) is free and ChatGPT (edit: and DALL-E 3) are not should be a red flag...

replies(9): >>swalsh+03 >>easton+e3 >>davegu+94 >>sparkl+u4 >>aenis+Z4 >>dragon+p5 >>HankB9+E6 >>DSMan1+fc >>JohnFe+CT
2. swalsh+03[view] [source] 2023-11-17 21:27:35
>>gjsman+(OP)
This is plausible to me, there's no way anyone is making money from my $20 subscription I use ALL DAY LONG.
replies(2): >>coffee+M4 >>jstumm+L6
3. easton+e3[view] [source] 2023-11-17 21:29:07
>>gjsman+(OP)
I have to guess the bulk of the cost is being eaten by MS in exchange for the exclusive ability to resell the model.
4. davegu+94[view] [source] 2023-11-17 21:32:32
>>gjsman+(OP)
"not significantly candid"

and

"no longer has confidence"

points to something more serious than underestimating costs.

replies(4): >>gjsman+F4 >>DebtDe+j5 >>paulpa+E7 >>ARandu+R9
5. sparkl+u4[view] [source] 2023-11-17 21:33:37
>>gjsman+(OP)
> Because the fact that Copilot is free and ChatGPT is not should be a red flag...

I'd assume that running a model that only needs to deal with a single programming language (the Copilot plugin knows what kind of code base it is working on) is _a lot_ cheaper than running the "full" ChatGPT 4.

replies(1): >>gjsman+65
◧◩
6. gjsman+F4[view] [source] [discussion] 2023-11-17 21:34:15
>>davegu+94
Other than costs or the allegations by the sister, "not significantly candid" could easily be short for, in my mind:

"not significantly candid in projections for profitability"

"not significantly candid in calculating operation cost increases"

"not significantly candid in how much subscribers are actually using ChatGPT"

etc.

◧◩
7. coffee+M4[view] [source] [discussion] 2023-11-17 21:34:43
>>swalsh+03
Is inference really that expensive? Anyway if the price is too low they could easily charge by query
replies(1): >>knicho+2c
8. aenis+Z4[view] [source] 2023-11-17 21:35:35
>>gjsman+(OP)
The expected value of a leading AI company is probably in hundreds of billions, if not trillions in the foreseeable future. He could be burning billions per month and he'd still be doing great.
replies(1): >>axiak+Z5
◧◩
9. gjsman+65[view] [source] [discussion] 2023-11-17 21:36:08
>>sparkl+u4
Sorry for being so precise, but Microsoft renamed Bing Chat to Copilot yesterday, has already rolled it out to all users of Microsoft Edge, and is rolling out a permanent button on the Windows 11 taskbar to access it.

This is what shouldn't add up: Microsoft is literally adding GPT-4, for free, to the Windows 11 taskbar. Can you imagine how much that costs when you look at the GPT-4 API, or ChatGPT's subscription price? Either Microsoft is burning money, or OpenAI agreed to burn money with them. But why would they do that, when that would compromise $20/mo. subscription sales?

Something doesn't financially add up there.

replies(4): >>sparkl+x5 >>ctc24+s7 >>rivers+V7 >>crucia+Ue
◧◩
10. DebtDe+j5[view] [source] [discussion] 2023-11-17 21:36:44
>>davegu+94
Not if the underestimation was to such a degree that it showed they could never even plausibly reach a break even point.
11. dragon+p5[view] [source] 2023-11-17 21:37:20
>>gjsman+(OP)
> Imagine if you were the CTO of a company, massively underestimated your AWS bill, and presented your board with something enormous.

Unless there was evidence you had not underestimated but were, e.g., getting a kickback on the cloud costs that you deliverately lowballed in your estimates, they might fire you, but they almost certainly wouldn't put out a press release about it being for your failure to be candid.

That language indicates that the board has a strong belief that there was a major lie to the board or an ongoing pattern of systematic misrepresentation, or a combination.

replies(1): >>synaes+ug
◧◩◪
12. sparkl+x5[view] [source] [discussion] 2023-11-17 21:37:53
>>gjsman+65
Sorry i assumed you were talking about Github CoPilot (also owned by MS via Github)
◧◩
13. axiak+Z5[view] [source] [discussion] 2023-11-17 21:39:31
>>aenis+Z4
based on what math? I can see how there can potentially be differentiators here and there to raise value, but I can't see how this statement can be taken prima facie
replies(1): >>sumedh+SF
14. HankB9+E6[view] [source] 2023-11-17 21:42:13
>>gjsman+(OP)
> Because the fact that Copilot is free ...

I found a tree trial and $10/month $100/year after that. I've asked them to consider a free tier for hobbyists that cannot justify the expense but I'm not holding my breath.

If there is a free tier I did not find, please point me to it!

◧◩
15. jstumm+L6[view] [source] [discussion] 2023-11-17 21:42:34
>>swalsh+03
Given the arbitrary rate limiting they take the liberty of doing, it's a very deliberate decision and entirely within their control to change at any point.
◧◩◪
16. ctc24+s7[view] [source] [discussion] 2023-11-17 21:45:45
>>gjsman+65
I don't thing there's necessarily anything there. Microsoft might be burning money because they've decided that browser adoption and usage is worth it to them. It doesn't have to involve OpenAI in any way.
◧◩
17. paulpa+E7[view] [source] [discussion] 2023-11-17 21:46:36
>>davegu+94
100% this. Firing your well-recognized and seemingly capable CEO means there's a fatal problem, or that he committed something so reprehensible that there was no option but to oust him immediately.

Maybe Sam had been trying to broker a sale of the company without consulting the board first? All speculation until more details are revealed but he must've done something of similar magnitude.

◧◩◪
18. rivers+V7[view] [source] [discussion] 2023-11-17 21:48:09
>>gjsman+65
You got me excited that Github Copilot was free. Was going to post to tell you it is, in fact, not free. I've been using Bing on Edge browser for a while now, it's super useful! Sad that they rebranded it to Copilot though, "I have been a good Bing :)" will be forever in my memory. [1] RIP Bing, you were a good chat mode.

[1] https://simonwillison.net/2023/Feb/15/bing/

◧◩
19. ARandu+R9[view] [source] [discussion] 2023-11-17 21:58:26
>>davegu+94
Underestimating costs could be the reasoning if Altman knew the costs would be higher then estimated, and didn't tell the board for an unreasonable amount of time. Burning through a ton of cash for months and not telling the board about it could be enough grounds for this sudden firing.

Of course we have no clue if that's what actually happened. Any conclusions made at this point are complete speculation, and we can't make any conclusions more specific then "this is probably bad news."

replies(1): >>jliptz+tb
◧◩◪
20. jliptz+tb[view] [source] [discussion] 2023-11-17 22:08:29
>>ARandu+R9
That only makes sense if Altman is the only one with access to the company's financials which obviously can't be the case. No one else noticed truckloads of cash getting flushed down the toilet?
replies(2): >>ARandu+4e >>maximu+mc1
◧◩◪
21. knicho+2c[view] [source] [discussion] 2023-11-17 22:13:03
>>coffee+M4
When I was mining with a bunch of RTX 3080s and RTX 3090s, the electricity cost (admittedly) was about $20/month per card. Running a 70B model takes 3-4 cards. Assuming you're pushing these cards to their extreme max, it's going to be $80/mo. Then again, ChatGPT is pretty awesome, and is likely running more than a 70B model (or I think I heard it was running an ensemble of models), so there's at least a ballpark.
replies(3): >>sodali+3f >>Sebb76+al >>698969+Qd1
22. DSMan1+fc[view] [source] 2023-11-17 22:14:15
>>gjsman+(OP)
> Imagine if you were the CTO of a company, massively underestimated your AWS bill, and presented your board with something enormous. Maybe something like that happened?

I think the problem there is that the original CTO is now the interim CEO and they are on the board. So while that kind of scenario could make sense, it's a little hard to picture how the CTO would not know about something like that, and if they did you'd presumably not make them CEO afterward.

◧◩◪◨
23. ARandu+4e[view] [source] [discussion] 2023-11-17 22:23:08
>>jliptz+tb
It's certainly possible. Company financials can get very complicated very quickly, and it's possible that Altman was the only person (or one of a small number of people) who had the complete picture.

To be clear, this is only one possible explanation for Altman's firing. And for my money, I don't even think it's the most likely explanation. But right now, those who rely on OpenAI products should prepare for the worst, and this is one of the most existentially threatening possibilities.

◧◩◪
24. crucia+Ue[view] [source] [discussion] 2023-11-17 22:27:25
>>gjsman+65
Microsoft is pulling browser, search and AI hype mindshare away from Google. That's worth burning money for.
◧◩◪◨
25. sodali+3f[view] [source] [discussion] 2023-11-17 22:27:59
>>knicho+2c
Batched inference makes these calculations hard - roughly takes the same amount of power and time for one inference vs 30 (as i understand it)
◧◩
26. synaes+ug[view] [source] [discussion] 2023-11-17 22:33:36
>>dragon+p5
I don’t think this is necessarily what happened (the CFO would certainly be implicated and it appears they were spared).
◧◩◪◨
27. Sebb76+al[view] [source] [discussion] 2023-11-17 22:54:13
>>knicho+2c
Datacenters probably do not pay retail rates on electricity, so they might actually run quite a bit cheaper (or more expensive if they use highly available power, but this seems like overkill for pure compute power).
replies(1): >>015a+HK
◧◩◪
28. sumedh+SF[view] [source] [discussion] 2023-11-18 00:33:02
>>axiak+Z5
> based on what math?

Based on future potential. Investors dont know how high will OpenAI go but they know that is going to go high.

◧◩◪◨⬒
29. 015a+HK[view] [source] [discussion] 2023-11-18 00:57:52
>>Sebb76+al
Sure, but everything else about a data center is more expensive (real estate, operations people, networking, equipment). There's a reason AWS is so expensive.
30. JohnFe+CT[view] [source] 2023-11-18 01:50:21
>>gjsman+(OP)
> Imagine if you were the CTO of a company, massively underestimated your AWS bill,

Microsoft is investing billions into OpenAI, and much of it is in the form of cloud services. I doubt there was a surprise bill for that sort of thing. But if there was, and Altman is the one who ordered it, I could see the board reacting in a similar way.

◧◩◪◨
31. maximu+mc1[view] [source] [discussion] 2023-11-18 04:12:55
>>jliptz+tb
I mean... he got fired so if that's what happened, they did notice.
◧◩◪◨
32. 698969+Qd1[view] [source] [discussion] 2023-11-18 04:24:22
>>knicho+2c
Presumably your miner is running 24/7 throughout the month. Not the same for ChatGPT which would answer maybe 10 sessions (with multiple pauses between queries) tops from a single person in a day.
[go to top]