zlacker

I think it's the difference between spending 10's of millions and a billion on computing resources.

When he started OpenAI, no one was spending anywhere near that much on compute. He should have listened to gwern.

>>JamesB+(OP)
Exactly. The idea that massive compute would lead to massive improvement was non-obvious.

It was quite reasonable to think that there would be rapidly diminishing returns in model size.

Wrong, in hindsight, but that's how hindsight is.

>>crazyg+vf
Google and all the big players in AI have known they need tons of data and hence compute power for processing it, for a very long time, way before OpenAI even existed. Anyone getting involved in that game would have definitely known.

>>crazyg+vf
>The idea that massive compute would lead to massive improvement was non-obvious.

Honestly no, it was obvious, but only if you listened to those pie in the sky singularity people. It was quite common for them to say, add lots of nodes and transistors and a bunch of layers and stir in some math and intelligence will pop out.

The groups talking about minimal data and processing have not had any breakthroughs in, like forever.