zlacker

There's also no reason to believe that incremental progress in transformer models will eventually lead to "AGI".

>>root_a+(OP)
Yes, but I think everyone would agree that the chance isn't 0%

>>snapca+62
I don't agree, I think many people would argue the chance is 0%.

>>snapca+62
Non-zero chances don't deserve the hype AGI is receiving, is the issue.

And a lot of AI experts outside of the AGI grift have stated that it's zero.

>>root_a+83
Are you one of those people? how can you be so confident? I think everyone should have updated their priors after how surprising the emergent behavior in GPT3+ are

>>snapca+62
It's a static single-pass feed-forward network. How could that possibly result in AGI?! (Queue famous last words ...)

>>snapca+62
Transformer neural networks are not capable of true recursion, which is an excellent reason to think that the chance truly is 0%.

>>snapca+R3
Perhaps you should update your priors about "emergent behavior" in GPT3+: https://arxiv.org/abs/2304.15004

>>root_a+83
I don't think the chance is 0%, but I do think that the chance is very, very close to 0%, at least if we're talking about it happening with current technology within the next hundred years or so.

>>snapca+R3
I don't think GPT3's "emergent behavior" was very surprising, it was a natural progression from GPT2, and the entire purpose of GPT3 was to test the assumptions about how much more performance you could gain by growing the size of the model. That isn't to say GPT3 isn't impressive, but its behavior was within the cone of anticipated possibilities.

Based on a similar understanding, the idea that transformer models will lead to AGI seems obviously incorrect, as impressive as they are, they are just statistical pattern matchers of tokens, not systems that understand the world from first principles. And just in case you're among those that believe "humans are just pattern matchers", that might be true, but humans are modeling the world based on real time integrated sensory input, not on statistical patterns of a selection of text posted online. There's simply no reason to believe that AGI can come out of that.

>>pton_x+L4
IMO it could become an AGI IFF it has an infinitely long context window. Otherwise I see absolutely no chance of it becoming a true agi

>>nickle+P5
That seems easy enough to fix

>>root_a+T6
I agree. I am baffled as to why there isn't more thought on developing AI starting from simple sensory input.

>>nickle+76
This is like saying that nothing special happens to water at 100 degrees because if you look at the total thermal energy, it's a smooth increase.

>>famous+u91
Please read the paper. The authors are using more precise and specific metrics that qualitatively measure the same thing. Instead of having exact string match being 1 if 100% correct, 0 if there is any failure, they use per-token error. The crux of their argument is that per-token error is a better choice of metric anyway, and the fact that "emergent abilities" do not occur when using this metric is a strong argument that those abilities don't really exist.

However thermal energy does not more precisely or specifically measure a phase transition. They are only indirectly linked - nobody would say that thermal energy is a better measure of state-of-matter than solid/liquid/gas. Your argument makes absolutely zero sense. Frankly it seems intentionally ignorant.

>>nickle+Nd1
I read the paper.

Per token error is a fairly useless metric. It's not predictive and it tells you absolutely nothing.

They say it's a superior metric but clearly the wider research community disagrees since no one has cared to adopt per token error as a metric in subsequent papers.

>and the fact that "emergent abilities" do not occur when using this metric is a strong argument that those abilities don't really exist.

If your conclusion is that those abilities don't exist then you clearly didn't read the paper very well.

They never argue those abilities don't exist, they simply argue whether we should call them "emergent" or not.

>However thermal energy does not more precisely or specifically measure a phase transition. They are only indirectly linked - nobody would say that thermal energy is a better measure of state-of-matter than solid/liquid/gas. Your argument makes absolutely zero sense. Frankly it seems intentionally ignorant.

Phase Changes are literally driven by changes in thermal energy.

Water boils when it absorbs enough thermal energy to break intermolecular forces keeping its liquid state together.

solid/liquid/gas is descriptive. It's not a measure of anything.

Anyway, the point is simple. Despite thermal energy driving state change after a certain threshold, that "point" doesn't look like anything special.

Smooth quantitative change sometimes results in sudden qualitative changes.

>>pton_x+L4
Just feed the output through a quality system (with retry if the quality is too bad), scaffold it a bit, then run it back into the LLM. Should work(tm)