>>EricMa+gk
Maybe. Goliath 120B took two different llama variants and interwove the layers. Surprisingly Goliath 120B quantized to 2bit is outperforming llama 70B 4bit in many benchmarks.

https://www.reddit.com/r/LocalLLaMA/comments/17vcr9d/llm_com...

>>nextwo+Ho
This is called (Algorithmic) Convergence; does the model stably converge upon one answer which it believes is most correct? After how much resources and time?

Convergence (evolutionary computing) https://en.wikipedia.org/wiki/Convergence_(evolutionary_comp...

Convergence (disambiguation) > Science, technology, and mathematics https://en.wikipedia.org/wiki/Convergence#Science,_technolog...

>>ghotli+oV
The model page is the only info I’ve found on it. As far as I can tell there’s no paper published on the technique.

In the “Merge Process” section they at least give the layer ranges.

https://huggingface.co/alpindale/goliath-120b

>>valine+Lb1
Ah, actually reviewing that more closely I found a link to it in the acknowledgements.

https://github.com/cg123/mergekit

>>koie+(OP)
It can "correct" because it just goes out and finds and produces a pattern template that matches the problem better/different (often just different, and it fails in new ways, in my experience). It never used reasoning to find the answer in the first place, and doesn't use reason to find the corrected answer.

The papers referenced here get into this: https://cacm.acm.org/blogs/blog-cacm/276268-can-llms-really-...

>>bongod+A41
A recent paper along these lines you might be interested in was Large Language Models Understand and Can be Enhanced by Emotional Stimuli: https://arxiv.org/abs/2307.11760

It makes complete sense and has been a part of my own usage for well over a year now, but it's been cool seeing it demonstrated in research across multiple models.

>>galaxy+GN1
As I tried to explain, it's not the code that people don't understand. People understand the code they wrote.

It's why the bunch of linear algebra on the weights works to do this particular task, and how it will respond to any particular task that is a bit mysterious.

Like imagine someone gave you the Taylor series expansion of the inverse Kepler equation[1]. So you just have a bunch of crazy fractions of powers of x that you add up. And then they say ok now the this function will very accurately explain the orbit of the planets.

You'd be able to do the steps - you're just adding up fractions. You'd be able to verify the answer you got corresponded to the orbit of a given celestial body.

But if you didn't have all the pieces in the middle (calculus mainly) there's no way you'd be able to explain why this particular set of fractions corresponds to the movement of the planets and some other set doesn't.

[1] https://en.wikipedia.org/wiki/Kepler%27s_equation scroll down a bit

>>Roark6+0P1
https://www.inf.ufpr.br/renato/profession.html

>>jibal+wR1
P.S.

https://www.google.com/search?q=%22python+app+calculate+road...

If you leave off the quotes (which were present in the comment I responded to) then of course you will get millions of irrelevant hits. Somewhere in that chaff there is some Python code that alleges to have something to with road trips, though it's not always clear what. If I give the same prompt to ChatGPT I get a nicely formatted box with a program that uses the Google Maps Distance Matrix API to calculate distance and duration, without a bunch of junk to wade through. (I haven't tried it so it could be a complete hallucination.)

>>mkl+FQ1
> distinguishing fact from fiction

Actually this part does seem in recent research to be encoded in LLMs at an abstract level in a linear representation...

https://arxiv.org/abs/2310.06824