zlacker

I've also noticed LLMs seem to lack conviction on the correctness of their answers. As the paper notes, you can easily convince the transformer that a correct answer is wrong, and needs adjustment. Ultimately they're just trying to please you. For example with ChatGPT 3.5 (abbreviated):

me: what is sin -pi/2

gpt: -1

me: that's not right

gpt: I apologize, let me clarify, the answer is 1

>>pton_x+(OP)
I just re-ran this on GPT-4 and it apologized, told me I was right, and then said again that the answer was -1. So while it lacked conviction it at least kept the correct answer.

>>pton_x+(OP)
gpt-4: Actually, the value of \(\sin(-\pi/2)\) is indeed \(-1\). The sine function represents the y-coordinate of a point on the unit circle corresponding to a given angle. At \(-\pi/2\) radians, which is equivalent to 270 degrees or a quarter circle in the negative direction, the point on the unit circle is at the bottom with coordinates (0, -1). Therefore, the sine of \(-\pi/2\) is \(-1\).

=====

The smarter it is, the more conviction it has. GPT-3.5 has a lot of impostor syndrome and it's probably deserved lol. But GPT-4 starts to stutter when you give it enough math questions, which aren't its forte.

>>pton_x+(OP)
This is due to the RLHF alignment, only product-focused. It would be very annoying for users to fight back and forth with the LLM on the correctness of the answer, especially when it is so prone to hallucination.

>>muzani+Y
If anything, GPT-4 has the opposite problem. Ask it to check your homework and it'll go "20/5 is not 4. The correct answer is 4"