I just asked "what does the JS ** operator do" and it made up an answer about it being a bitwise XOR. 1 ** 2 === 3. The fact that all these LLMs will confidently suggest wrong information makes me feel like LLM is going to be a difficult path to AGI. It will be a big problem if an AI receptionist just confidently spews misinformation and is unable to tell customers they are wrong.
So do many humans. The expression of ignorance and self-doubt must certainly be woefully underrepresented in training data.