I love the convergence with philosophy here.
This is the second reply that's naively just asserted a tautology. You can't define "knowledge" in terms of "knowing" in the sense of the English words; they're the same word! (You can, I guess, if you're willing to write a thesis introducing the specifics of your jargon.)
In point of fact LLMs "know" that they're right, because if they didn't "know" that they wouldn't have told you what they know. Which, we all agree, they do know, right? They give answers that are correct. Usually.
Except when they're wrong. But that's the thing: define "when they're wrong" in a way rigorous enough to permit an engineering solution. But you really can't, for the same reason that you can't prevent us yahoos on the internet from being wrong all the time too.