He also relates these issues to Christian philosophy, which I find very interesting. In particular, he claims that the a priori belief in the objects defined by Peano Arithmetic, is equivalent to worshipping numbers, as the Pythagoreans did.
I think this is the best starting point if you're interested in reading about his ideas: https://web.math.princeton.edu/~nelson/papers/warn.pdf
http://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_the...
(large pdf) https://web.math.princeton.edu/~nelson/books/pa.pdf