He also relates these issues to Christian philosophy, which I find very interesting. In particular, he claims that the a priori belief in the objects defined by Peano Arithmetic, is equivalent to worshipping numbers, as the Pythagoreans did.
I think this is the best starting point if you're interested in reading about his ideas: https://web.math.princeton.edu/~nelson/papers/warn.pdf
http://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_the...
(large pdf) https://web.math.princeton.edu/~nelson/books/pa.pdf
http://m-phi.blogspot.com/2011/10/nelson-withdraws-his-claim...
The whole episode: - Reputed Princeton mathematician publishes paper questioning the foundations of math - Terry Tao 'disproves' him in a series of blog comments is one of the more interesting recent happenings in the world of math
I was referring to earlier work by Nelson in which he explains why he thinks Peano Arithmetic may be inconsistent.
Later, in the incident you referred to, he claimed to have an actual proof. Terry Tao showed the proof was wrong, and Nelson accepted this.
Usually this is left out, because Peano Arithmetic is treated as a MVP for mathematics. But Nelson claims Peano Arithmetic may be inconsistent, and proposes a weaker system.
1) Not unique to exponentiation, it's also true of 281474976710656 + 79766443076872509863361.
2) This is a vague notion and you're going to have to deal with the problem of the smallest number that isn't expressible (I believe it might be the same as "feasible numbers", and there's a whole theory here)
3) the whole tenor of his critique suggests something a bit more foundational (predicativity, I guess...?)
As you say, the notion of a "real number" is vague without the details (which I'm not really able to describe properly, because I'm only summarizing something I vaguely understand. I'm not an expert on mathematical logic). The best source is Nelson's book "Predicative Arithmetic", https://web.math.princeton.edu/~nelson/books/pa.pdf In the book, he defines what he means by a real number, and shows why exponentation doesn't satisfy the property that if a^X is a real number, then a^(X+1) is.
Most of us take some axioms and believe them, ultimately because they correspond to physical experience - if you take a pile of n pebbles and add another pebble to the pile, you get a pile of n+1 pebbles, and n+1 is always a new number that doesn't =0 (that is, our pile looks different from a pile of 0 pebbles). Maybe there's some (very large) special n where this wouldn't happen - where you'd add 1 and get a pile of 0 pebbles, or where you can have n pebbles but you can't have n sheep. But so far we haven't found that. So far the universe remains simple, and the axioms of arithmetic are simpler than conceivable alternatives.