Then I realized that I was writing about compiling for ARM and this post is about x86. Which is extra weird! Why is the compiler better tuned for ARM than x86 in this case?
Never did figure out what gcc's problem was.
I would try a more specific flag like -ffinite-math-only.
So, yes when targeting VFP math. NEON already always works in this mode though.