Then I realized that I was writing about compiling for ARM and this post is about x86. Which is extra weird! Why is the compiler better tuned for ARM than x86 in this case?
Never did figure out what gcc's problem was.
I would try a more specific flag like -ffinite-math-only.
I like to think that the flag should be renamed “-Ofuck-my-shit-up”.