You'd be really surprised. Your point is a valid one, but one I seldom see brought up - most people are just worried that for every 100 immigrants there's a couple of actual terrorists in the lot... Mostly because nobody's told them how stupid that notion is.
Oh some people try. Hell, I told that to many people myself. People don't care about things like actually being reasonable. Instead, they're afraid and are looking to rationalize it.