We shouldn't tolerate ignorance, just because it's the lowest common denominator, exhibited by some people from the lower-middle class, as you put it. It's the world views behind ignorance that are most damaging, and I think the wealthier classes are probably just as guilty as the lower classes there, even if they can hide it behind gentility.
I think it's a lot different to ask people to adapt to a culture of mutual respect than to ask a woman to adapt to a locker room bro culture.