You're not fighting the upper class. It's the blue collar workers and the people who hire them who support ICE and strict immigration.
You are fighting the upper-class, while some of the working-class people are mislead to fight on the other side. Slowly but surely they'll realize where to go, but often the promises of wealth and what not gets to strong for the individuals to at least try to move up.