You've never heard of the term "brain drain"?
However, you must remember the bit about limiting who is allowed in the country. If you were a German with a hypothetical burning desire to flip burgers at In-N-Out Burger, what are the chances of you getting a work visa? I would say effectively nil. So you're not going to see those people even if they would arrive in a world without borders.
That's a great point. Legal immigration (to somewhere desireable) is not an option for much of the population. What interest do they have in preserving it?
A good question. It is quite possible that they don't – that a different segment of the population has that interest.