Then, if fruit pickers made $20 an hour, you would be getting all of your fruit from Mexico/Central America, and fruit and all products made from it would be more expensive.
Other jobs like gardening and landscaping would simply either not be done, or get done by property owners themselves. Now they will be tired, be less productive at work and have less time for their kids, or, see their property value drop while their local crime rate goes up.
It would be very bad to kill it without replacement.