So you're not free to grow your own vegetables either; just like fishing, farming is regulated to manage limited resources. Things get ugly fast when you start raising pigs in your city apartment, or start polluting with pesticide runoff, or start diverting your neighbour's water supply...
Gardens are a thing, and you do not need your house to be on agricultural land to grow a garden, at least in my state.
Most people don't have that, and can't afford that, hence why they take the route of earning money some other way, and using the money to buy food made by others, from supermarkets. They can supplement their diet with home-grown fruit and veg, but few can sustain their family on home-grown produce.