So you're not free to grow your own vegetables either; just like fishing, farming is regulated to manage limited resources. Things get ugly fast when you start raising pigs in your city apartment, or start polluting with pesticide runoff, or start diverting your neighbour's water supply...
Gardens are a thing, and you do not need your house to be on agricultural land to grow a garden, at least in my state.