At least in the US, far more people should be removed from society, yes. Even a brief search will turn up large numbers of rapists and murderers (for example) that are sentenced and released over and over again. This is indeed arbitrarily cruel to law-abiding citizens.
As far as it goes, I think conditions in prisons should be far nicer than they are now. In these days of videoconferencing, it needn't be very different than lockdown. I don't believe in punishment; I believe in permanent removal from society.