I think the real issue is that the US has some socio-economic issues that we've as yet let unresolved: drugs (as mentioned), policing strategy, judicial/legislative strategy, welfare, education, etc. I don't think institutionalization is a major contributor.
A quick google search finds a source claiming that about 10-20% of the prison population (a few hundred thousand) are seriously mentally ill:
http://www.psychiatrictimes.com/forensic-psych/content/artic...