That was a temporary blip because the underclass had been temporarily exported to a bombed-out Europe that was unable to meet its production needs, giving the US a huge market and very little manufacturing competition. Then came the Nixon shock, stagflation, and Reagan.
https://fred.stlouisfed.org/graph/?g=cWvT
Also, 25% of homes in 1950 didn't have indoor toilets. It was very much not modern quality housing.
Note, more households in the US own their home now than they did in 1984. (I don't have numbers on this one back to the 50s.)
https://fred.stlouisfed.org/series/USHOWN
Fear about this is possibly suppressed memories of the 2008 recession, which was really bad!