I'm going to risk a political statement and say that this is why I'm mostly hopeful about DOGE, even if parts of it are a shit show.
Building civilization comes with a hefty dose of institutional entropy, which keeps accumulating, despite (or often because) good intentions and competence. Everybody is improving their piece of the map, but this means you get stuck in a lot of spots of local maxima. Some can be fixed from a level above, but some need a round of creative destruction every 10 years or so.
I've read this yesterday: https://unchartedterritories.tomaspueyo.com/p/why-japan-succ...
It's a good read and a good blog for many reasons, but the relevant part to this conversation: Japan managed to keep a very high level of living even through decades of economic stagnation and aging population in large part by having a sane zoning system. Yes, that simple. They have 12, nation-wide, mostly inclusive zoning types. This means the permitted building types carry over as you move up the categories, allowing mixed-use development by default.
And indeed, you can actually go to Japan and buy a house for about the price of a decent car - which coincidently used to be the case in most of the world, before the double pressure of zoning/coding on one hand, and migration towards urban centers on the other squeezed the housing pricing way above what actual costs would have it be.
Musk seems to want things to scale; fewer people to achieve more productivity. People that already fall through the cracks aren't going to suddenly find themselves better off via a system that scales better, because better scaling actually creates wider cracks.
The median flows better, at the cost of the fringes.
Your comments regarding Japan are interesting. Japan's definitely an interesting example to use due to the odd, unenviable economic situation, but that makes your point stand out more rather than less, I think.