There's a combination of factors, of which some are:
- Holding cost of land is near 0 in most cities, so you can use it extremely unproductively without any issues. There's a Christmas tree farm in a central suburb in my city; it could easily hold a 5-story apartment block or 8 townhouses housing tens of people instead. [Solution: land value tax]
- In most Western democracies, there are significant regulatory requirements that make it more expensive to build housing. This isn't necessarily entirely bad, but it does increase the cost to build more housing, thus reducing its supply. [Solution: there's no real easy solution for this one. Some regulations are necessary, it's hard work to make the regulations really good]
- In most cities, there are restrictions on what you can build and where. Again, not always entirely bad per se, but the settings are usually dialed up to the point of absurdity. This means even if you wanted to build more housing within some suburb, you might be unable to build it densely enough to increase how many people are housed. [Solution: more flexible zoning laws to enable more freedom in what people can build on their land; if you want a standalone house, go for it, if you want a 28 story apartment block, go for it, if you want a 3-story townhouse row, go for it, if you want a work-and-live store with an apartment on top, go for it]
- Even in areas where there are fewer restrictions, e.g. new land areas opened up for development, developers are incentivised to keep the available supply of housing low to keep the price high. This means that a large development of hundreds of houses will often only release the houses 5-10 at a time, artificially reducing supply. [Solution: increase the holding price of land, aka land value tax]
Zoning, permitting, and property taxes (especially in California) encourage people to just sit on old properties instead of using them more efficiently.
Agree .. but why is something I'd love to learn