Whenever I do, someone will point out that houses today are, on average, bigger.

Houses are bigger because lots are bigger, so developers have to build bigger houses on them in order for the improvement value to be high enough to turn a profit.

Lots are bigger because the zoning code was designed to make them too expensive for minorities to afford, once the Fair Housing Act came through and de-jure segregation and restrictive deed restrictions were outlawed.

In other words, not only is "the houses today are bigger" not really the rebuttal people saying it think it is (because it's not driven by genuine market forces), they're also defending institutional racism.