I'm pleasantly surprised he said that.

From The Daily Mail:

Tory backbencher Julian Brazier, author of a paper on rising population, said: ‘Late motherhood is one more example of how property prices in Britain are making family life increasingly unattainable.’

Mr Brazier, MP for Canterbury, said: ‘Despite the downturn, we have the highest property prices in the world in relation to incomes.
There is also the effect of immigration, which is responsible for two fifths of housing demand. It is important that the Government succeeds in getting immigration down.’

Well done for pointing out that high house prices are bad for 'family life', something which the Tories claim to support (but don't).
-------------------------------------------
He's a bit off piste with the immigration sideswipe though. It's true that recent immigrants have a slight advantage when it comes to social housing - the IPPR published some research which purported to show they don't, but they were too honest for the own good and the figures they included said otherwise.

What he probably means is that the net number of immigrants per year (presumably divided by two to give number of households) is equivalent to two fifths of the number new houses built per year*. So yes, getting immigration down is one possibility, the other is just allowing more houses to be built.

* Net immigration from 'New Commonwealth and EU8' was about 160,000 a year, divide by 2 = 80,000, for a long time, new construction every year was about 200,000, 80,000/200,000 is two-fifths. Of course, now that the Tories have got in, they have restricted new construction to 100,000, so next year they'll be able to say that immigrants take up four-fifths of housing.