Were we all racist before 1997?

I personally don't have any particularly strong opinions about immigration or racism one way or another, but I have noticed that the minute anybody, such as UKIP, of whom I am a member, mentions 'immigration' they are immediately branded as 'racist', which is a bad thing to be of course (my view being, there is very little that a government to 'combat racism' directly, and all this political correctness just makes it worse). While it is true that racists oppose immigration, I refuse to accept that the reverse automatically holds.

Anyway, if a political party said that they were going to reduce immigration by two-thirds, would they be branded racist? Almost certainly. If a political party said that they were going to reduce immigration to pre-1997 levels, would they be branded racist? Probably. But I simply do not remember people complaining bitterly before 1997 how restrictive the UK's immigration controls were and that we should let more people in; or indeed that Labour's 1997 election campaign promised that net immigration would treble. Which begs the question ... were we all racist before 1997?

Let's look at the official statistics* to put all this in perspective. It's nice to start with emigration of British Citizens (because that's pretty uncontroversial). Gross emigration is the red series and net emigration is the green one (click to enlarge):
Then let's look at immigration from Old Commonwealth (that's the term used in the official statistics, which I assume means Canada, Australia and New Zealand) and the 'EU 15', i.e. Western Europe excl. the UK. Both gross and net immigration appear to have drifted upwards since 1991, but the figures are far lower than the number of British Citizens emigrating, the bulk of whom probably go to these countries, so that's nothing to get excited about either (click to enlarge).
Then there's gross and net immigration from the New Commonwealth (Pakistan, Bangladesh, India plus African countries); the newest ten EU countries (Eastern Europe plus Cyprus and Malta) and 'rest of the world' (i.e. non-Commonwealth Africa and the USA). The figures were low and stable until 1997, and gross immigration has more than doubled and net immigration more than trebled since then (click to enlarge).
You could argue that net immigration from these countries is 'only' twice as much as the net emigration of British Citizens and nothing to worry about in a country of sixty million people.

You could argue that forty per cent of new arrivals from these countries end up in social housing** (the ones who get social housing are less likely to leave again), assuming two immigrants per household (they start having babies once they're here) that makes 50,000 cases per year; with four million units of social housing and a turnover of one-in-twenty units per year only 200,000 households make it to the top of the list every year and thus that a quarter of available social housing goes to immigrants.

You can argue anything you like, to be honest, I'm just trying to put it all in perspective.

* All these charts are based on Table 2.01a taken from 2 Series (TIM calendar year) available at the NSO Website.

** The official version is '60% are in privately rented accommodation' - heck knows how many of these are in privately rented accommodation but getting Housing Benefit, which costs us even more than social housing, but hey.