It was more the later additions to the EU a few years later that were actually problematic and got people's backs up.
It didn't help that we were allowed to restrict immigration from those countries but didn't as the government needed mass immigration to disguise the fact there was no growth.