So an American permanently living in the UK will be considered an expat. A polish will be called an immigrant. Despite the actual definition of the word, "immigrant" has a negative nuance artificially attached to it so people use it to this effect.
In Europe I noticed that skin color makes less of a difference than "source country". Many British people will still treat African-American individuals as expats, and an Albanian as an immigrant, once they find out where they came from.