That's not strictly true - British Indian Ocean Territory has permanent inhabitants, just not any native ones (never had had them, really - it was uninhabited until 1793). US military Diego Garcia base is there...
It's bullshit for other reasons, and expulsion of Chagossians to build the base is a tragedy - but not due it being empty territory (it's not).