Not even their ancestors at colonial times benefitted much from it: the industrial working class of Britain was in dire position despite Britain being a colonial Empire. That money and power went to the ruling classes and their middle class bootlickers.
It's very silly on the small scale. It's no less silly on the large scale, you are simply more accustomed to the cultural understanding the colonialism is not something you have any responsibility for.