The remnants of colonialism continue to produce winners and losers economically, with the winners stuck in local maxima where they extract value from the people, but the people themselves see only marginal benefit, and development is stuck at a snail's pace.
As with seemingly everything in life, the incentives for the different players really don't line up. Consumers lose, producers lose, and only a select few middlemen win anything at all.
The reason New York City is the biggest city in the US is because when the Erie Canal was built, the agricultural riches of the Midwest had a route to world markets. Where you have a major seaport, you also need major banks and major insurance companies to smooth out the financial needs of traders and shippers, providing the funds right away back to the farmers, instead of them waiting till the voyages were complete. (without the Erie Canal, New Orleans would have become the largest city in the US)
Yes, there is a lot of money in trading, banking, etc. At every step of the transaction pyramid, a %age is added to the price, and the %age fees charged on that go up accordingly. But that measures the true value of the product at each stage; if you have a cheaper way of getting the same product to the same stage cheaper, the (supposed) riches will be yours.
The socialist instinct ("anybody getting rich must be cheating") unfortunately obscures the real problem ("monopolists and cartels controlling supply and setting prices are the true enemies of the people") which hinders solving it; by putting capitalism in your gunsights, you make enemies out of natural allies.
Also the fact that everyone alive today is so as a result of either being on the winning side of colonialism or from their ancestors otherwise clubbing somebody else over the head.
That's a poor justification for it, though, and I don't want to live in a society that goes out of its way to view itself as Hobbesian.