As far as the public understands that complying with a new law is expensive, and why GDPR compliance in particular is expensive, it is obviously more expensive for "bad" companies: don't expect the same compassion and tolerance with which other types of customer disappointments (e.g. raising prices) are received. Your competitors who do not retreat from the EU are obviously caring more for customer privacy, and/or better organized, and/or less reliant on excessive data collection. They are not going to be considered stupid because they spend more than they should on doing the right thing.
You admit bad organization ("there are some areas where we would have to re-architect to gain full compliance"): not trying to comply with the GDPR is clearly not a "rather simple business decision", it's a decision to accept failure instead of losing even more money, and you aren't going to look good even if it's the rational choice in your situation.