This shouldn't be controversial at all and I think your take is 100% correct wrt exceptions (gov work, access to children etc). The very "Once a felon always a felon" thing going on in the united states is a secret life long sentence that completely defeats the idea of redemption.
Now because that freedom of association, freedom to chose who you do business with does not exist, we think this is reasonable.
However, it is not. I own a business. I should not be required to hire anyone I do not want to. Full stop!
And it doesn't have to be a race card thing, Say I hate the Iraq war. Say I would like to never hire any veteran who contributed to the deaths of over 1 million Iraqi Citizens. Or say I worked with Iraq war veterans in the past and I hate their military attitude. I am not allowed to do so. If I bought a company, I would want to fire all veterans. No, you are not allowed to do so.