Where do you get your information from?
Ideally we don’t have people punching genocidal racists because they don’t exist to punch. But if they exist and they’re marching in the streets, often protected by a police force which is (certainly now) quite obviously full of far right-wing violent authoritarians, I’d have other priorities than condemning the anti-fascist.