Ideally we don’t have people punching genocidal racists because they don’t exist to punch. But if they exist and they’re marching in the streets, often protected by a police force which is (certainly now) quite obviously full of far right-wing violent authoritarians, I’d have other priorities than condemning the anti-fascist.
I disagree with Nazis. I abhor their ideology. I also disagree with punching peaceful marchers, even if they espouse a hateful ideology. "I'm going to use violence to take away your right to peaceful demonstrations" is also an ideology I despise.