Where do you get your information from?
Ideally we don’t have people punching genocidal racists because they don’t exist to punch. But if they exist and they’re marching in the streets, often protected by a police force which is (certainly now) quite obviously full of far right-wing violent authoritarians, I’d have other priorities than condemning the anti-fascist.
I disagree with Nazis. I abhor their ideology. I also disagree with punching peaceful marchers, even if they espouse a hateful ideology. "I'm going to use violence to take away your right to peaceful demonstrations" is also an ideology I despise.
Punching them is just proactive defense of those others.