That's an interesting question re: genetically-driven tendency to criticize those who advocate having less children. I've had the same experience when voicing my opinion as it seems you have, and I'm still not quite sure where its rooted. From the outside, it just seems like a kind of species-selfishness, like "we can be invasive but no one else can or we'll exterminate them!" like we do with deer.
I've found there's a whole philosophy that seems to line up with my (and maybe your) perspective:
https://en.wikipedia.org/wiki/Deep_ecology