Now apply your logic the other way (parents having many children) and see how meaningless your conclusion is.
That the unicorns known as children in large families still see all the other families around them who have just one kid, and internalize that as the norm too, or maybe their own family does influence them slightly and they bump their ceiling up to 2 or 3, and within a few generations that's right back to the mean again?
After age 5, what percentage of waking hours do children spend away from their family anyway? What percentage of adults that they are familiar with (at least to the degree that they know how many children those adults have) are their own parents? Are the people (in the US, they'd be mostly women) that teach them also parents to some high number of children?
Even the Duggars must have grown up internalizing a number far lower than the 20+ their own parents had.