I think you're half-right about the critical thinking skills, but the other half is shared context. The bigger the group the less shared context they will have. This gap doesn't necessarily scale linearly, but it also doesn't scale for free unless you're in a cult.