Perhaps something to help you cope with this is that our brain although fragile has a lot of redundancy around it.
On a fall or bounce our neck will decelerate our fall/bounce, there is no need to be an athlete, our muscles have a contraction reflex if they are violently streched (myotatic reflex), less so for very young babies with weak neck muscles (think less than 6/9 months).
Then our skull is filled with fluid which has inertia so the force on the skull is not directly transmitted to the brain. You have to slush the liquid around quite a bit before your brain experiences any meaningful force.
Effectively it is only with some extreme force or internal bleeding compressing the brain that brain damage would occur.