"You are standing in a field looking at the stars. Your arms are resting freely at your side, and you see that the distant stars are not moving. Now start spinning. The stars are whirling around you and your arms are pulled away from your body. Why should your arms be pulled away when the stars are whirling? Why should they be dangling freely when the stars don't move?"
It's the same reason the universe has an average speed (unlike what you might expect from special relativity), although it is unclear if this is true for the entire universe or just the portion we can see. We can measure how fast we're moving w.r.t the cosmic microwave background radiation though (it is red-/blue-shifted in a particular direction).