And the distance - most asteroids pass by much further out than even the moon, so their motion would be hard to detect.