We're as much a part of the planet as its rocks, and we care. You could say that your bones don't care, but the organism that is you cares. In the same way the planet cares through the subsystem that includes us even if its rocks don't care.
You're right if you define a planet as its rocks, and would also be right to assert that humans don't care if you defined a human as its skeleton.