Plenty of dangerous things stored in `~/`, they don't even need password for ssh-key if there is ssh-agent running (this is in case of dangerous process running, not just upload).
This is why I store keys on a hardware key that requires me to touch it when used and manually start ssh-agent when doing a lot of `git push`.