Not necessarily: Depends on whether your key is passphrase-protected and how your SSH agent is configured (if you use one). You can have the standard OpenSSH one ask you for confirmation of every key usage, for example.
> consider a vpn please
But also consider how you'll fix a broken VPN without SSH access.