I know port scanners are a thing but the act of using non-default ports seems unreasonably effective at preventing most security problems.
Its security through obscurity, which puts you out of view of the vast majority of the chaos of the internet. It by no means protects you from all threats.