For example: SSH clients and servers. I've tried playing with various apps from F-Droid to make it happen, with varying success. If I have just Linux, I know what should work. Another one is backups. I have a better sense of what I need to back up on a common Linux system than an Android-like one.
Last year, I realized that the direction that smartphones have taken is not a direction that is good and useful for me, so I decided that I'd better design and build my own smartphone before I need to replace the one I have. I have it breadboarded up and fundamentally working. It should be completed this year.
One of my very first decisions was that the phone was going to run ordinary Linux, for all of the reasons you cite. I'm VERY excited about being able to ditch Android!