I find the idea of Linux phones pointless. Instead of trying to create an app ecosystem that will never compete against proprietary-ecosystem optimized and well-isolated one, we can leverage it. I wouldn't want an OS where the banking app can peek into my browser. Desktop OSes are still like that and that's crap OS design in 2025. It has been crap since 90s. Linux being popular has significantly hindered OS innovation in open source world.
If you would like to help, I think helping projects like LineageOS or GrapheneOS is better. You can also try joining reverse engineered driver efforts for open source drivers like Freedreno. You can help porting device-specific kernel drivers to mainline. So we can boot whatever kernels we want on normal Android phones with Mesa OpenGL.
"A lot of C experience" doesn't really tell anything. Have you worked with cross-language systems? How much you know about ABIs? How about interface definition languages (IDLs)? Have you actually written a production driver for Linux systems? Have you implemented any system-level service that got deployed a number of nodes? You need to join somewhere and improve things marginally.
As I mentioned if the core driver libraries for the userland are reverse engineered (like Freedreno driver in Mesa instead of closed-source Qualcomm stuff) and the kernel drivers are ported to mainline you'll have a better mobile OS than any GNU system can achieve in the same time frame.
If you still want your GNU environment, there are already ones that implement it like Termux.
I don't know why you need ssh with Xorg on your phone though? There are ways like scrcpy to control your phone remotely. Apps like KDE Connect on Android already give quite a bit remote control opportunities too. There are open source remoting apps that fit the Android user interface better like RustDesk. If you still want an Xorg running, there is nothing stopping you from encapsulating it in an Android app too.