I find the idea of Linux phones pointless. Instead of trying to create an app ecosystem that will never compete against proprietary-ecosystem optimized and well-isolated one, we can leverage it. I wouldn't want an OS where the banking app can peek into my browser. Desktop OSes are still like that and that's crap OS design in 2025. It has been crap since 90s. Linux being popular has significantly hindered OS innovation in open source world.
If you would like to help, I think helping projects like LineageOS or GrapheneOS is better. You can also try joining reverse engineered driver efforts for open source drivers like Freedreno. You can help porting device-specific kernel drivers to mainline. So we can boot whatever kernels we want on normal Android phones with Mesa OpenGL.
"A lot of C experience" doesn't really tell anything. Have you worked with cross-language systems? How much you know about ABIs? How about interface definition languages (IDLs)? Have you actually written a production driver for Linux systems? Have you implemented any system-level service that got deployed a number of nodes? You need to join somewhere and improve things marginally.
As I mentioned if the core driver libraries for the userland are reverse engineered (like Freedreno driver in Mesa instead of closed-source Qualcomm stuff) and the kernel drivers are ported to mainline you'll have a better mobile OS than any GNU system can achieve in the same time frame.
If you still want your GNU environment, there are already ones that implement it like Termux.
What makes AOSP a better and more meaningful target for a truly open source system is being also open source and having a much better userland implementation that is supported by specific kernel patches that isolate client programs from direct hardware access and from each other. Moreover it has better libraries for creating mobile apps. It has a better and established ecosystem full of experts, some of which already implement FOSS apps.
No GNU system has anything close and getting close is a huge engineering effort that you don't need to do.
What AOSP does is actually workaround or extend Linux kernel with a system layer that makes the runtime for apps look more like a microkernel. It is a much secure design for carrying a general purpose computer with GPS, Camera and Bluetooth on.
So why not just improve Apache-licensed AOSP? Port the closed drivers to open source ones and everyone can utilize all of AOSP's benefits, without being controlled by Google or phone manufacturers too (as long as you can modify the software). FOSS friendly phone vendors can always help with porting efforts and remove blockers.
I don't know why you need ssh with Xorg on your phone though? There are ways like scrcpy to control your phone remotely. Apps like KDE Connect on Android already give quite a bit remote control opportunities too. There are open source remoting apps that fit the Android user interface better like RustDesk. If you still want an Xorg running, there is nothing stopping you from encapsulating it in an Android app too.
> ssh with Xorg
For example to run gufw on the Phone, but access it on my computer? Sure I could also learn to do it in the shell, but it is so damn convenient. Sometimes it's also nice to run an editor directly on the device itself and not run the editor locally on an ssh-mount.