As I mentioned if the core driver libraries for the userland are reverse engineered (like Freedreno driver in Mesa instead of closed-source Qualcomm stuff) and the kernel drivers are ported to mainline you'll have a better mobile OS than any GNU system can achieve in the same time frame.
If you still want your GNU environment, there are already ones that implement it like Termux.
I don't know why you need ssh with Xorg on your phone though? There are ways like scrcpy to control your phone remotely. Apps like KDE Connect on Android already give quite a bit remote control opportunities too. There are open source remoting apps that fit the Android user interface better like RustDesk. If you still want an Xorg running, there is nothing stopping you from encapsulating it in an Android app too.
> ssh with Xorg
For example to run gufw on the Phone, but access it on my computer? Sure I could also learn to do it in the shell, but it is so damn convenient. Sometimes it's also nice to run an editor directly on the device itself and not run the editor locally on an ssh-mount.