I'm not a compiler expert... But it seems each ARM64 board needs its own custom kernel support, but once that is done, it can support anything compiled to ARM64 as a general target? Or will we still need to have separate builds for RPi, for this board, etc?
Once you get into the CPU though the Aarch64 registers become more standardized. You still have drivers and such to worry about and differing memory offsets for the peripherals - but since you have the kernel running it’s easier to kind of poke around until you find it. Pi 5 added someone complexity to this with the RP1 South Bridge which adds another layer of abstraction.
Hopefully that all makes sense. Basically the Pi itself is backwards while everything else should conform. It’s not Arm specific, but how the Pi does things.