I wonder if there could be some way to sign a dynamic library to allow it to create direct system calls and then pass that as a kernel command line argument at boot?
It is not the only operating system in the "unstable kernel interface" group though. Linux is actually the only one with a stable system call interface.
I've written somewhat at length about this:
It's wrong to say that Linux stands alone in having a stable syscall interface. FreeBSD [0] and NetBSD [1] both retain syscall compatibility for old binaries (the former apparently with some exceptions permitted); DragonFly BSD also appears to keep old syscalls in place. In fact, I only know of Windows, OpenBSD, and presumably macOS as mainstream desktop OSes without mostly-stable syscalls.
[0] https://wiki.freebsd.org/AddingSyscalls#Backward_compatibily
[1] https://www.netbsd.org/docs/internals/en/chap-processes.html...