Android is extremely complex so I think many of the custom ROMs possibly have some security rookie mistakes and quite a bit security bugs due to mishmash of drivers. Android is still better than most of the Linux distros due to its architecture though. The default setup of many distros doesn't have much isolation if at all.
I would easily believe that many Android systems have vulnerabilities owing to the horrific mess that is their kernel situation. That said, I personally doubt that aftermarket ROMs are worse than stock, as official ROMs are also running hacked up kernels.
Do you mean OEM drivers or the Android Kernel, specifically?
Google invests quite a bit on hardening the (Android Commons) Kernel including compile-time/link-time & runtime mitigations (both in hardware & software).
Ex: https://android-developers.googleblog.com/2018/10/control-fl...