The Librem 5 doesn't have one driver exposed to the baseband. It has every single USB driver in the kernel exposed to it, because the baseband can present any descriptors it feels like and engage whatever driver it wants, or a combination thereof by presenting itself as a composite USB device, since USB is plug&play. That is a massively larger attack surface. All you have to do is find one exploitable bug in any USB driver in Linux, and you're in.
This can be mitigated with USB descriptor filtering, but the Librem 5 guys haven't implemented that yet, because they don't actually care about security. So while their marketing department is lying about DMA access for the competition (heck, as far as I know no iPhone gas ever given the baseband unchecked DMA access to the system, but Purism claims they all do!), their engineering department can't even bother to lock down the attack surface of the baseband to something smaller than "every single USB driver in the kernel".
Also, for what it's worth, the Librem 5 doesn't even have an IOMMU at all. They can't even use the PCIe ports in their SoC because that would give whatever you plug into them full DMA to the system. This also means that driver bugs that result in bad DMA descriptors for embedded SoC devices will directly escalate to full memory access; there is no safeguard by having to engage the IOMMU subsystem first to map them.