an increase in complexity would rule out reduction of attack surface. in fact attack surface would be guaranteed to increase
As a counter-example -- removing all of Linux's privilege checking would make the code a lot less complicated, but the attack surface would increase a million-fold. In this case, the Librem 5's separation of the baseband such that communication is done over USB (a protocol which doesn't have DMA) is a security improvement over giving the baseband DMA access.
if the security boundary is baked into the code or the design of the system, and also assuming it doesn't introduce more bugs, then I agree[1]. Security controls that get introduced on top do risk an increase in attack surface. An additional interface is by definition a an additional "surface", the question is if it can be attacked.
[1] you could still argue that more lines of code always means more bugs (but let's assume it's very close to bullet-proof)
I have no experience with PCIe so maybe it's harder with USB to abuse the host system, than with PCIe these days.
You can think of USB as being similar to using a TCP/IP protocol between multiple machines capable of executing code, and having to execute code to handle higher level protocols, like HTTP or whatnot. If there's a code execution bug anywhere, the USB capable device will be able to exploit it.
And by default, there's a code-execution bug on all normally configured Linux machines. If you'll not create a USB "firewall", modem can just create a virtual keyboard and kernel will happily accept all input from it, for example. So modem can just type whatever it wants to your shell. It will be obvious, but, it's still device->host RCE.