The baseband is permanently attached to a public network. Not having control over whether that connection actually is up is a huge security hole. The entire baseband software stack runs in supervisor mode. There are no non-executable pages, there's no stack protection.
EDIT-1: Qualcomm baseband chips have location tracking baked in. Even with a clean OS and no tracking apps, the baseband does it. The tracking data is commercially available: https://web.archive.org/web/20180514003056/https://www.qualc...