The slice is only narrow because it’s practically impossible. If there were an option presented to end users “let X.com read data from Y.com?” there would be a rich ecosystem of alternative UI’s for any website you could think of.
These alt-UI’s would be likely to have better security practices than the original, or at the very least introduce competition to drive privacy/security/accessibility standards up for everyone. Whereas currently if the Origin has the data, they have full ability to impose whatever draconian practices they want on people who desire to access that data.