The server needs to parse the message headers, so it can't be an opaque blob. If the client uses IMAP, the server needs to fully parse the message. The only alternative is POP3, where the client downloads all messages as blobs and you can only read your email from one location, which made sense in the year 2000 but not now when everyone has several devices.
POP3 is line–based too, anyway. Maybe you can rsync your maildir?
I use imap on my mobile device, but that’s mostly for recent emails until I get to my computer. Then it’s downloaded and deleted from the server.
I don't have an IMAP account available to check, but AFAIK, you should not have locally the content of any message you've never read before. The whole point of IMAP is that it doesn't download messages, but instead acts like a window into the server.
IMAP is an interactive protocol that is closer to the interaction between Gmail frontend and backend. It does many things. The client implements a local view of a central source of truth.
POP can download the email, and that's about it.
Gmail basically is IMAP with a couple extras, and your desktop (via a browser) and your phone (via a dedicated app) can both see the same messages. Only the phone can work offline though, because there is little demand for a dedicated desktop email client, it's always via a browser. But Google could easily make such a thing if they wanted.