You point it at your email, and you've opened a vector for prompt injection and data exfiltration - all as an integral part of the features you want (read my emails, send some emails).
Your local LLM won't protect you there.
You could probably write some limited tools (whitelist where mail could be sent) - but it goes against the grain of "magically wonderful ai secretary".
Security is not a convenience.