Yeah, that is annoying that Android Auto requires such permissions.
OTOH, I have used it twice in two models of car, and in both instances it would stop working until I unplugged and replugged the phone. I just keep the phone visible in the cup holder now.