I also have some difficulties doing "brain work" while listening to music for the same reasons, I just start paying more attention to the music.
I'm a hobbyist musician and have been playing trumpet and saxophone since I was around 8~10. I wonder if that changed the way I (or we) perceive music