I totally agree, I will not watch a video with any issues like these.
However, I also hate knowing that I'm hearing mono audio where stereo could be used. 99% of the videos I watch don't have panning issues, so to just turn off ALL stereo seems like such overkill to me...