I purchased a license of a proper antivirus software to avoid that bug and the performance issues gone away.
When you install another AV software, Windows Defender steps down and leaves scanning to the 3rd-party security solution. I selected one of the most lightweight ones I could find. It has been a net win for me.
One shouldn't need to do this, but it has worked so far, for years now.