I've been using XCode for 10 years. For me, it's only improved and I don't have any real pain points. They are definitely fixing bugs. I make software for iOS, macOS, car play, and apple watch.
Sure sometimes I've got to reset or clear a cache, but this has never stopped my day.
What is so horrible about XCode?
If you were able to use a magical, fast, bug-fixed, version where everything 'just worked' at pace, I imagine you would begin to realise just how much you have been putting up with!