I've been using XCode for 10 years. For me, it's only improved and I don't have any real pain points. They are definitely fixing bugs. I make software for iOS, macOS, car play, and apple watch.
Sure sometimes I've got to reset or clear a cache, but this has never stopped my day.
What is so horrible about XCode?
Most of the problems I encounter now resulted from the fiasco that is SwiftUI. I used to really enjoy development, and built a few mobile apps singlehandedly... even at a well-known company or two. But now I'm struggling to get one chunk of functionality at a time done on a from-scratch app, because SwiftUI and the underlying "reactive" paradigm were so poorly conceived and defectively implemented. Development is now miserable drudgery, and I never feel confident about the result. The absurd gymnastics to trick the UI into doing what you want; and the hypocritical, conflicting edicts for Swift and SwiftUI... it all barely hangs together.
The fact that Apple rolled out a new UI toolkit that didn't support the most fundamental paradigm in mobile applications, a stack of progressive views for the user to proceed through, tells you how far off the rails they went since introducing the original and solid SDK.
And testing is going to be a nightmare... or simply futile.