I've been using XCode for 10 years. For me, it's only improved and I don't have any real pain points. They are definitely fixing bugs. I make software for iOS, macOS, car play, and apple watch.
Sure sometimes I've got to reset or clear a cache, but this has never stopped my day.
What is so horrible about XCode?
Often autocomplete is hopeless and doesn't help you with the most simple things like picking from multiple initializers, or changing to a different signature of a function call.
the project setup is this mystery Xcode project file, instead of a standardized yml or something that anyone can modify and understand.
I have to say provisioning has improved a lot. I remember back in 2008, it was really a pain to get anything working.
This is not necessarily about Xcode, but maybe it should be: screen shots for your app. they need screenshots for 454 device types, and zero automation in their own tooling.
the layout is also very inflexible. they dictate a couple of panels and that's how you _must_ use them. that's unlike any other modern IDE.