I've been using XCode for 10 years. For me, it's only improved and I don't have any real pain points. They are definitely fixing bugs. I make software for iOS, macOS, car play, and apple watch.
Sure sometimes I've got to reset or clear a cache, but this has never stopped my day.
What is so horrible about XCode?
When I used it in a team, I had to write a build step that would fail the build if the pbxproj file wasn’t sorted. (Plus a custom target that would sort it for you.) It was the only way to make sure it never got unsorted in the first place.