I've been using XCode for 10 years. For me, it's only improved and I don't have any real pain points. They are definitely fixing bugs. I make software for iOS, macOS, car play, and apple watch.
Sure sometimes I've got to reset or clear a cache, but this has never stopped my day.
What is so horrible about XCode?
Stepping through the debugger too fast will sometimes put the debugger in a weird state where step never breaks again and all other breakpoints stop working.
Git pull through the UI with stash and merge can blow away your local changes if there is a conflict. The changes aren't stashed. They're just gone.
Xcode likes to sometimes recompile files that haven't changed slowing everything down, sometimes significantly depending on the file. No idea why.
It can get very confused if you're missing a parenthesis in the wrong place in a SwiftUI View leading to opaque swift compiler errors about code being too complex.
Even mildly complex use of a swift #Predicate will cause an error about it being too complex forcing you to break them down into smaller pieces and even then it takes far too long to build even on a brand new machine.
The simulators are quite slow to start/update/run and xcode sometimes fails to shut them down completely when quitting leading to them just continually running eating memory unless you kill the processes manually.
The simulators also are really limited in their functionality. No background processes, spotlight, network degradation simulation, out of memory killer, etc.
The profiler sometimes just fails to start an app correctly, immediately ending a run forcing you to close the profiler and reopen it again before it'll start working.
Symbol refactor (rename) can be painfully slow where the UI just locks up until it can find all the references.
Xcode likes to duplicate package dependencies in xcodeproj. It just creates new hashes for the same library and adds it as a dependency over and over again, so when the link phase happens, it adds libraries repeatedly over and over and over again unless you manually clear them out. Not sure what causes this, perhaps updating the version or merges between users.