Mm. On the Studio, my desktop switching is pretty quick - 17 frames from tapping the ctrl/arrow. I have an M4 Max which is actually 1 frame faster at 16 frames (though measurement-error probably accounts for that).
In either case that's less than 1/3 of a second. It's nowhere near what you're seeing. I'd be tempted to take the machine to an Apple Store and say "this is broken, please fix it".
>>spaced+(OP)
What frame rate are you using on those monitors? I ask because this is faster at lower frame rates. This is surprising and I very much want what you have.
Are you using software like Rectangle, Better Touch Tool, etc?