I've experienced this too. Swapping seems very slow and inefficient on Linux. No matter how large your swap is, once you hit the point where you're using twice your physical RAM, the system become unresponsive. My guess is Linux doesn't prioritize processes properly for desktop use, and background threads, despite being from minimized windows (minimized by the user in a desperate and futile attempt to open a terminal so I can kill some processes), are free to demand memory with the same priority as everything else.