zlacker

We had a fun bug where our VPN was crashing on macOS. The error was pretty clear, we were subtracting two timestamps and getting a negative, which should never happen as these were from a monotonic clock. We spent lots of time analyzing all of the code to make sure that the arguments were all in the right order and being subtracted from the right values and everything looked fine.

However we still saw these crash reports from one device (conveniently the partner of the CEO, so we got full debug reports). However the system logs were suspicious, lots of clock jumps especially when coming out of sleep. At the end of the day we concluded it was bad hardware (an M1 Max) and the OS was trusting it too much, returning out-of-order values for a supposedly monotonic clock. We updated the code to use saturating arithmetic to mitigate the problem.