zlacker

>>imslav+(OP)
Seeing a 20% margin of error for some of their tests (due to VMs, noisy neighbors), makes me curious how others solve this problem. Dedicated hardware seems like a good investment, otherwise you need to constantly overcompensate with multiple runs.

>>yellow+g41
At Framer (framer.com) we actually had a bit lower variance (I think it was closer to 15%) after spending three months of solid engineering, ending up with a dedicated Mac Mini running Chrome.

Apart from the admin overhead (things got stuck on OS updates) we ended up abandoning the setup because the variance was too big to get anything useful out of running tests for every pr.

The most reliable way for us to monitor performance today is to count slow frames (>16ms) across all actual users and divide them by total time spent in app. It’s a rough proxy, but pretty accurate at showing us when we mess up.