zlacker

>>imslav+(OP)
Seeing a 20% margin of error for some of their tests (due to VMs, noisy neighbors), makes me curious how others solve this problem. Dedicated hardware seems like a good investment, otherwise you need to constantly overcompensate with multiple runs.

>>yellow+g41
Hey, I am one of the authors of the article and the systems described.

The default 20% margin of error is indeed pretty wide, and it is intended to catch large obvious regressions (e.g. an algorithm accidentally becoming quadratic instead of being linear)

As we described in the blog post, we have the second system based on the real hardware. This system is on-demand. If an engineer has a suspect commit or an experimental change that might affect performance, they would schedule a 30min-1hr run on that queue, where we run selected benchmarks 9-15 times each on laptops of various strength. In this configuration, the margin of error is closer to 2-3% from our observations so far. To get more confidence, you would want to run even more trials, typically we advise 9 iterations, though.

We also do all our daily benchmarking on those laptops too.

Edit: in addition to preventative testing, we also track production metrics in a similar way as described by the sibling comment

>>imslav+m81
Thanks for your reply. Can you share how these laptops are managed? I took another look at the article but didn't find that information there.

I.e I'm curious if there's a cloud provider managing them for you or you guys keep them in a closet somewhere.

>>yellow+ay2
We omitted the details for the sake of brevity (the article is already really long), but we run agents on the test devices and plug them into CI. Agents allow remote access for servicing remotely, and that's enough most of the time.

For things like system updates and taking care of the hardware - we do it manually today. The fleet is still small, so it is manageable but in the future we would like to consider a vendor, if we can find one.

>>imslav+vr4
Thanks!