zlacker

[return to "Measuring the impact of AI on experienced open-source developer productivity"]
1. AvAn12+Bn[view] [source] 2025-07-10 18:50:21
>>dheera+(OP)
N = 16 developers. Is this enough to draw any meaningful conclusions?
◧◩
2. sarche+Hp[view] [source] 2025-07-10 19:04:01
>>AvAn12+Bn
That depends on the size of the effect you’re trying to measure. If cursor provides a 5x, 10x, or 100x productivity boost as many people are claiming, you’d expect to see that in a sample size of 16 unless there’s something seriously wrong with your sample selection.

If you are looking for a 0.1% increase in productivity, then 16 is too small.

◧◩◪
3. AvAn12+5F[view] [source] 2025-07-10 20:39:29
>>sarche+Hp
“A quarter of the participants saw increased performance, 3/4 saw reduced performance.” So I think any conclusions drawn on these 16 people doesn’t signify much one way or the other. Cool paper but how is this anything other than a null finding?
[go to top]