zlacker

[parent] [thread] 0 comments
1. nequo+(OP)[view] [source] 2022-09-22 19:57:27
> Without preregistration it's almost impossible to determine exactly how cherry-picked these differences were

It is hard to credibly preregister studies that use observational data. It also seems hard to design an experiment around the roll-out of a social-media service that we know ahead of time to be successful.

Instead, what is usually done on observational data is (1) making clear what the statistical assumptions are that are required to establish causality, (2) testing possible violations of the assumptions, and (3) testing whether the data is consistent with alternative explanations.

So in such papers, results don't come for free. We need to think seriously about what reasonable theories we can have, and whether the data matches each theory.

> without insight into how the study controlled for the time axis and potential confounding variables about the non-random selection of schools for the rollout, it's difficult to say more.

The paper does also use alternative assumptions that lead to alternative statistical specifications. They also look at various intermediate outcomes to see if they are consistent with their proposed narrative. Such defensive writing is what blows the PDF up to almost 80 pages.

[go to top]