zlacker

>>azhenl+(OP)
I regret causing confusion here. It turns out that this correlation was true on the initial small data set, but after gathering more data, the correlation went away. So the real lesson should be: "if you gather data on a lot of low-frequency events, some of them will display a spurious correlation, about which you can make up a story."

>>norvig+R9
You and/or Google HR were also prominently quoted as saying, IIRC, that GPA, standardized test scores (and interview ratings?) had no observable correlation with job performance either. I always wrote that off as just range restriction/Berkson's paradox, but did those also go away?

>>gwern+Ie
Interview ratings would be a surprising one, hard to imagine not overhauling the system after discovering that.

>>titano+8f
I remember reading something about Google interview ratings being uncorrelated to job performance, but that's clearly a conditional correlation based on the person being hired. For all we know, the ratings might be highly correlated with job performance up until the hire/no-hire cutoff. After all, their primary purpose is to make that binary hire/no-hire decision. Hopefully, the scoring system is hyper-optimized to be a good signal right around the hire/no-hire boundary, as the scores themselves aren't that useful for obvious hires and obvious no-hires: the scores are a decision-making tool.

In order to really get a good assessment of if the interview ratings were effective, they'd need to also hire some random unbiased sample of those who fail the interview process. There are alternative ways of slicing the data to help give insight, such as looking at only those who barely passed the interview process, or looking only at the bottom 10% of performers. However, when you're looking at such a highly biased sample (only the small-ish percentage of people hired), it's hard to say what the correlation is across the entire interview population.

At the risk of repeating myself, we don't particularly care the predictive power of the scores across the whole range, only their predictive power across those who aren't obvious no-hires and those who aren't obvious hires. That's the range where the power of the interview scores as a decision-making tool is most important.

Also, if two metrics disagree, it's not clear which one is problematic. It's possible that a poor correlation indicates that there's a problem with the performance rating system.