Random Correlations

I’ve noticed people frequently misusing data to find correlations between seemingly unrelated data sets and inferring a relationship. While they’ll generally volunteer that they haven’t proven causality, they frequently claim that there must be some underlying relationship for the p value to be so low.

I built a toy to try and show the error in this. Essentially, you can take almost any real life data and infer a relationship, especially if you perform multiple tests. Here I take a number of data sets from Quandl and plot whichever have very low p-values.

Wait for it to load, then hit the “Another Relationship!” button

The causes of these ‘relationships’ vary, but a few key factors that I think are generally worth checking. These don’t invalidate the slope or intercept, but they may call the test statistics into question (e.g., p value).

• Are the residuals normally distributed?
• What if I detrend the data?
• Are the residuals autocorrelated?
• Do the residuals have constant variance?
• Are there any points with a lot of leverage?
• How many relationships did I test before finding this? Do I need to apply a multiple testing correction?