Ranking Factors: Do the Studies Matter?

on under Search Engine Optimization.

As most of you probably already know, Search Metrics recently released their (apparently) yearly collection of ranking factors, to which the SEO industry threw up its arms in excitement and went forth to announce the news to their followers (as am I).

But all is not well.

Michael Martinez recently critiqued this and similar studies, calling them "amateur-level theory." Upon reading it, I noticed a small, non-central statistical jargon mistake and promptly ranted about it in the comment section (with perhaps excessive zeal).

The whole experience made me realize just how little most SEOs know about the statistics behind these kinds of studies, and I felt compelled to address the question of whether or not these studies matter at all, and if so, why?

The Trouble with Correlative Studies

"Correlation does not equal causation." You've hopefully heard this dozens of times, because any remotely legitimate study is going to preface the data with this revelation. The fact that two things seem to coincide with one another doesn't mean that one of them causes the other. In fact, it doesn't even always mean that they share a common cause.

More to the point, the correlations we see in these kinds of studies are very weak. The strongest ones hover in the 30-40 percent range. Even in the social sciences, that's considered a "moderate" correlation.

Another issue is that we tend to believe that a larger correlation implies a stronger relationship. But that's not always true either. For example, the Search Metrics study revealed that putting the keyword in the title was associated with a 0 percent correlation. That makes it sound useless. In reality, it's quite possibly the most important factor, because it was present in nearly every search result.

Correlation is about explained variance. When there is no variance to explain, correlation doesn't help us very much.

And these studies suffer from other problems as well:

  • They rarely mention the crucial phrase "statistical significance," or talk about margins of error, so it's difficult to tell whether the results mean anything. (For what it's worth, after running some calculations, I determined that the Search Metrics study had a large enough sample size to justify all their correlations as significant, in the sense that they exist, not that they are influential.)
  • I'm not sure their samples qualify as truly random. They set upper bounds on the traffic for the keywords and lower bounds on the ranking positions.
  • The top ranking for one search result is not identical to the top ranking for another search result, and the number of positions between two results has a different meaning for each search result. These studies would be more meaningful, and may possibly be more conclusive, if they focused on a binary outcome (either one site ranks above another, or it ranks below it).

That last one really bothers me, because if they used it they could report results in terms of an odds ratio. This would be really nice, because it's something that people actually understand intuitively.

An odds ratio of 5 means that you're 5 times more likely to outrank another site when a certain factor is in place. That's something people actually get.

So Do the Studies Matter?

In a word, yes. But probably not for the reasons that you think.

These studies don't teach us anything about the algorithm. Instead, they show us a few things that successful marketers are doing.

Google's +1 button probably isn't actually a ranking factor, especially considering that they have publicly claimed it isn't. And yet according to the Search Metrics study, it has the highest correlation: 40%. That tells us that if your content tends to get +1s, you're probably doing something that the search engine's like.

Successful SEOs don't waste their time trying to reverse engineer the algorithm. Instead, they conduct experiments on their own sites, measuring the impact of their efforts. Ideally, they have experimental sites of their own, and can test the impact of one thing at a time, isolating variables to the extent that it is possible in this industry. They focus not on rankings, but on traffic and conversions.

Image credit: SearchMetrics