Split Testing Ideas: Why You're Not Getting Anywhere

on under Tactic Tuesday, Website Testing.

If you've ever read a single guide on split testing ever, you've almost certainly heard this before:

"Test one thing at a time."

Well, I hate to break it to you, but when most people take that advice to heart, they end up shooting themselves in the foot. The people who will tell you to do this are citing ideas from science and statistics out of context, and convincing you to waste your time.

Since the argument is that this is the scientific way to do things, let me start with an example from serious science: pharmaceuticals.

Yes, what they do is basically split testing. They take a random sample of people who need treatment, they randomly assign half of them to take a fake drug made of sugar (placebo) and half of them to take the real drug. Then they compare results to see if they've reached statistical significance, hopefully demonstrating that the real medication is better than the placebo.

And yes, it would be stupid of them to instead put them on ten different medications and see what happens, because they would have no idea which medications were doing the work, causing the side effects, etc.

But let me tell you what pharmaceutical companies don't do. They don't take the sugar compound from their placebo, randomly switch out one carbon atom for a hydrogen atom, run a split test, and hope for the best. And that's smart. Because the FDA would probably put them in jail.

But that is essentially what most "conversion rate optimizers" are doing when they switch out one individual page element for another. They are living in a bizarro world where it's possible to get from a crappy landing page to an awesome one by switching out one individual line of text at a time.

Most split testers are treating each and every page element like it was part of a massive pharmaceutical regimen, when really the whole damn landing page is the pill in question.

I want to avoid speaking in absolutes here. Yes, sometimes you should consider changing just the headline or just the CTA. But this shouldn't really even be on your mind until you've tested your entire core landing page concept to death, and you're sure the basic premise you've settled on is the right one.

Here's the thing. When you create your original landing page, you're using it to create one cohesive message. When you change one little piece of it at a time, you're either not going to change the message very much, or you're going to create a hodgepodge of fragmented ideas that mean next to nothing to your consumer.

Instead of testing only one thing at a time, test only one relevant thing at a time. Here's how.

Start with the Ten Principles of Influence

We've gotten into this before, but basically, there are ten principles from consumer psychology that are hugely relevant to all marketing and sales.

  1. Reciprocity - Do something for them, and they'll do something for you.
  2. Commitments - Get them to make a small commitment, and they'll be more likely to make larger commitments in the future.
  3.  Authority - They're going to trust you much more if you seem authoritative or if you've been given the seal of approval from somebody who is.
  4. Social Proof - If others trust you, visitors will be more likely to trust you, especially if they have something in common with the people who trust you.
  5. Rarity - The more rare the opportunity, the more appealing it is.
  6. Affinity - Visitors tend to work with you if they like you, or feel like they have something in common with you.
  7. Loss Aversion - Visitors are more likely to take risks to avoid a loss than to win something.
  8. Status Quo Bias - Visitors prefer inaction to action and will choose the path of least resistance.
  9. Default Bias - When visitors face an overload of options, they tend to choose no option at all, or what appears to be the "default" option.
  10. Availability Bias - People decide how likely something is based on how easy it is to remember, not on how probable it actually is.

Think of these principles like medications. When you do your tests, you'll only want to change the "dosage" or existence of one principle at a time. But remember, these are principles you are testing. Adjust one of them, and your entire landing page is probably going to look different.

You're probably going to want to start with landing pages that use only one of these principles, because the best landing pages will tend to have just one of these as the dominant message. Once you identify the strongest principle, you can start testing secondary principles, and so on.

You're going to want to do this for a while, because people respond to messages far more strongly than they do to specific design quirks (barring complete ugliness, which is really about authority anyway).

Now You Can Test Other Things

Once you're sure you've found the right landing page concept, you can start testing smaller things like headlines, calls-to-action, directional cues, white space, colors, etc., as long as these aren't changing the message.

Think of it like this. When you test these smaller things, it's like you're scaling a mountain. Yes, you want to get to the top of the mountain, but you need to make sure you're starting at the base of the right mountain first.

When you start by split testing page elements instead of concepts, you will usually end up scaling the local bunny hill, instead of Mt. Everest.

And that's a problem.

Image credit: Steve Jurvetson

  •  
  •  
  •  
  •  
  •