Thursday, September 23, 2010

A/B Testing - We'd Love Your Thoughts

It's been on our agenda for a long time but urged on by some commenters on this blog, particularly Niki Scevak who I caught up with last week it's finally made it to the top of our to do list!

Based on our traffic and conversions we think we'll need to run each test for about 1-2 months to collect enough data to get a statistically significant result, so we want to make sure that the things we test have a reasonable chance of making a decent impact to our conversion rate.

We have a few ideas of things to test but we're probably a little too close to the website to judge it properly, so we thought we'd ask for people's thoughts here.

If you were going to run an A/B test on Shoes of Prey with the goal of increasing our sales conversion rate, what would you test first?

Update: A friend made the good point that I haven't actually explained what an A/B test is and not everyone is a web geek - good point! An A/B test is a structured way of testing two different variations of something on a website to see which has a better outcome, in our case, conversion rate. Simple tests can include testing two different photos on a home page to see which converts better, or two different coloured 'Add to cart' buttons. There's a great example on the website Which Test Won? at the moment of a blue shadow behind an 'Add gadget to your site' button increasing versions by 50.5%! Most variations aren't quite that dramatic, but 20% improvements are not out of the question for a lot of A/B tests.


  1. I'd do a homepage test.
    Version A: As is, primary message is to design your own shoe
    Version B: Present 2 options; 1. Browse our range and customise. 2. design your own shoe.
    And maybe Version C: Primary message Browse shoes and customise, secondary message Design your own shoe from scratch.

    My objective would be to test if getting people to product pages/exploring/inspired first, before presenting a blank slate, helps convert prospects to buyers.

  2. Before doing any testing, test yourself with

    I would test landing pages for niche needs - weddings, odd foot sizes, cross dressers, etc. That way you don't need to mess with any of the current site.

  3. Do you have the ability to test the impact of the phone number and street address to see what impact the reassurance of a physical presence has?

    Keep the blog posts coming Michael - they make great reading and Janine Allis had better watch her back!

  4. what would be fascinating is to combine the A/B testing with understand which of your messages relates to the stages of AIDA, and how you can measure a customers progress through Awareness, Interest, Desire and Action. With the stat's you've already got, this could form some very interesting decision making and design criteria.

  5. Thanks for the great ideas!

    Jeni - That's a fantastic idea, it would be great to know whether having people start with a blank template converts differently to having them start designing with a blank slate. That's the sort of thing where there could be a big difference in conversion rates... we should definitely test that one, thanks!

    Chris - That's a good suggestion, it would be good to know what that impact is. I'd be amazed if including an address and phone number had a negative impact, but we could test whether adding them, or at least our phone number to all our pages has an impact versus just having them on the contact us page as we do now.

    Dom - Good thoughts Dom, a series of tests along those lines would be great. We probably don't have enough data yet to be able to do that, we think for now we can only run one test everyone month or two but I like that idea for down the track.

  6. Hey Michael, thanks for the shout out. I'd focus on your checkout process as that's the closes to the money and has the highest leverage.

    Check out this blog post as he's incorporated a lot of the important concepts:

    For the important psychological factors behind testing, checkout:

  7. Test (significant) reduction of choices on the site. Customers actually want solutions not so much choice!

    See: "Paradox of Choice" in Wikipedia.

  8. Niki - Good suggestion to focus on the checkout process. We're actually redesigning that at the moment and it's satisfying to see that the new version that's in the works is actually quite similar to the one the experienceblogger post goes through. That's an amazing improvement they had, more than a 100% increase in conversions!! The wider funnel post is interesting too, that provides a provides a great framework for coming up with ideas of things to test. Thanks for forwarding those both through Niki!

    Merchandizing 101 - this is a great suggestion. It's going to take a bit of work to do, but Mike and I were discussing this yesterday and think we should test a version of our shoe designer with half the options removed. We should go through and remove the least popular half of all the options we have then test if that increases the conversion rate. Thanks for the great suggestion, we don't have time to do it immediately but we'll definitely be doing this in the not too distant future!