Posted September 9, 2008 9:21 am by with 8 comments

Tweet about this on TwitterShare on LinkedInShare on Google+Share on FacebookBuffer this page

Last week we gave you an opportunity to win a copy of Bryan Eisenberg’s new book “Always Be Testing.” The response was so overwhelming, I asked Bryan if he would mind answering a few questions about the book and the concept of mult-variate landing page testing.

Here’s the interview:

For those that are not aware, what are the main differences between A/B testing and multi-variate testing?

Bryan Eisenberg: A/B testing is where you test the impact of one value of a particular variable (“A”) versus another value for that same variable (“B”). For example, does a green button have a bigger impact on your clients than a red button? Here the variable is “button color” and the values are “green” and “red”. Do customers respond better to a coupon with “Buy One, Get One Free” or do they prefer “50% off”? Again, one variable (coupon pricing) and two different values. Of course you can have more than just two values: obviously you can also test blue, yellow and orange buttons, or “Free with $10 purchase” or “spend $300 before November 15 and get your Thanksgiving turkey free”.

“A/B” testing techniques can work great when you are comparing the results of one page versus the results of a second page. For example, which landing page from your radio commercial converts more customers, the one with the great copy and no pictures or the one with ok copy but a picture of a pretty lady? You can do this for any number of pages and for any number of changes on those pages.

Multi-variate tests deal with multiple variables at the same time. For example, you want to test button color (as above) but also button shape (round? square? rectangle?) and button text size (“small”, “large”). Because there are multiple variables, the test is referred to as “multi-variate”. You might also accurately describe “A/B” testing as “uni-variate” because it only tests one variable, but only the geeky people would say it that way.

Multi-variate tests are best when you want to test isolated variables on a page. For example, you may want to test 3 versions of your headline, 4 versions of your main image and 6 versions of your call to action to find the best combination for that particular page. So instead of having literally different pages, you have different combinations of those variables on a single page, swapping out the various versions to see which combination does best.

You can do both kinds of tests easily with Google Website Optimizer. GWO refers to all multi-page tests as “A/B” tests and all single-page tests multi-variate.

What are some of the web page variables I should be testing at a minimum?

Bryan Eisenberg: In “Always Be Testing” we talk about 30 different factors that we found influence conversions. You should be testing all of them. To get started, however, you should look at headlines, calls to action and trust-building elements. These are all elements we covered in a fair amount of details on our previous “Always Be Testing” webinars that we recorded with Tom Leung, Product Manager for Google Website Optimizer at Google and can be viewed online at .

[Andy’s note: Bryan Eisenberg will join the Get Elastic team for an exclusive free webinar this Thursday, September 11th.]

Is this book only good if you use Google Website Optimizer?

Bryan Eisenberg: No, in fact it’s focused on how to do good testing, generally, with specific examples shown from Website Optimizer. This book leaves off where “Call to Action” ended, giving you insights into what you should test in order to improve your conversion rate. We wrote the book about GWO because, well, the publisher asked us, and also because we know people have to get started testing somewhere and a free solution is a good place to start. The book is maybe 20% about GWO and 80% about how to get more out of your marketing by testing and creating a culture of testing at your organization.

How long should I run a test? What if I don’t get enough visitors or need to finish the test early, will it still provide value?

Bryan Eisenberg: The short answer is until GWO reports that your results are 90% confident to beat the original. The simple answer is at least one week to allow for day of the week variability and in about 2 weeks if you don’t see any clear winner, pause the test, prune a couple of the losers, or stop the test and redesign your experiment for something that has a greater impact. The point is to move the needle impactfully — even if the direction is not where you wanted — and to tweak and move on to the next round of testing and optimization.

Can you share with us a case study of a company that benefited from A/B testing?

Bryan Eisenberg: We have several stories in the book, but one of my favorites is Jigsaw Health. Before working with us at FutureNow and using Google Website Optimizer, the company’s cost per acquisition (CPA) ran anywhere from $150 to $300 per month. By using Google Website Optimizer and making adjustments to Google AdWords™ campaigns, Jigsaw Health reduced the CPA from that $150-$300 range down to under $40 and their conversion rate went from 10.3% to 19.6%. That’s slashing acquisition costs by more than 75% while at the same time almost doubling conversion. That sort of combination of FutureNow’s Persuasion Architecture® combined with intelligent testing using even a free platform like Website Optimizer that can really impact the bottom line. But you have to get started!

Awesome! Thanks Bryan! For regular insight from Bryan and the FutureNow team, check out their GrokDotCom blog.

  • That Jigsaw Health result is extraordianry … we’re currently trying to persuade a client to try Google Website Optimizer, but they’re somehow reluctant to “hand over data” to Google. Perhaps this post will give us some more ammunition.

    Ami Ohayon’s last blog post..From Jerusalem to Boston

  • Definitely excited to get my hands on this book. Thanks for answering my question Bryan.

    Utah SEO Pro’s last blog post..Building an SEO Friendly Site – SES San Jose 2008 Presentation

  • Although I guess I’m theoretically a competitor of Bryan’s, “Always Be Testing” is a great read, especially if you’re getting started with GWO or website testing. Even for seasoned testing professionals, Bryan has a great way of asking questions that get you thinking, especially when you’ve stared at the same site for months or years.

    Dr. Pete’s last blog post..Search Without SERPs: The Future of SEO?

  • @Utah SEO Pro – happy to answer your question.
    @Dr. Pete – thanks for the kind words about the book 🙂

    Bryan Eisenberg’s last blog post..Estimating Gender

  • Pingback: Ask Bryan Eisenberg + Free Website Reviews()

  • Steven LaLonde

    Bryan, thanks for a sweet read! Kudos to you for the book and Google for the free product (GWO) which will both help strengthen the culture of ‘always test, test, test’ within organizations!

  • Steven,

    Thank you for the kind words. Amazon reviews are always welcome 😉

    Best wishes,

    Bryan Eisenberg’s last blog post..99 Bottles of Beer On the Wall

  • Hey Andy!

    I just came across this interview, but it’s got me thinking that I DEFINITELY need to pick up a copy of this book.

    Thanks for doing the interview and for posting it here.


    Jonathan Kraft