User Experience

The case for AB testing

posted Aug 11, 2011 1:31 PM by Ricardo Escalon

AB Testing is not as straight forward to implement in large corporates as it may be for small garage websites. In large corporates you often have to convince senior level management about the need to invest 2 weeks of development and testing just to get a testing framework. The arguments for having AB testing are many, but managers often come back and say to you that you can achieve the same thing by just running a trial over a period of time. This article debunks that myth.

Why can't you just run atrial

In general you can't just run a trial because, like statisticians like to say, there are too many independent variables. By independent variables I mean that on each release there are around 20 things that may impact the success metric that you are interested in, how can you be sure the one change that you are interested is the one that has caused the metric to go up or down? The main goal of AB testing is to isolate your independent variables so that you can understand the impact they have on conversion rates.

What are the benefits of AB testing?

Well the main benefit is knowing whether a change has been a good one or a bad one. 50% of all uneducated decissions are wrong. AB testing helps you pick out the wrong ones as early as possible. The not so obvious benefit is that you can ask your coders to keep on coding past regressionn and progression testing as you can put code in the test group (B) and turn it off if it doesn't test well, or turn it on late as soon as you know it does test well.

When selling to senior level management you want to talk about dollars. There are two basic calculations that can help you justify the need for AB... to be continued I'm falling asleep...

Selling IA and Navigation Changes: Less Common Sense and More Dollar Sense

posted Jun 16, 2011 2:44 PM by Ricardo Escalon

The problem with us UXers is that we're always intellectualising our field. We use words like IAs, navigation frameworks, and "enhance the user experience". These mean very little to executives who get judged on how much money they can bring in or save. To get our work approved sometimes we just have to talk their language. After all we don't get much face time with them so we want to make the most of it.

How did you go convincing that stubborn executive of the need to change the IA? I bet the story went something like this.

Bobby is the UX gure and John is the executive.

Bobby says:

"Hey John, I wanted to talk to you about the need to redesign our IA, create a consistent navigation framework."

John replies:

"What is the problem with our structure, look Bobby what we need is to acquire more audience, we need a browsing experience where people can browse sequentially through our content like an cool iPad app"

Bobby tries to reign it back to core issue by saying:

"Interesting idea John, but I think that there are obvious problems with our current IA that could significantly enhance the experience."

John looks at Bobby with a puzzled look and says:

"What exactly is the problem."

Bobby tries to think of a quick example and says:

"Well, no one is coming back to edit their content???"

Trying to be polite John replies:

"Bobby, your new at our business and I appreciate your input but we've known for years that Australians don't generate content and I don't think changing the IA will change the way Australians behave online. Listen, I've got another meeting with senior management. Let' talk another time."

That's it your two minutes with the senior executive gone, you couldn't convince him that you needed to redesign the IA and the navigation the first steps in creating a good user experience.

What if instead the conversation went something like this:

Bobby starts with:

"Hey John, what if I told you I could make a change to the site that is worth a million dollars in equivalent search engine marketing (SEM) spend?"

John raises one eyebrow and says:

"Yeah I'd be want a hear about it."

Bobby throws at him his elevator pitch:

"All we need to do is improve the conversion rate (or the ratio of payment transactions to visits) by 0.5% and we can do this. I reckon we just need to try a few tweaks to the nav and do a special type of testing that allows us to isolate these tweaks from the 20 other changes we do every release so we can be certain that the 0.5% improvement is thanks to our tweaks. The plan is to try different navs and incrase conversions."

John says:

"What are you waiting for, let's make it happen!"

If you missed the point, it's that we really have to have our story straight, we can't use our own speak to communicate with executives. We have to latch onto something they care about and bring about change through that.

1-2 of 2

User Experience

The case for AB testing

Selling IA and Navigation Changes: Less Common Sense and More Dollar Sense

Have you successfully sold IA and navigation changes to your organisation? How did you go about it?