Field Notes Inside an Integrated Communications Agency

'A/B Testing' is the practice of directing web traffic to multiple alternative designs to determine which is optimal. This method raises significant questions regarding the role of a designer and the need for a traditional design approach when deciding which design is 'best.' Are we being cut out of the equation?

Panelists: Corey Chandler (Lead Interaction Des, eBay Inc), Jake Cressman (Producer, Electronic Arts), Chris Maliwat (Sr Dir of Prod Mgmt, Vuze Inc), Micah Alpern (Design Dir Social Se, Yahoo! Inc), Elliot Shmukler (Principal Prod Mgr, LinkedIn)

They showed examples and had us vote on the ones we thought would test better (we usually were not right and the better designed one didn't generally win.) You have to test to see what is good. Testing gives you a measureable level of rightness and makes it easier for you to decide. Allows you to be precise.

 

How Netflix designs a Winning Web site: Their designers say they don't have any predictions, because that would color what was created. Designers become better at designing for audiences with experience. Netflix is known for being innovative with their design testing. They say they don't make choices, but they do. They had more designs than what they presented to audiences. They had to make choices to get somewhere. They did have a point of view.

Since A/B testing exposes your design to nondesigners, designing is hard once you have introduced A/B testing. Anyone can have influence before it goes to audiences. You have everyone introducing ideas that can be equally tested. A project manager may request 20 different locations for an ad. Does it make sense to have that many variables? "Its just a test, lets see if we get results" sometimes the absurd ones were effective, but they had no reason why.

A/b testing helps you optimize, but it does not help you revolutionize or come up with BIG changes. The tasks are microtasks.

There is a big difference between an expert and a novice using testing as a north star. Do you have a model that explains why something is true?

A/B testing doesn't necessarily show you impact with numbers. You have to make a pretty big change to be statisticly significant in your results.Change one thing at a time. Make one change, see if it increases sales and then make another change. As prototypes tools become better, we can get better at sharing with a group of people to test.

Linked In and Yahoo have done testing with people in the millions. That is a lot of people to have a bad experience. Original A/B testing - ship captain that thought citrus prevented scurvy, so he did a user group and gave half his shipmates citrus, half did not. He got his results, but yikes to the people that did not get the citrus!

Sometimes a learning period is difficult - we put things out there for a little bit and then we change it and then as time passes you see the metrics shift or improve, then you change it and there is a collective groan from people. How much time should you have something up? You put something new on a site and people poke on it because it is new.

Isolated ecommerce sales tracks. Removing the navigation that is not necessary. Removing distraction is good, but you need to be able to get back to where you were effortlessly. Don't do it in the shopping cart because that is still part of decision making.