Imagine a world where we’d know the impact that an experience would have before we shipped it.
This is not as impossible as it sounds, and most product companies are or have been building experiences this way for a few years now. Popularised by ecommerce sites and lean startups, validating design decisions by A/B testing and hard data is commonplace nowadays. But trying to apply some of these techniques to enterprise software can be much harder. There is a great article on UX mag debunking the 6 myths of data driven design. It’s an awesome read, but I would go a step further, and say that we need to move away from data-driven design and into a world of data-informed design.
Data-driven vs. data-informed
Data-driven design looks to ship fast, optimise at every step and let the data drive many of the design decisions. Often (but not always) it is possible to get large percentage improvements with small tweaks as pages have similar and standardised layouts, or a startup only has a few features to optimise for and one specific type of customer to speak to.
However, this way of working can be much harder to execute when you work in a complex product environment, with many features, experiences and types of customers to optimise for. Continuously shipping experiments may not be the right way to work when millions of customers rely on your products to get their work done, as is the case at Atlassian. But how do you keep trying to optimise and learn, whilst retaining a well thought-through design approach? I believe there’s a slightly different approach which I’m terming “data-informed design”. Here’s an internal Atlassian example of being data-informed, instead of data-driven.
We shipped an experiment to improve our onboarding experience for new customers. From a data perspective the result was a huge failure. It resulted in -12% engagement (time spent in product) and neutral conversion. So did we throw it away and move to something else? No, the team believed in the experiment we had designed and our qualitative research was telling us that we were on the right track. Our conviction turned out to be correct. We took some lessons from the data, applied some insights from our qualitative research, iterated on the experiment, and with tweaks to the design it became a success at +22% conversion, with engagement staying neutral.
This is being data-informed, not data-driven. We used the data we had and combined it with qualitative feedback and our design intuition to produce an iteration that was ultimately successful. We weren’t binary, we didn’t throw it away completely at the first sign of trouble.
Here’s another example of being data-informed from Airbnb. I love this article discussing their experimentation and in particular this quote about a major redesign of the Airbnb search results page:
A lot of work went into the project, and we all thought it was clearly better; our users agreed in qualitative user studies. Despite this, we wanted to evaluate the new design quantitatively with an experiment.
Emoticon Emoticon