Data-Driven Design In The Real World

Posted on Oct 22, 2015 by Administrator

As more designers and writers look to analytics to inform their decisions, many still struggle to implement their findings in a sustainable, ongoing way. Too often, testing and analysis are one-off activities, providing plenty of important-looking numbers but not lot of context or specific direction.

After more than five years helping content and design teams capture, measure and understand website performance data (client-side at Bazaarvoice and now at Volusion), I’ve learned a lot about connecting the dots between data and design improvements. Today, I want to share some of those lessons with you.

In this article, we’ll:

• see what a good data-driven model looks like on paper,
• look at a real-life example of the model,
• share some resources to help you get started with testing.

Defining “Data-Driven” Design

Before we can talk about using data to improve design, we have to define what we mean by “data.” This will help with a very common challenge: no shared language between designers and writers and their analytics team and tools.

QUALITATIVE VS. QUANTITATIVE DATA

In most Web-based projects, there are two general types of data, and you’ll often see these discussed in articles about website optimization:

• Quantitative data
  Numerical data that shows the who, what, when and where.
• Qualitative data
  Non-numerical data that demonstrates the why or how.

Most analytics tools, such as Google Analytics, provide a lot of quantitative data, such as who has come to your website, how they got there and what actions they took.

What these tools don’t tell you is why. Why does a certain group of visitors take one action, while a different group of visitors choose another? Why does one piece of content keep visitors on your website longer than another? That’s when we turn to qualitative data. Whereas quantitative data shows scale, qualitative data gives perspective. It helps us understand not just what happened, but why and how it happened.

“The qualitative/quantitative issue is really a misunderstood area in research, especially to people who haven’t been exposed to broad-based training,” says Dave Yeats, senior UX researcher at Bazaarvoice. In his over 10 years of conducting user research, Yeats has developed a greater appreciation for the qualitative side of things:

“I’ve come across too many instances of people dismissing qualitative research as ‘anecdotal’ because they don’t understand how non-numerical data is still data.”

A good data-driven model must include both. And a good working understanding of how the two relate to each other will not only provide better insights, but also improve communication between team members.

The Key To Success: Be Empirical, Be Specific

The very best data, be it qualitative (i.e. non-numerical) or quantitative (i.e. numerical), is always empirical. Empirical data is any type of information gathered through observation or experimentation. The best empirical data answers specific questions — because when data is specific, taking action on it becomes easier.

When looking for general empirical data, such as “metrics for the website” or “how the website is performing,” you can end up with data that, while interesting, doesn’t lead directly to specific actions. Or, as Google Analytics evangelist Avinash Kaushik colorfully puts it:

“All data in aggregate is crap.”

You can’t isolate variables when looking across big aggregated metrics (such as overall page views or downloads.) This makes it difficult to hypothesize about why you’re seeing what you’re seeing. There are just too many moving parts to know.

Plus, different portions of a website — indeed, even different pages within subdirectories — have different, smaller goals. Sure, they all feed into large site-wide goals, such as sales or downloads or content consumption, but optimization must happen within smaller visitor groups, traffic segments or groups of pages. Let’s look at an example.

A Textbook Example Of Good Data-Driven Design

To better understand how to focus on empirical data, both qualitative and quantitative, let’s look at a hypothetical problem for a content-oriented website.

Let’s suppose you run an online periodical or research website. Keeping visitors engaged is a big goal! You’ve been asked to make design and content changes that will help retain visitors. Where do you start?

You could log into your analytics account and check exit and bounce rates. For our purposes, we can define these as follows:

• Exit rate
  The number of times a visitor leaves your domain from a page, divided by that page’s total views. Generally expressed as a percentage.
• Bounce rate
  The number of times a visitor enters a domain on a page but leaves before viewing any other page in the domain, divided by the total number of views of that page. Also generally expressed as a percentage.

Upon sorting all of your pages by exit rate and then bounce rate, you find that two pages have much higher rates than the website’s averages. Based on this quantitative data, you look up the pages. One page contains a prominent link to a sister website — this means you’re intentionally sending people someplace else. You’re not as concerned, then, by the high exit and bounce rates on that page, because that page is designed to be an exit point. But the other page contains a long, important article and no direct, intentional reason to leave.

Why are visitors bouncing and exiting so often, then? Time to turn to qualitative data! “My favorite thing to do is combine observational research (watching somebody use a site) with in-context, self-reported data asking people about their presence,” says Yeats:

“‘What are you thinking about right now?’ ‘What is your response to that?’ This combination of observation, stimulus and probing for data paints the full picture.”

Yeats is right! This is a great opportunity for user testing. And because you’ve narrowed down your efforts to a single page (maybe a couple for additional context), testing becomes more practical. You’ll also be able to determine whether any design changes you make are working, because you’ve identified specific, empirical metrics that quantify success: exit and bounce rates.

Setting Your Data Up For Success

As you look to improve your use of data in making decisions about design and content, do the following:

• Develop a common language with your analytics team or, if you’re also pulling the data, then with your analytics tool. Educate your team on specifically what you mean so that they understand the importance of the metrics you’re providing.
• Use quantitative and qualitative data together — even if people are skeptical at first of the qualitative points.
• Always use specific, empirical data — don’t offer “high-level” metrics. Find data points that answer specific design questions and, thus, illustrate whether design or content changes “worked.” (In our example, we used bounce and exit rates.)
• Remember that success does not mean the same thing for all pages or types of visitors. Consider how returning visitors might have different needs from new visitors, or how visitors from an email might have different needs from visitors from organic search. Think about the goals of individual pages and subdirectories and how they might differ.

So far, we’ve talked about the best-case scenario for data-driven design: using quantitative data to identify issues and to benchmark current performance, and then using real-time qualitative user testing to understand why you’re seeing those numbers and how to improve them.

Continue reading on SmashingMag

---