I want to use the power of metrics to drive our organization toward a strategic goal of providing a fabulous user experience. Toward that end, I'm drafting an approach that I'll lay out here, including the implications for both management and for product/development teams.
Step One--Incent management
I've worked with senior management in my group to include user experience (UX) metrics in the annual goals of managers in each product line. They, in turn, will put these metrics into the goals of their product owners. Here's the language for those goals:
Prior to going live, [product/functionality] is ranked by users on the Standard Usability Scale (SUS). Threshold = 70/100. Target = 90/100.
On the live web, the User Experience Indicator for [Audience_segment] will be xx or higher by [date].
To be clear, salaries and bonuses are riding (in part) on these goals. If I'm a manager or product owner, I get more money if I provide a great, usable, experience for my end users.
The remainder of this document introduces these goals in more detail.
Standard Usability Scale
Language for Objectives
Prior to going live, [product/functionality] is ranked by users on the Standard
Usability Scale (SUS). Threshold = 70/100. Target = 90/100.
Background
The Standard Usability Scale (SUS) is a widely used instrument for measuring usability. The SUS focuses on just one aspect of the user experience: usability. It asks users the degree to which they agree or disagree with each of 10 questions:
- I found the web site unnecessarily complex.
- I thought the web site was easy to use
- I think that I would need the support of a technical person to be able to use this web site.
- I found the various functions in this web site were well integrated.
- I thought there was too much inconsistency in this web site.
- I would imagine that most people would learn to use this web site very quickly.
- I found the web site very cumbersome to use.
- I felt very confident using the web site.
- I need to learn a lot about this web site before I could effectively use it.
- I found this site was easy to use.
This score will normally require testing with 5-12 users. Our User Experience Team can help a product team determine the optimal number of users. We also plan to build the SUS into virtually all usability tests, and we plan to make usability tests easily accessible to product teams. We want it to be super-easy for a team to know the current SUS score of its product.
Appropriate Use
Use SUS in 2 places:
- on an increment of functionality, prior to going live.
- on an entire web experience that is already live, as part of the User Experience Indicator for a given audience segment (see “User Experience Indicator” below)
SUS should be administered under controlled circumstances, typically as part of a usability test
The User Experience Team also recommends using SUS as exit criteria for a sprint or release; essentially, an SUS score of 90 is part of the definition of “done.”
User Experience Indicator
Since the SUS measures only usability, we need at least one more metric to measure overall user experience. We have not finished defining a standard metric for this, and we do not have sufficient baseline data, so our 2008 objectives will be to create baselines that will enable us to set hard targets in 2009 and beyond.
Language for Objectives
2008: For [audience_segment], establish a baseline measurement of the User Experience Indicator.
2009 and beyond: The User Experience Indicator for [Audience_segment] will be xx or higher by [date].
The Metric Being Created
The User Experience Indicator will most likely be a combination of 3-5 questions, most likely addressing the following dimensions:
- Would you recommend this web site to a friend?
- How satisfied were you with this site?
- Compared with other web sites, how well did this site meet your expectations?
- System Usability Scale
A key piece of our organization's strategy is to provide an unexpectedly enjoyable experience for our users—we want them to say, “wow!” Our assertion is that by combining answers to these questions, we will be able to elicit a reliable measure of the overall user experience, including the “wow factor.” If we have hit the mark with the wow factor, responses to all four of these dimensions will be very favorable.
Appropriate Use
These metrics are best associated with the overall experience of a segment of users (e.g., member, brokers, etc.), with a site already live, rather than for an individual feature or a site in development.
The User Experience Team will create a simple formula for combining these 4 measurements into a single User Experience Indicator and will make it easy for product teams to produce a User Experience Indicator for their product. We hope to have this methodology ready in Q2 2008.
Background & Rationale
The User Experience Indicator is a very high-level metric. It measures our success at providing a great user experience, but it is not intended to tell us why a user’s experience was good or bad. We have a whole collection of tools available to dig deeper into the “why” questions. This metric gives us one simple indicator of how an audience’s experience is improving over time and in relation to other audience segments. Here’s the significance of each dimension of the Indicator:
Would you recommend this web site to a friend?
This is based on the work of Frederick F. Reichheld, as originally published in the Harvard Business Review piece entitled The One Number You Need to Grow. It has since been adopted by a wide range of industries and is gathering steam as a standard indicator of business success. This is the basis of the “Net Promoter” discipline. Here’s a brief synopsis of the original article:
Companies spend lots of time and money on complex tools to assess customer
satisfaction. But they're measuring the wrong thing. The best predictor of
top-line growth can usually be captured in a single survey question: Would you
recommend this company to a friend? This finding is based on two years of
research in which a variety of survey questions were tested by linking the
responses with actual customer behavior--purchasing patterns and referrals--and
ultimately with company growth. Surprisingly, the most effective question wasn't
about customer satisfaction or even loyalty per se. In most of the industries
studied, the percentage of customers enthusiastic enough about a company to
refer it to a friend or colleague directly correlated with growth rates among
competitors. Willingness to talk up a company or product to friends, family, and
colleagues is one of the best indicators of loyalty because of the customer's
sacrifice in making the recommendation. When customers act as references, they
do more than indicate they've received good economic value from a company; they
put their own reputations on the line. The findings point to a new, simpler
approach to customer research, one directly linked to a company's results.
We hypothesize that a user with a “wow” experience is more likely to say they would recommend the site to a friend.
How satisfied were you with this site?
While the “would you recommend” question has many proponents, there are also those who argue it does not adequately address overall satisfaction. E.g., maybe I would recommend this site because it’s the only place in the world I can buy a particular product, even though my experience of the web site itself is horrible. Or maybe I love the site, but I wouldn't recommend it to a friend because it's not relevant to any of my friends (or colleagues). Because of these limitations, we add a basic satisfaction question.
A key limitation to this kind of generic satisfaction question is that it doesn't help us understand why they're satisfied or dissatisfied, so I've heard concerns that a satisfaction score isn't useful. However, in this instance, we're using the score as part of a measure of success, rather than as formative research. If my satisfaction scores are not high enough, I am then incented to figure out why my users are dissatisfied and what I can do about it. We have lots of other tools available to help with those tasks.
The User Experience Indicator will be particularly sensitive to the extremes on this scale. Since our goal is to produce an unexpectedly enjoyable experience, we don’t want people to be only somewhat satisfied, we want them to be thrilled (wow!). So we will initially aim for a threshold of “satisfied or extremely satisfied,” but we will quickly move to a target of changing users from “satisfied” to “extremely satisfied.” For example, in a November 2007 survey of registered members of kp.org, 86% said that they were either satisfied or very satisfied with kp.org. This looks very good and would be a very difficult number to improve. But a closer look at the data shows that this 86% is a combination of 49% satisfied and 37% very satisfied. A meaningful User Experience Indicator would incent teams to increase the percentage reporting “very satisfied.”
Compared with other web sites, how well did this site meet your expectations?
This dimension adds two elements not covered by the previous two:
- Comparison to other web sites (across industries). These are a critical to the context in which users access our web sites. It’s important to go across multiple industries, because their perceptions are ultimately based on their experiences with their favorite sites (shopping, banking, blogging, etc.), rather than only with our direct competitors.
- Experience in relation to expectations. Our strategy is to provide an “unexpectedly enjoyable” experience, so we want to find out how well we did relative to what they expected. As users’ expectations increase, we will need to continue innovating to stay ahead of their expectations. This dimension helps us understand the “unexpected” part of the “wow” factor.
System Usability Scale (SUS)
See above for an introduction to this metric. When included in the User Experience Indicator, the SUS score applies to an entire web presence for a given audience, rather than to an increment of functionality. It is measured in a production environment.
The User Experience Team intends to provide a framework that makes it easy to regularly measure the SUS for each major audience segment.
Next Steps
To use metrics effectively, and organization needs to do 4 things:
- Define metrics that measure what's important
- Make it easy to measure these metrics
- Make these metrics widely visible
- Formally incent staff to meet targets for these metrics
This post is an initial crack at #1 and #4. The User Experience Team will create the User Experience Indicator methodology. As we all collect and compare the resulting data, we will analyze it for validity and will refine the methodology over time.
How does this sound to you? Please comment.
