Empathy, Humility, and Moxie

It starts with empathy.

Every journey to a great user experience starts when someone somewhere empathizes with a user in a way that is authentic, human, and compelling. It starts when someone realizes that the most important person to pay attention to is not the person paying for the project or implementing the software, though these folks are certainly important. It starts when someone really, truly, deeply pays attention to the person who will end up using what's being created. User experience starts when someone starts caring about the user.

Next comes humility.

Anyone who has watched usability testing or ethnography or a focus group knows the feeling. I remember my first usability test - I had spent weeks with a talented team designing a great user interface. All we needed to do was validate the design with some end users and possibly make some tweaks. An evening of usability testing provided a huge heap of humility. Labels that were intuitive to me were enigmas to the users. Buttons that were totally obvious to me were invisible to users. Most of the people I showed it to couldn't even figure out what it was supposed to be.

Sometimes we argue with the first few people who have trouble with a design:

"This first test participant is an outlier."
"The second test participant isn't a good representative of our customer base."
"The third test participant must just be stupid."

But by the time we get to the 4th user who can't find their way through our design, we start to realize, maybe the problem isn't with the users. Do this enough times with enough designs and we learn a very healthy sense of humility. Two quotations that help keep me humble:

Several years ago we were testing a new home page design. We brought in several users and gave them what we thought were a handful of simple tasks to complete. By the time we got to the third user I was already starting to hear the humility music playing, and it came to a crescendo when the moderator asked, "so where would you click?" and the user responded, "I'd click over here on 'careers' so a could get a job with these bozos and fix their damn web site!"

And I remember Haim Hirsch, my first user experience mentor, who had conducted hundreds or thousands of usability tests, tell me, "I've never conducted a test where I wasn't surprised by something." So that's my goal - to be humble enough to assume that my design has major problems, and to be grateful to the users who will help me find those flaws.

Empathy, humility, and...

...moxie.

From the dictionary:

moxie
–noun Slang.

1. vigor; verve; pep.

2. courage and aggressiveness; nerve.

3. skill; know-how.

Empathy is about caring. Humlity is about knowing that I don't know everything. And moxie is about knowing that I do know some very important things, and in some cases I know more than my users do. Moxie means that I sometimes contradict the users. It means I have the expertise and the larger vision to create something they hadn't thought of or to do something in a way they would not suggest. We need to conjure up some moxie, because users are great at identifying problems, but not necessarily always great at identifying solutions. User needs should drive all key decisions; user design suggestions should get added to the list of possibilities.

Moxie in the absence of empathy and humility is a recipe for disaster. But in combination with them, moxie enables us to break paradigms, innovate, and create solutions that meet our users' needs and desires better than anything they could have designed themselves.

Empathy - I care
Humility - I don't know everything
Moxie - I know something

User Research as a Commodity (part 3 of 7)

The Problem
I’d like to describe an approach we’ve using at Kaiser to make it easier for development teams to incorporate user insights into their work. Just about every development team on the planet could benefit from more user research—usability testing, card sorts, label tests, brand reactions, cognitive interviews, etc. The more exposure teams get to their end users, the more user-centered their work will be, and the better the user experience (UX).

In my experience, there are two main reasons why teams don’t do more user research:

1. They don’t understand its importance
2. It’s too hard. You have to schedule it into the project plan, find a place to do it, prepare stimuli, recruit participants, deal with incentives, figure out what to test, and then spend time actually testing.

There’s a cool relationship between these two barriers to research: If we can make it super-easy for teams to do the research, they’re more likely to actually do it, and once a team does a little user research, they usually understand its importance and want more. The key is to prime this cycle.

So how do we get this cycle going? How do we make it incredibly easy? How do we change user research from a hassle that interrupts the work to a commodity that can be easily acquired on-demand?

The Challenge
Last year we commissioned an agile-like team to build a new web site for brokers--the professionals who help employers select and purchase health plans for their employees. The team needed to produce a beta in two months and a fully operational site in four months, and they needed to do it on an entirely new technology platform.

The pressure was on. At that time, our typical waterfall timeline for even simple projects was over a year. We typically did a week or two of usability testing in conjunction with the requirements phase, before development, or even technical design, started.

We had already done some ethnography with brokers, and the Product Manager was totally convinced of the need to incorporate usability testing, etc. into the design & development work. But many others, including developers, the project manager, and sponsors, thought of user research as a nice-to-have that was likely to blow the tight schedule.

Everybody was excited about moving to agile, but we didn’t have a clue how we were going to fit two weeks of usability testing into the work. Should we do it mid-way through when we’d have good comps to show users? But we couldn’t afford that kind of a break in the project plan. Or maybe front-load the testing, because the developers needed to take several weeks early on to get their environments in order? But we didn’t yet have any idea what the new technology would allow in terms of the UI. Complicating matters, our usability specialists were in Pasadena, CA, while the agile-like team was in Pleasanton, CA—350 miles away. Not only that, but brokers are difficult participants to recruit—they work in offices all over the country and they have very busy schedules, so the team couldn’t just take a paper prototype down to the nearest Starbucks and ask them what they think (as we’ve done with some other audiences).

When asked, the sponsors and most of the team thought the best thing to do would be to crank out an initial site on a tight timeline, doing the best we could based on intuition and a few heuristics, and then test it with users after we went live. We were going agile, they figured, so we can change it easily after we’re live. After all—it’s only UI. ;-)

Clearly, the UX people didn’t understand agile, and the software people didn’t understand UX.

The Solution: Testing as a commodity
The Product Manager and I were convinced that we needed to expose our work to end users prior to going live, and we could see there was no way we’d get even a week out of the schedule. So we decided to try something new—prescheduled testing. Here’s how it worked:

Every other Thursday morning, four or five brokers (our target audience) would show up at our offices in Pasadena. Our user research specialist would work with each participant for about an hour. While this testing took place in Pasadena, we piped audio and video of the testing up north to Pleasanton so the team could watch in real-time and IM questions and comments to the moderator in Pasadena.

Since we knew the testing schedule several weeks, and even months, in advance, we were able to easily schedule a room for the testing. One of our admin staff, who’s particularly good on the phone, took on recruitment. With multiple dates prescheduled, recruitment was easier—“OK, you can’t make it next Thursday; how about two weeks later? How about the Thursday of the following month?” We also had an admin person manage all the incentive checks, greet the participants in the lobby, and help with set-up and tear-down, all of which was essentially the same each week.

In the early stages testing consisted mostly of card sorts and cognitive interviews. As the project progressed, we moved to various stages of UI, focusing on whatever the team had just built or was about to build—one week working on a page layout; the next week focusing on a particular widget.

Results
The key breakthrough for us was that we made it super-easy for the agile team to get the benefit of user research. They didn’t have to stop what they were doing, they didn’t have to deal with logistics, and the only planning they needed to do was to be sure they had something to show users by Thursday morning, along with some good questions to ask.

It worked great. The team made constant course corrections based on the research. We would typically end sessions by asking the participants, “on a scale of 1-5, how easy was this site to use? (5 is easiest)” In two months of iteration we moved this from a 2.5 to a 4.5. The team could come up with ideas and never have to wait more than two weeks to test the ideas with users.
Sponsors could tune in to view the testing whenever they wanted, and they were delighted to see their customers, the brokers, delighted. The personas became real people. We virtually eliminated arguments within the team about what would work best for users. Instead of pressing the point, the Product Manager could just say, “I’ll ask them this Thursday.”

Why it worked
I can’t stress enough the importance of two key elements:

1. A regularly scheduled research time, scheduled up to months in advance.
2. Logistical support from outside of the project team

Scaling

So the pilot was successful. Everybody loves the initial site. Everybody wants to go agile. That means we’ll soon be looking at up to 20 agile teams operating simultaneously, working on multiple sites that support several different audience segments (brokers, Kaiser members, employers, etc.). Can we scale this approach to work in that environment? Here are my thoughts so far…

Brokers are now actively involved in a beta program, and we still have every-other-Thursday prescheduled research available to them as a tool. As we start up an agile team for employer groups, I think we can use a very similar model. The challenge will be with Kaiser members, because we’re likely to have many agile team running simultaneously, focusing on different content and functionality, and attending to different subsegments of the member audience. It sound pretty unwieldy to give each team its own half-day slot every other week.

What if we instead treat the user research environment and participants as a commodity available to all the teams? Every Monday and Wednesday we bring in members all day long. Every Tuesday we bring in employers. Every Thursday we bring in brokers. We post a schedule, and individual agile teams can sign up for time.

Not every agile team will have an hour’s worth of user research tasks for participants each week, but the commodity approach helps out again: We can bring in 8 participants in one day and spend an hour with each. During that hour, we may do a simple 5 minute “find this content” task for Team A, a longer “complete this transaction” task for Team B, and a broad “see if we just broke the UI” task for Team C.

What if teams don’t generate enough tasks to fill up the time this week? No problem--our “Platform UI Team” will maintain a backlog of non-urgent areas to test with each audience, so they can fill in as needed. What if Team D has some questions, but next Monday’s schedule is already filled by other teams? No problem--we’ve got open time on Wednesday’s schedule.
With the recruitment and logistics down to a routine managed by admin staff, and the bulk of the testing budget managed as a shared service (and thus “free” to the teams), this leaves the user research specialists and Product Owners free to formulate good tasks and questions and to apply the results of the research.

Back to metrics
Here’s one more possible extension—we haven’t tried it yet, but we’re toying around with the idea.

Kaiser Permanente as a whole is struggling to become a more metrics-driven organization. Our physicians are internationally recognized for how well they practice evidence-based care, but our business practices haven’t yet caught up. Over the next months and years, our performance will be increasingly judged on metrics, and our agile teams will be measured and incented based on metrics like:

• Health outcomes
• Sales
• Operational efficiencies
• Time-to-market
• Cost
• Backlog burndown

Along with these “bottom line” metrics, we’ll also pay a lot of attention to leading indicators—the metrics that show early on whether we’re moving toward improvements in the bottom line metrics. As we become a more metrics-driven organization, how can we ensure that people pay attention to user experience?

Those who already “get it” will know that the best way to achieve great business results is to provide a fabulous user experience designed around the needs and perspectives of the end-users. But for those who don’t yet get it, time-to-market, cost efficiencies, and even sales can seem to be more important than user experience. How can we use the metrics system to incent people to both perform good user research and to use that research to improve the user experience?

SUS Tracking as a Commodity
A while back we had some significant availability issues with one of our sites—too many unplanned outages. A one-page dashboard of metrics was arguably the most powerful influence for fixing the problems. Suddenly, people at every level of the organization could easily see, on a weekly basis, how many times it experienced slowness, how many minutes it was down for planned outages and how many minutes it was down for unplanned outages. The numbers were bad. The numbers were visible. The numbers got executives and team members to come together to make those numbers improve dramatically.

What gets measured gets managed.

What if we published a similar metric for user experience? What if teams were incented to “bring your user experience metric up a point?” What if teams could see that their recent release moved their score up (or down), and what if all of our peers could see our user experience scores? How could we do that without wasting a whole bunch of time and getting in everybody’s way?

What if each site (or product) had a regularly scheduled UX Checkup? I’m thinking maybe every 2-3 months. This would be part of the user research schedule. The user research specialists could work with Product Owners to create a script . Then every couple of months, without the product team needing to lift a finger, we would take participants to the site in the production environment and run through the script.

• Finally fixed that annoying bug? Score goes up.
• Haven’t yet implemented the feature the users most want? Score goes stays the same.
• Rushed to get a new feature in on time and screwed up the IA in the process? Score goes down.

My incentive pay in 2007 was based in part on an objective of brokers scoring the site at least ‘4’ on a 1-5 usability scale. Could we possible scale this so that everyone is incented to provide a measurably excellent user experience? There are two fairly obvious candidates for this metric:

• The System Usability Scale (SUS)
• A generic satisfaction question

SUS deals with usability quite nicely, but leaves out other critical aspects of the user experience, whereas satisfaction questions are notoriously unfocused.

The two might work well In combination, with the SUS score assessed in-person and the satisfaction question routinely asked via survey.

Inspirational Conclusion

Creating a fabulous user experience is all about a user-centered culture with access to the tools of user-centered design, combined with the ability to deliver software and operations support. The biggest lever we have for creating a user-centered culture is exposing everyone involved to their end-users in valuable ways. If we can do that, the users will thrive, and so will the business.

One way to promote exposure to end-users is to remove the main barriers to basic user research by offering pre-scheduled user research that is easy, scalable, and measurable.

Please comment!