Week Thirteen - Usability Test Planning and Usability Testing

Required Preparation

Please review this module before class for optimal engagement and understanding.

Learning Objectives

• Understand the evolution and importance of usability testing in product design.
• Learn various methodologies and techniques used in usability testing.
• Recognize the benefits of usability testing in different stages of product development.
• Apply the 3-step rule for effective usability testing throughout the development process.
• Understand the importance of participant diversity in usability testing.
• Differentiate between various types of usability testing and their applications.
• Master the creation and use of essential testing documentation, including test plans, scripts, and NDAs.

Slides

MAD9034 13-1 - Usability Testing

MAD9034 13-2 How to run a test

Usability Testing

Usability testing is the practice of testing how easy a design is to use on a group of representative users. It involves observing users as they attempt to complete tasks and can be conducted repeatedly, from early development until a product’s release.

User Testing: Why & How (Jakob Nielsen)

What Happens in a Usability Test?

In a typical usability test, a test moderator gives test participants a series of tasks that they must perform with the prototype or product. The tasks represent actions that an end user would typically carry out with the finished product. During the test, the moderator observes each participant’s steps, routes, and reactions, often also recording the test session on video.

Usability testing can take various forms, but at its core, it involves observing users as they interact with a product. A common methodology is the think-aloud protocol, where participants verbalize their thoughts, feelings, and frustrations while using a product. This method provides valuable insights into the user’s thought process and helps identify areas where users may encounter difficulties. Another advanced technique is eye-tracking, which tracks where and how long a user looks at different areas of a screen. This can be particularly revealing in understanding user attention and engagement. These methods combined offer a comprehensive view of how users interact with and respond to a product, highlighting areas for improvement and ensuring the final product meets user needs and expectations.

Why Conduct Usability Testing?

The benefits of usability testing are manifold. It not only helps in identifying usability issues before a product is fully developed but also enhances user satisfaction by ensuring that the product meets their needs and expectations. By involving users in the design process, usability testing can significantly reduce the time and cost associated with reworking features after a product’s release.

Summary:

• Cheaper to fix problems found prior to coding.
• Learn if participants are able to complete the specified tasks successfully.
• Learn how long it takes to complete the specified tasks.
• Discover qualitative information (how happy are they with your service).
• Identify required changes to enhance usability.

3-Step Rule for Usability Testing

Step 1 (early stage):

• Use any general user to help encounter basic usability issues (navigation, etc).

Step 2 (mid project):

• Use users with a broad spectrum of ability (beginner, advanced).

Step 3 (later stage):

• Use target users to refine any specific issues (persona).

• Expanded Content: This rule provides a structured approach to usability testing throughout the development process. In the early stages, testing with general users can uncover basic usability issues like navigation problems. As the project progresses, involving users with a broader range of abilities, from beginners to advanced users, helps in identifying more nuanced usability challenges.

Participants

Recruit 5 users for a usability session. Typically, 5 users will surface 80% of the problems. Beyond 5 users, you will start to see the same patterns. The best practice is to test with 5 users, incorporate their feedback and your observations in the prototype and then test with another 5 users.

Usability Testing w. 5 Users: Design Process

The choice of participants is crucial in usability testing. While the rule of five participants is a good starting point, it’s important to ensure diversity among the test users. This includes a mix of genders, ages, technical proficiencies, and other demographic factors relevant to the product.

Types of Usability Testing

Hallway Testing

Who? Anyone (Minus people working on project) What? Asking randomly selected people to test a feature of a product Why? Help you discover “brick walls”, issues that don’t allow users to continue

Lab Testing

Who? Target User What? Real-time communication between user and evaluator conducting tests in a controlled environment Why? Formal setting with proper equipment to record, observe etc

Remote Testing

Who? Target User What? Testing remote users via conferencing software or asynchronous methods: click streams, user logs etc Why? Cheaper, you can test with anyone around the world, can simulate real life context

Guerilla Testing

Similar to hallway testing, but even rougher & quicker. Usually < 15 mins. You approach a person, ask them if they would like to answer a few questions about your product, give them a couple of tasks to do, observe their interaction, ask about their experience, and you’re done.

Each type of usability testing serves a specific purpose. Hallway testing, for instance, is quick and informal, ideal for early-stage feedback. Lab testing, on the other hand, is more controlled and provides in-depth insights.

Testing Documentation

Test Plan

A test plan is a document which captures:

• Who will be your participants? (profiles)
• What are you going to test? (scenarios)
• Where will the test be held? (remote/lab/etc)
• When will the test be held? (date and time)
• Why are you testing those scenarios? (test metrics)
• How are you going to conduct the test? (method)

Why Create a Test Plan?

Three main reasons:

1. To help yourself get organized.
2. To communicate the plan to stakeholders, team, and management for feedback or approval.
3. To guide you (or your colleagues) while executing the usability test.

Your Test Plan Should Include the Following Sections

• Project name
• Purpose
• Location
• Schedule
• Sessions
• Participants
• Scenarios
• Quantitative metrics
• Roles

Test Script

A test script is a document which contains the words that a facilitator will say to the participant during the test. Sometimes also referred to as a protocol or moderator guide.

Introducing a Participant to a Usability Test: A Demonstration

Why Write a Test Script?

Creating a script helps you to:

• Focus your mind on what exactly you’re testing, so that your usability test doesn’t drift and remains focused.
• Ensure test consistency across multiple test participants.
• Talk about different user scenarios.
• Clearly articulate the different goals you’re testing.
• Put your users’ minds at ease.

Your Test Script Should Include the Following Sections

• Welcome
• User profile questions (if required)
• Introducing the test and product
• For each task…
  • Introduce the scenario
  • Introduce the task, and specific instructions to be given
  • Assistance to offer (if needed, and when)
  • Follow-up questions
• Overall follow-up questions
• Satisfaction rating
• Conclusion

NDA

An NDA (non-disclosure agreement) is a contract between the usability test participant and the business, which prohibits them from sharing information about your site or app when participating in your test.

Why Use an NDA?

An NDA allows you to share pre-release concepts and products with external stakeholders, while protecting them from being circulated, and potentially ‘leaked’ externally. It protects your ideas from getting stolen!

Outline

As an NDA is a legal document, you should not try to write one yourself without the necessary legal expertise. There are many templates available. Ensure that you enter:

• Your project/company name
• What will be tested
• What you will be observing and documenting
• Any special conditions/what participants ARE permitted to share

User Testing Steps

Mobile Application Usability Testing Process

Step 1: Preparation

Using your interactive prototype(visual mockups) with your test plan, conduct a usability test with 2 participants in-class and 3 participants outside of class for your scenario.

1) Scenario One: In your Usage Scenario document

• Summary: In your Usage Scenario document
• Tasks:
  • browses ___
  • navigates to ____
  • do something specific
  • navigates to ____
  • do something specific

Step 2: Testing

1. Introduction:
   • introduce yourself
   • promise confidentiality to the user (Sign NDA)
   • explain what you will be testing and the test method
     • You are testing the product, not the users.
     • Have users trying some features to help you identify usability issues so that you can improve the app user interface to achieve better user experience.
     • Think aloud method: Ask users to say what they are looking at, what they are trying to do, and what they are thinking.
2. Collect Demographic information:
   • Take the participant’s demographic information: Full Name, Gender, Age Range, Occupation, Browsing Device, Technical Ability etc.
3. Assign the tasks (scenarios from your Test Plan/Test Script)
   • The test user walks through test scenarios.
   • The researcher observes the performed task and takes notes on the steps taken by the user.

Note: Whenever possible, record each test session.

Generic questions to ask:

Before Task (profile questions):

• What do you do for living? What are your hobbies?
• What kinds of internet searching do you do on your phone vs your computer?
• Can you think of the last time you looked for a cleaning service?

Good opening questions (by Steve Krug): Please take a look at this app and tell us what you think it is: What’s the first thing you notice? What can you do with this app? What products or services are offered? Who is this app intended for? Just look around and speak everything that comes to your mind.

During Test:

• What are you thinking right now?
• Can you tell me more about what you just did?
• I notice you [did something] - can you explain that to me?

When struggling:

• Where would you tap (click) in this case?
• How do you think that would work?
• How might you get assistance to figure this out?

After Test:

• I noticed you had an issue with [something], what would have made it easier for you to understand?
• Do you have any questions for me?

Step 3: Document

• Take detailed notes of each user’s behaviour, comments, errors and completion (success or failure) Note: Use Quantitative Metrics from your Test Plan
• Review the errors or issues found to identify any required changes to the Visual Design and/or your Interactive Prototype(visual mockups)
• Summarize each test
  • ParticipantID (e.g. Tester 1)
  • Timecode of the issue (When have a video recording of your session, E.g. 24:12)
  • A brief description of the issue
  • The area of the site where the issue occurs (e.g. product detail page)
  • Possible solutions
  • Severity rating

Step 4: Report

• Summarize your test findings and analysis report and add them to your final project slide.
• Revise your UI design and prototype(interactive visual mockups) accordingly for the final project submission.
• Usability Test Results & Analysis slide content will include these 3 Questions and Answers
  1. What issues did the participants have while interacting with the prototype(visual mockups)?
  2. Based on the issues, what changes did you have to make to your visual design & interactive prototype(visual mockups) to improve usability?
  3. How does your App meet the persona goals?

LOGISTICS OF UT

Before Tests

• Ensure your prototype supports your test plan & script
  • Include sufficient screens/states in the interactive prototype (visual mockups)
  • Ensure your script acknowledges gaps in functionality (404 coming soon)
• Test material: Test plan, prototype, and test script & test notes
• DO A PILOT TEST!! (aim for approx 20 mins test e.g. with your classmate/test partner)

Test

• Decide if you want to take your own notes, or partner up with a classmate to take notes for each other.
• Find a participant.
• Run your test, while your note taker takes notes.
• Try to limit it to about 20 mins.
• When finished, take 5 mins to sync with your note taker (if you have one) or just jot down any additional notes
• Repeat with the next participant! (Take turns if you have a partner)
• Make sure you each perform your test on 3 participants

Best Practices

• Make sure your participant feels comfortable.
• Silent pauses are ok. Focus on watching and listening.
• Remain neutral. Do not show disappointment if tasks fail.
• Avoid leading questions or comments.
• If participant asks for help, start with “What do you think?”
• Stick to the script!
• Watch the clock!