Week Fourteen - Usability Testing Reporting

Learning Objectives

Learn to effectively summarize and communicate usability testing results.
Gain the ability to translate raw data into meaningful findings.
Learn to assess the severity and impact of usability issues.
Acquire skills to propose practical solutions based on test results.
Understand how to present findings clearly and compellingly.

Slides

Usability Testing Reporting

What is a usability test report?

A document that summarizes the results of usability testing, focused on communicating findings (differentiated by levels of severity) and recommendations.

It can consist of direct quotes, paraphrasing, summarized observations, screenshots, or whatever will effectively communicate your findings.

Why create a usability test report?

Communicate results to teams and stakeholders.
Identify and prioritize improvements.
Record findings for future projects.

Do you always need a big report?

Quick findings are sufficient when:

Usability testing is familiar to the team and stakeholders.
Most team members and stakeholders participated in or observed the testing.
During quick, iterative testing where formal reports slow down the process.

How is it shared?

Presentation with a slide deck.
Read-out of the report, highlighting key points.
Circulating a document (email or other).
Posting on a research platform like Dovetail.

How to make the data compelling?

Use screenshots with highlighted key areas.
Include impactful video clips from test sessions.
Add direct quotes from participants.
Display larger data sets as graphs.

How to translate data into findings?

Identify initial themes you noticed in testing (and leave a ‘general’ category for anything that doesn’t fit)
Go through your results, one participant at a time, find quotes or observations, and sort them into the themes (tag with participant #)
When finished, see if additional themes emerged in ‘general’
For each theme, you will have one or more findings. For example:
Participants struggled to interpret several icons
Participants could successfully filter the results list

How to prioritize severity?

For each finding, prioritize severity on a 3 or 4 point scale. For example: 0 Positive observation (not a usability problem) 1 Minor problem (cosmetic or otherwise minimal) 2 Serious problem (frustrating to users, inefficient, may cause task abandoning) 3 Critical problem (it prevents users from completing tasks)

How to prioritize impact?

Use a 3 or 4 point scale:

0: Positive observation (not a usability problem).
1: Minor problem (cosmetic/minimal).
2: Serious problem (frustrating, inefficient, may cause task abandonment).
3: Critical problem (prevents task completion).

How to translate findings into recommendations

For each finding, you should include a recommended action.

Leave as-is & monitor (if low-impact)
Implement a specific fix (if you know what would fix the problem)
Explore further (if the fix is not obvious and needs an iteration)

How do you come up with a specific fix?

Based on frequency and task importance:

Many participants, key task affected.
Many participants, infrequently-used feature affected.
Few participants, key task affected.
Few participants, infrequently-used feature affected.

If you are not certain that the fix will do all of this, recommend further exploration/investigation.

How to Translate Findings into Recommendations

For each finding, recommend an action:

Leave as-is & monitor (low-impact).
Implement a specific fix (if known).
Explore further (if the fix is unclear).

How Do You Come Up With a Specific Fix?

Ensure the fix:

Follows good design principles.
Corrects the usability problem without introducing new issues.
Isn’t just what users asked for without design team vetting.

To Ensure Your Report Leads to Positive Changes

Be specific for clear understanding.
Organize and rank findings by importance.
Use quotes, videos, etc., to bring data to life.
Include positive findings to preserve good aspects.
Choose the right format for your audience.

Presenting Your Work

Types of Presentations

Understanding the context and purpose of different types of presentations is crucial in UX design. These can include:

Scope and Strategy Presentations: Discussing project boundaries and objectives.
Design Reviews: Sharing design progress for feedback and direction.
Research Results: Presenting findings from user or market research.
Stakeholder Presentations: Gaining buy-in or approval at critical project junctures.
Final Deliverable Presentations: Showcasing the completed project.

For effective presentations, consider:

Audience: Who are they and what is their interest in your project?
Purpose: What do you intend to achieve with this presentation?
Agenda: Outline the key points and structure of your presentation.
Stakeholders: Identify the decision-makers and influencers.
Desired Outcome: What is the ideal result of this presentation?

Scope and Strategy Presentation

When: Early in the project.
What: Define what’s included and the project goals.
Level of Detail: Focus on the problem to be solved, not design proposals.
Content: Include market, product, and user research.

Design Reviews

When: Throughout the project.
What to Cover: Design direction, broader perspectives, concerns, constraints, and course corrections.
Content: Present enough to elicit the feedback you need.

Research Results

When: Post-research activities.
What to Cover: Share learnings, discuss results, and collaborate on recommendations.
Content: Present data/results and recommendations, either formally or informally.

Presenting to Stakeholders for Buy-In

When: At critical decision points.
What to Cover: Scope, goals, value, and impact of the project.
Content: Research, concepts, storyboards, technical evaluations, and effort scoping.

Presenting Final Deliverables

When: Project completion.
What to Cover: Final output and supporting materials.
Content: Focus on key elements; be prepared with additional material for questions.

Giving and Receiving Feedback

Good Feedback:

Specific, objective, and aligned with product/user goals.
Open to diverse opinions without needing consensus.

Poor Feedback:

Vague, subjective, based on personal preference.
Expects immediate agreement from the receiver.

Receiving Feedback

Understand that feedback targets the design, not you personally.
Aim to improve the design, even if it requires significant changes.
Stay open and non-defensive; seek clarity on feedback.
Consider all feedback, but you don’t have to implement every suggestion.

Planning Presentations

Preparation Essentials

Know your time limit and essential content.
Understand your audience and their background knowledge.
Choose content that will drive the desired feedback or discussion.
Consider the presentation setting: low-tech, mid-tech, high-tech, or remote.

DOs and DON’Ts

DOs	DON’Ts
Clearly state the presentation’s purpose.	Overwhelm with irrelevant details.
Approach with an open mind.	Focus on effort over substance.
Ensure your material communicates your intentions effectively.	Apologize for perceived shortcomings.
	Rely on a script or get defensive.
	Ask generic questions like “Do you like it?”