Heuristic Evaluation of Healthcare Technology

PEP Talk Logo

In this first PEP (Practical Exercises for UX/HF Practitioners) talk, our Sage President and Sr. UX/HF Consultant, Shannon Halgren, PhD taught our Medical Technology UX/HF Meetup group how to conduct a Heuristic Evaluation of Medical Technology.   See our PEP Talk Introduction post which describes the PEP Talk format and our Medical Technology UX/HF meetup.   We hope you can come join us at our next meeting!

A summary of the talk is provided below, but we also encourage the reader to red the presentation slides which provides more detail.

Click here to view

Conducting a Heuristic Evaluation of Healthcare Technology
PEP TALK Overview

Heuristic Evaluation (HE) is a part of a group of activities that identify use-related risk factors and improve product usability.  It is also a proven budget-sensitive usability inspection technique.  The results of a Heuristic Evaluation serve as inputs to the product design and the use error analysis (uFMEA or similar).

HE is a great technique to use at multiple points in the product development cycle.  Check out this table below to see when conducting a HE might be right for you!

When to Use HE
WHen Not to USe HE
  • Mid to late design phase (formative)
  • Prior to usability testing (find low hanging fruit)
  • When you need feedback fast
  • When budget is a factor
  • As part of your documented use process
  • As a substitute for user testing
  • When designers are struggling with a design problem

Heuristic Evaluation Step-by-Step

There are three steps to the Heuristic Evaluation process: 1) Define, 2) Evaluate, and 3) Report.

Step 1: Define Users Tasks Heuristics Evaluators

THE FIRST STEP IS TO DEFINE.

As Shannon discussed in her talk, before starting the HE review users, tasks, heuristics and evaluators all need to be defined. The table below discusses these categories to determine for each of these items in more detail.

CATEGORY
ITEMS TO DETERMINE
Users
  • Job Title
  • Experience
  • Responsibilities
  • Motivators
  • Skills
  • Age
Tasks
  • Identify tasks using task analysis or use scenarios
  • Map tasks to user type
  • Identify primary and secondary tasks
  •  Consider
    • Workflow
    • Frequency
    • Inputs & outputs
    • Work Environment
  • Define usability and clinical risk severity ratings
Heuristics Choose your set of heuristics. If you are evaluating a medical device, Shannon recommends using the set developed by Zhang et. al [2] specifically for medial devices:

  1. Consistency
  2. Visibility of System State
  3. Match between System & Real World
  4. Minimalist
  5. Minimize Cognitive Load
  6. Feedback
  7. Flexibility & Efficiency
  8. Good Error Messages
  9. Prevent Errors
  10. Provide Clear Closure
  11. Undo/Reversible Actions
  12. Use Users’ Language
  13. Users in Control
  14. Help & Documentation
Evaluators
  • Number of evaluators – recommended is 3-5 (more if usability critical)
  • FDA guidance specifies evaluators should be HF experts
Step 2 - Evaluate

THE SECOND STEP IS TO EVALUATE.

In the evaluation step, evaluators will evaluate the device and task workflow from the point of view of both a user performing tasks using the device and as a UI expert.  In both cases, the evaluators use the chosen set of heuristics and take notes as they work.  Evaluators typically perform independent reviews and spend one or two hours on their review.  Longer evaluation sessions may be necessary for larger or  complicated interfaces with a substantial number of dialogue elements [1].

Step 3 - Report

THE THIRD STEP IS TO REPORT.

The Heuristic Evaluation report will include an overview of the HE process and present the findings.  Report formats may vary, but typically each heuristic is scored independently and specific violations are listed and rated in terms of usability and risk severity.  This report (which can be in either word doc or excel format) can then be used to support quality discussions between the evaluators and the product team.

 

IN SUMMARY

Heuristic Evaluation is a usability engineering inspection technique for finding usability problems in a user interface design.  It is an excellent method to use to order to catch the “low-hanging fruit” prior to investing in usability testing with end users.  HE involves a small set of expert evaluators who examine the interface and assess its compliance with “heuristics,” or recognized usability principles. HE results are an excellent input to the product design and use error analysis (e.g., uFMEA).

References

[1] Nielsen, J. 1992. Finding usability problems through heuristic evaluations. Proceedings ACM CHI’92 Conference (Monterey, CA, May 3-7); 373- 380.

[2] ZHANG, J., et al. (2003) Using usability heuristics to evaluate patient safety of medical devices, J of Biom Info (36)1:23-30

BY SHANNON HALGREN / ABOUT AUTHOR
Shannon Halgren headshot
Speaker Bio

Shannon has extensive experience in the area of product user experience and human factors, with an expertise in user research and healthcare technology. Through Sage Research & Design, Shannon and her team have helped start-ups as well as established Fortune 500 companies like Dell, Apple, Facebook, Intel, Netflix and eBay create products with excellent design and usability and have helped healthcare technology clients such as Abbott, Ascom and Hach design safe and FDA-compliant products. Prior to founding and growing Sage for over 10 years, Shannon co-managed the usability group at Macromedia and founded the usability group at Apple/Claris. Over the years, Shannon has presented her work at several professional conferences (HFES Healthcare Symposium, UXPA, CHI) and in professional journals and books. She holds a M.A. and Ph.D. in Experimental Psychology from Rice University in Houston, Texas where she specialized in Human-Computer Interaction.