One of the biggest challenges researchers face while conducting usability testing for medical devices is assessing and understanding why a use error occurs. (“Use error” is the human factors term for any error involving a human operator.) Getting to the root cause of each use error is essential, because the nature of the error informs the fix.
One of the places we look for use errors is during formative usability studies for medical devices, when we’re gathering information to help clients improve a product before the design is finalized. We also probe use errors during summative or validation studies, once the product’s design has been finalized and the client is tasked to prove to the FDA that it will be used safely and effectively by the intended users.
Getting to the root cause of each use error is essential, because the nature of the error informs the fix.
Though documenting use errors during formative testing isn’t required by the FDA, we always aim to run formative studies as thoroughly as we do our summative studies. That’s because making improvements is generally easier, more effective, and less costly the earlier you are in the process.
A PCA Method Primer
In November, 2017 I attended a three-day AAMI Human Factors for Medical Devices course that surveyed the latest FDA updates for human factors medical device testing, industry case studies, and practice sessions. Although these topics were all informative and very helpful to understand for our research practice, one of the most interesting things I learned about was the PCA method for evaluating use errors in formative or summative usability testing.
Researchers typically use a combination of methods to probe errors, including interviewing study participants to ask why they did something and observing them while they use the product. The PCA framework, which the FDA formally recommended in their human factors guidance of February 2016 — is a great tool for identifying why each error occurred.
PCA stands for perception, cognition, and action, the main categories of use errors. Users either have trouble perceiving stimuli, cognitively processing that stimuli, or acting on it. The figure above, Device User Interface in Operational Context (adapted from Redmill and Rajan, 1997), presents a model of the interactions between a user and a device, the processes performed by each, and the user interface between them. The critical element in these interactions is the device user interface.
Once we locate use errors via interviewing and observation, our team sorts those errors into one of the following categories, to help inform the next steps.
Perception errors may involve failures to see visual information, such as a display being partially obstructed or reflections distorting the screen; or to hear auditory information, for example when there is ambient noise or information overload during the task at hand. It can also include instances of misreading information.
Cognition errors fall into three subcategories: memory failure, rule-based failure, and knowledge-based failure. Memory failure is the inability to recall knowledge that was gained before, or omitting a planned step. Rule-based failure involves the misapplication of a generally accepted rule. Knowledge-based failure is based on improvisation under unusual circumstances or misinterpretation of information due to an incorrect mental model.
Action errors are caused by events like failure to reach controls, contact with the wrong component, inappropriate force applied to component, or failure to activate a control (e.g. not enough force).
The PCA Method in Action
Recently we had the opportunity to interview participants in a clinical study about their use of a diabetes insulin pump. This population was new to pumps — they had always managed their insulin through other methods, including pills, an autoinjector pens, and/or syringes and vials.
One major finding from the interviews was that participants were sometimes double-dosing themselves (during simulated use, not injecting insulin) with the pump, and not realizing it.
There were three main reasons this was happening:
- The user rounded up the dose amounts.
- The user did not understand the difference between dose types.
- The user accidentally pressed a button too many times.
We sorted each error into a different PCA category:
- The user rounded up the doses because they either forgot or misread the dosing instructions, therefore it is an error in cognition, if they forgot, or perception, if they misread.
- Not understanding something is a sign of a cognitive error. The user did not understand some of the key concepts related to the product.
- Accidentally pressing a button multiple times could be an action-based error if the nature of the button made it difficult to press the number of times intended. If the users pressed the button too many times due to miscounting, this would be a cognitive error.
Once a use error is slotted into one of these areas, it needs to be addressed through mitigation, or improvements, to make the device safer and more effective to use. The PCA framework helps organize errors so that their severity is more straightforward to assess (is it an error, a close call, or a difficulty?) through risk analysis and to help frame the best mitigation approach.
The FDA’s Recommended Mitigation Approaches
The FDA recommends different paths for mitigation by offering three categories of improvements, in order of preference from best possible product to least best possible product. The FDA’s mitigation categories are “inherent safety by design”; “protective measures in the medical device itself”; and “information safety”:
Safety by design involves changing the design of the product to be inherently safer — this could include modifications to the design’s form or function.
Protective measures in the device itself means incorporating safety guards or adding alerts for hazardous situations. This could involve design changes to the interface, such as building in areas where you can’t enter info unless some other info is already entered. (This could impede the usability of the product, which is something you need to measure against risk.)
Information safety involves altering the user manual to include cautions or warning statements, or training users to avoid the error.
The FDA’s categories of improvement are a good illustration of why we recommend usability testing early and often. “Safety by design” changes are much easier and less costly to pursue early in the design and development process.
Choosing the Best (For Now) Approach
At this point in the process we recommend a fix we think is best, and we always include “safety by design” options. Each use error may require a different approach to resolve it, but ultimately the answer relates to the client’s risk analysis and product cycle.
Although sorting information into neat categories effectively brings us this far, this last part of the process isn’t as straightforward. Much discussion is sparked, and typically we end up with a solution that is somewhere in the middle, that involves some additional development effort but significantly increases safety and efficacy.
The value of study results and recommendations doesn’t stop here. They often continue to exert influence over a next-generation design once the first generation design is on the market.