Medical device usability testing serves different — and equally important — purposes throughout the product design process, from generative through evaluative research.
Too often, “simulated use” translates to a table with a few chairs and a sink, as opposed to a simulated use environment that truly represents a home or a medical environment. The current industry standard is to bring the use environment for usability testing only as far as required.
I’ve always felt that it’s best practice to test a product’s usability with an “as-close-as-possible-within-reason” approach. This, hopefully, gets participants in the mindset of the product’s context of use and can improve the quality of their feedback.
If it’s ultimately the best thing for the product, why aren’t we challenging the norm by aiming to bring the simulated use environment as far as possible?
Why Invest in a High-Fidelity Use Environment?
As the need for hospital simulation centers has grown, we’ve seen ever-increasing access to a large diversity of resources for usability testing. But even with these resources available, testing outside of market research facilities is often met with resistance to go beyond the basics. This resistance typically has two major drivers: The first is cost, and the second is concern over the consistency of environmental factors.
Reason #1: Increased Participant Engagement
Creating a more complex use environment can be more expensive than using a facility. But I’ve found that the additional budget required to produce a more representative simulated use environment is more than recouped when it ultimately increases participant engagement.
This is because failing to invest in a deeper-fidelity use environment is more likely to result in money lost to wasted time, space, and compensation. I’ve frequently seen potential data scored as an artifact and thrown out after participants stated that they did not perform the tasks as they would have, because they “were not in a real-life situation or environment.” (In user testing, an “artifact” is misleading data resulting from a flaw in study protocol, in this case the simulated use environment.)
The additional budget required to produce a more representative use environment is more than recouped when it ultimately increases participant engagement.
That’s exactly the kind of thing that might happen if you’re using a market research facility to test a product meant for use in an operating room. There are so many “things” in a real OR. There are overhead lights over the table that can be adjusted. There are carts all over the place with supplies. The anesthesiologist cart is off to one side. The supply cabinet is often in the room as well, and equipment can often be retrieved on the spot.
Some may say these details aren’t necessary, but they are to the representative users. So are the additional sets of hands typically found in an OR. (Picture the busy ORs in episodes of the popular ’90s TV show, ER.)
Reason #2: Fewer Use Errors and Artifacts
Resistance also comes from concerns over the consistency of environmental factors. The truth is, a higher fidelity environment reduces instances of environment-related use errors and artifacts. I’m thinking of a recent study held in a simulated catheter lab.
Instead of having to make justifications in the protocol for this study, we were able to state that we were in a hospital with natural background noise; using representative lights; and even real intercom announcements! Having as realistic a scenario as possible prevented study artifacts. Participants came out of that study saying things like, “I felt like I was in my OR at work. I didn’t have to force anything!”
Here’s another example of how a study held in a market research facility might produce inexact results: During study conduct, there are situations when the moderator will indicate to the participant to use the device exactly as they would in real life. For example, let’s say that the device is intended to be used at the patient’s bedside to help place a peripheral line.
Participants came out of that study saying things like, “I felt like I was in my OR at work. I didn’t have to force anything!”
Most of the time in this situation the user will be standing, but if we are seated at a table using the device, the participant may decide to just go ahead and use it sitting down because this is “close enough.” However, ergonomically speaking, using anything from a seated position versus a standing position can create numerous challenges.
Ultimately, a more accurate simulated use environment adds tremendous value to your study conduct. It assures the regulatory bodies that you’ve recreated the use environment in such a way as to prevent artifacts and observe previously unanticipated use errors.
How To Determine the Optimal Level of Fidelity for a Usability Study
The human factors and usability industry conducts usability studies of all kinds in traditional market research facilities to great success, in many cases.
But there are scenarios when one should seek to go beyond the basic simulated use environment. Asking yourself these questions will help you determine what’s necessary:
- Are there complexities in your use environment that will be difficult to simulate in a traditional market research facility (e.g., lighting specifications, large diagnostic devices, etc.)?
- Is the product you’re testing typically used in a variety of use environments that may pose challenges specific to the location?
- Does the user interaction with the product’s user interface differ in locations where the product would be used (e.g., ER versus ICU, outpatient versus inpatient, ambulance versus ER)?
- Is the user environment an operating room or some other complex, multisensory environment?
Even if the project budget won’t stretch to include the price of a high-fidelity use environment, there are readily available and affordable tools and methods you can use to create a space that increases participants’ ability to “do what they would normally do” — and to minimize study artifacts.
There are affordable tools and methods you can use to create a space that increases participants’ ability to ‘do what they would normally do.'”
In the example I provided earlier, about the participant using the device from a seated position instead of from a standing position as it’s used in real life, it could have been as easy as having a table set at the same height as a patient bed in the center of the room. The interview portion of the study can be held off to a corner while the product being tested is over by the “patient bed.”
Then, as the study is ready to proceed into testing, the moderator could stand up and move toward the “patient bed” and hand the product to the participant to begin. This simple re-arrangement of the use environment could have very well mitigated against any potential artifacts due to a non-representative environment.
The underlying reason to invest in a more accurate simulated use environment? It’s just good research in the proper context! Would you conduct a clinical trial with someone who might be a representative patient? Of course not! You would ensure that the patient has all of the appropriate indications to represent how the drug might help treat the disease. Let’s start thinking about our use environments in the same way.