While visiting Colonial Williamsburg, Virginia with my family a few years ago I got differing responses from the costumed staff members. The woman making candles pretended not to know about modern electric lighting, but the gardener was very open and honest about the limitations of his role-playing.
While he performed his gardening tasks in full costume and used authentic tools and techniques to tend his heirloom vegetables, he explained that 18th-century wells were much deeper than the one he was using to fetch water. His “well” was just a shallow hole supplied with tap water so that visiting children could help water plants. It helped them experience how labor-heavy irrigation limited gardening in the colonial settlement.
A real well would be more prone to carrying water-borne parasites, and Colonial Williamsburg could not afford to send kids home on a stretcher or infected with giardia. The authentic-looking gardener summed up the tension underlying any reenactment with the rhetorical question: “Just how much authenticity can you stand?”
Reenactments for Usability Testing
You could ask the same question about user testing medical devices, which is also a kind of dramatic reenactment. FDA’s Draft Guidance — Applying Human Factors and Usability Engineering to Optimize Medical Device Design (June 2011) — emphasizes realism in the tasks, training, users, environment, and the product to be tested. The guidance specifies simulated use of the device in “realistic situations” (p. 20) and states that validation testing should incorporate “a production version of the device, representative device users, actual use or simulated use in an environment of appropriate realism….” (p.23).
Clearly, the most valid study will be one that most closely represents actual use — representative users for participants, a realistic environment, the actual device, and a good simulation of the actual state of the user (level of arousal, training, forgetting what was learned in training, etc.). But when all of these factors are simulated with a high degree of realism and authenticity, study participants can be put at risk.
Trade-Offs in Testing Injection Devices
Think for example of testing an auto injector — if the device used in testing includes the needle, participants will be exposed to a potential accidental needle-stick injury, even if the simulation calls for performing the injection into a pad strapped to the body. If the device contains the actual drug, we might expose study participants to additional hazards.
Would conducting the simulated use study with a needleless/drug-less device provide the same data about the safety and effectiveness of the auto-injector? You might think so, but consider the degree of caution users might have working with a device they know contains a needle. The extra care people take when they know there is a potential risk might have a significant impact on the outcome of the study.
While the FDA is not asking us to expose our study participants to unnecessary risk, we must make a judgment call as to where to draw the line between authenticity and participant safety.
Tips for Reconciling Authenticity and Effectiveness
How to bridge the divide? Here are some ideas for troubleshooting authenticity in user-testing injection devices such as auto-injectors, pen injectors, and syringes.
In a recent study, we looked at intramuscular injection. We found that off-the-shelf practice injection pads were not thick enough to account for the length of the needle on the device. The needle would bend if it hit the hard backing of the pad. We ended up building our own injection pads with parts from commercially available injection pads. A colleague taught me the trick of gluing two injection pads together (just the soft part) to accommodate longer needles.
Inform the participant of any risk.
Include the risk of a needle stick in the participant’s informed consent document. Tell the participant at the start of the study session that the device is just like the real device except it does not contain an active drug. Remind them that it has a needle.
Protect the participant.
Having a proactive plan for how to protect the participant allows you to get as realistic as possible while still prioritizing safety. For example, fill the injection devices with a safer fluid that has the same viscosity as the drug. Make sure any needles used in a study are sterile. Think in advance about if and when the study moderator should intervene — for example, when a participant’s behavior looks as if it will put the participant or others at risk of a needle stick.
Plan for accidental needle sticks.
All study personnel should be trained regarding blood borne pathogens and know how to handle and dispose of a needle as biohazardous waste after an accidental needle stick. Be mindful that shipping biohazardous waste is tricky. Check related state regulations, and have first aid provisions on site to allow a participant to treat a stick. Document any such events.
Remember that you can do an excellent job of simulating the most important aspects of user experience without putting the patient at unnecessary risk. You do have to draw the line at some point and limit some of the realism. Think carefully about what aspects of use are key to conducting a realistic simulation. Then you will be in a good position to decide for yourself how much authenticity you can stand.