Usability validation is a review using objective means to determine whether specified users can effectively and efficiently achieve the specified usage goals (intended purpose) in the specified context of use.
Regulatory requirements for the usability validation
The Medical Device Regulation MDR explicitly calls that manufacturers must identify and control risks that arise from a specific context of use and by the characteristics of the user (for example level of education, intellectual and linguistic skills). The FDA does the same.
Read all the usability requirements of the MDR here: MDR usability requirements.
To demonstrate that these requirements are satisfied, it is necessary to validate usability. The IEC 62366 and the Human Factors Engineering Guidance Document of the FDA provide guidance on how this usability validation must be carried out.
1. Characterization of the user
Thus, the usability validation must be carried out with typical representatives of the specified user group, usually in the form of participant observation. This test, of course, assumes that you have specified:
- The user group, for example, based on age, education, experience with the product, physical and intellectual abilities.
- The context of use, such as “mental workload”, the environment (temperature, humidity, brightness, …) the tasks that are to be carried out and many more
- User goals, the (medical) purpose is usually referenced here.
That users and the use environment must be specified, is a requirement of the IEC 62366 in Section 5.1.
2. Involvement of representative users in the usability validation
Only for the usability validation will you need necessarily representative users and representative context of use (use environment).
3. Number of subjects in the usability validation
The IEC 62366 demands representative users in the validation of serviceability. How many of these there are, however, it does not say. This is hardly doable, the number also depends on how homogeneous the different user groups are.
The FDA, however, references in its Guidance Document “Applying Human Factors and Usability Engineering to Optimise Medical Device Design” a study that mentions more concrete numbers.
As you can see, you can find, on average, 85% of the utilisation problems with five representative users, with ten you can already 95%. Those are concrete figures!
Update: The FDA requires at least 15 participants per user group in the Human Factors Engineering Guide, and even more for high-risk products.
4. Usability validation plan
According to IEC 62366-1, the safety-related usage scenarios must be involved in the usability validation plan. The validation plan must also determine the acceptance criteria for the usability validation. The criteria for acceptability should always be the correct execution of a task or subtask without use errors or difficulties of use. As soon as an error or difficulty is observed, the cause must be analyzed, and a risk evaluation must be carried out in accordance with ISO 14971.
Acceptance criteria such as the following are not expedient: “X% of users must have completed a task in a maximum of Y minutes”. Why only X% of them and not all? The fact that not all of them can accomplish the task is more of a hint for usability problem. And why in Y minutes? Where did this number come from?
The achievement of time goal is still nothing for the risk management. Because it could also mean that an acceptable but a non-achievement device (from the perspective of risk management!!) will be developed. The other way around, it could also be that in the process of maintaining targets itself remains an unacceptable risk.
Lack of Usability leads to damages only through a chain of causes. The FDA will also not comment on usability targets, but targets that affect the risk. With the usability engineering process, you will be (hopefully) aware of potential risks. And this will then be examined in risk management.
Methods for usability verification and usability validation
There are several methods for testing the usability. A distinction is made between the following:
Usability verification: Inspection
The verification of the suitability for use will be successful with an inspection. This is a verification procedure whereby one or more experts examine your products on
- whether the specified requirements, such as those found formulated in style guides and standards are implemented and / or
- whether the product (in principle) is able to meet the usage requirements. If the usage requirement specifies that the physician must identify the patients with a low hemoglobin, and the system does not display the hemoglobin value, then this requirement is not met.
There are several methods for verification, such as the cognitive walkthrough.
Usability validation: Participatory observation
However, the inspection alone is not enough. You have to run through the core tasks within a participatory observation of genuine users in an actual or simulated user environment and let the security functions run. Only when users are actually able to achieve the targets, and thus prove that the user requirements are met, will the usability of your product be validated.
This assumes that you know all the core tasks. This will lead you to the frequently used functions. Next you need to know all the safety-critical functions. These result from the risk analysis.
User surveys
And so why do they still require the user surveys? When it comes to the purposes of IEC 62366 they don’t actually. Use questionnaires to compare products or development statuses quantitatively. And use interviews to objectify complaints.
Formative and summative evaluation of usability
What the FDA has already implemented, will be included in the second edition of IEC 62366 collection: The terms verification and validation of serviceability will be “replaced” by the terms formative and summative evaluation. But, is this really a replacement?
Actually, the two pairs of concepts have little to do with each other – apart from the fact that they are all related to the testing of usability. The two pairs of terms refer rather to different dimensions: usability validation and usability verification differ in the objective of the test. The terms formative and summative evaluation differ in the time of the test.
For example, examining a GUI prototype during development would be a validation and a formative evaluation. A check of serviceability of the finished product with representative users in a representative environment of use would also be a validation, but a summative evaluation.
What you should not confuse the usability validation with
“Classic” system / product validation versus usability validation
You will find the difference between these processes described in another article.
Usability validation versus clinical trials
I came across an article in the “European Medical Device Technology” on serviceability. (The link http://www.donawa.com/medical-device/files/EMDT%20Jun11%20Usability.pdf is unfortunately no longer available). In it, the author points to a paragraph of IEC 60601-1-6, which strikes me as very important:
It should be noted that clinical investigations conducted according to ISO 14155-1 and usability testing for verification or validation according to this standard are two fundamentally different activities and should not be confused.
In other words:
- The usability study has to validate towards the goal, if the specified user in the specified context of use can effectively and efficiently achieve the objectives. A prerequisite for the validation of usability are representative users.
- Clinical studies, however, must prove the goal that the product has a positive benefit-risk ratio. Patients are prerequisite for a clinical trial.
Although both studies are possibly carried out in the clinic “in real terms”, the objectives are nevertheless different, though not overlapping. It is often useful to separate the tests of one another.
Usability validation versus field test
One of my favorite clients had the idea of validating his software in a field test, in order to meet the relevant legal requirements for validation. He almost committed an offense.
His thought to test, whether the purpose and usage requirements are met in the field was really good. At the end of the day, the IEC 62366 demands that one involves “real” users. But to test the application in a field test in an uncontrolled way, corresponds to placing it on the market in an uncontrolled way. That is a criminal.
The correct approach would have been to carry out this test either in a simulated environment like a usability lab or in a strictly defined clinical trial. But at the clinical trials some conditions are met.
But especially for software requirements this is often not required. For this reason, I recommend the Usability Lab. Anyone who has observed the manufacturer, how they view the users working with their software in a concerned, unbelievable and stunned way, will no longer suspect the following: That a usability lab is suitable for identifying usability issues. There’s no faster and cheaper way to experience, what many do not even really want to know when it comes to validating: The naked and often brutal truth about their product.
Further information
Usability lab
We have a usability lab, where we test customer products and demonstrate various methods. From Cognitive Walk-through to the expert evaluation. From eye-tracking to software-based usability testing (usability validation).
We have validated different products and recognized, which of these applications have partly catastrophic defects.
Links to the subject usability
- The Federal Institute for Occupational Safety and Health has published the ergonomics compendium “application Ergonomic rules and testing the usability of products” (German) with the involvement of notified bodies.
- ANSI / AAMI HE75: 2009 – Human Factors Engineering – Design of Medical Devices originates from the FDA environment
Both documents do not focus on software. But they contain specific provisions on the specification of the usability of products – in contrast to IEC 62366, which is a standard process. And another thing the two documents have in common: a volume of several hundred pages.
Contact us to find out what Johner Institute can do for you.
