The term summative evaluation can be translated as “final evaluation.” A summative evaluation checks at the end of the development phase whether users can operate a device or a user interface safely.
This article provides an overview of the regulatory requirements for the “summative evaluation” (sometimes also referred to as “summative assessment“) and gives you tips on how to meet them.
Read an article on formative evaluation here.
What does “summative evaluation” mean?
IEC 62366-1:2015 + AMD1:2020 define summative evaluation as the evaluation of the user-product interface that is carried out at the end of the development of the user-product interface. Objective evidence must be obtained to determine whether the user-product interface can be used safely. The original English version of the standard reads:
“USER INTERFACE EVALUATION conducted at the end of the USER INTERFACE development to obtain OBJECTIVE EVIDENCE that the USER INTERFACE can be used safely.”
In accordance with the standard, manufacturers must check whether use errors or difficulties occur that could
- lead to hazards or
- to hazardous situations, or
- even lead to harm to the patient or user.
The standard notes that the summative evaluation also corresponds to the validation of the user-device interface:
SUMMATIVE EVALUATION relates to validating the safe use of the USER INTERFACE.
The FDA does not use the term “summative evaluation”, but refers to “human factors validation testing“:
Human factors validation testing is sometimes referred to as “summative usability testing.”
This article on usability validation describes the procedure in detail.
Regulatory requirements
a) Requirements of IEC 62366-1
The standard requires manufacturers to consider the following points for a summative evaluation:
- Selecting safety-related us scenarios
In doing so, the manufacturer should either include all safety-related use scenarios for summative evaluation or select them based on the severity of potential harm (see also FDA: “Critical tasks”). - Specifying users/ user profiles
Manufacturers shall document the users/ user profiles involved in the final evaluation. Profiles must be created specifically for “usability tests”. - Creating a plan for the summative evaluation
In the plan, manufacturers shall specify- which methods they want to use for the final evaluation and justify them (usability tests are not mandatory but provide objective data),
- which part of the user-device interface they will evaluate,
- which criteria shall be met to decide whether users can use the device safely (acceptance criteria or “correct use”), and
- (in the case of usability tests) which use environment and methods are used to collect and evaluate the data, particularly the use errors.
- Performing a summative evaluation
Finally, manufacturers are required to carry out the final evaluation. If safety-related user problems are found, it must be investigated whether the user interface is a cause. Manufacturers must then decide to revise or rather improve it to reduce or avoid the use problems and errors. Manufacturers must justify all decisions, considering and evaluating the risks and the residual risk.
If the manufacturers modify the user interface after the evaluation, a further summative evaluation is necessary.
b) FDA requirements
The FDA formulates its idea of professional usability engineering – calling it human factors engineering – in the corresponding guidance documents:
- Draft Guidance for Industry and Food and Drug Administration Staff – Applying Human Factors and Usability Engineering to Medical Devices
- Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management
c) Conclusion
Unlike the FDA, IEC 62366-1:2015 + AMD1:2020 do not require mandatory usability validation in the form of a usability test, i.e., a participatory observation with representative users in a representative (also simulated) use environment.
However, the non-binding “IEC 62366-2:2016” states:
“Summative evaluation generally involves performing a usability test under conditions of simulated use.”
The usability test, i.e., human factors validation testing (FDA), is not formally prescribed for formative or summative evaluation. Nevertheless, the usability test is the most sensible and best method, especially for summative evaluation, to capture all use errors and difficulties in the best possible way.
Methods for summative evaluation
Several methods are available for summative evaluation, such as:
- “usability tests,” with or without a participating observer (e.g., through a one-way mirror).
As mentioned, this method is recommended and is also prescribed by the FDA. - cognitive walkthrough, e.g., with intended users:
This method is more interesting for the formative evaluation. More communication can take place, e.g. between users and moderator, especially when testing with several users at the same time. - inspection by a usability expert:
This method is also more suitable for formative evaluation. Usability experts identify design weaknesses in relation to possible use errors on the basis of standards and experience. However, they are not intended users and may think and act differently.
Summative evaluation can also include several methods at any time. As a rule, however, only one is chosen for cost reasons.
For “legacy devices” that were placed on the market before the publication of the standard in 2015, no summative evaluations are necessary if they meet the requirements of IEC 62366-1 in 5.10 “User Interfaces of Unknown Provenance” (UOUP) and Annex C.
Conclusion
Manufacturers must prove that their user product interfaces do not lead to (unacceptable) risks and harm for patients or users. The latter is possible if use errors and difficulties occur precisely when performing hazard-related tasks.
The best way to prove this is with usability tests.
Because IEC 62366-1 also interprets this way, manufacturers must provide reasons for their different approaches.
Benefit from the usability experts at the Johner Institute. They carry out usability tests for you worldwide and ensure that you find and eliminate use problems quickly and pass audits and approvals in all markets without any problems with compliant usability engineering reports.
Contact us to find out how you can meet the requirements of IEC 62366-1 and the FDA quickly and with minimal effort.
Change history
- 2024-04-12: Article largely revised and updated
- 2016-01-22: First version of the article created
