Summative evaluation: The following points are essential

The term summative evaluation can be translated as “final evaluation.” A summative evaluation checks at the end of the development phase whether users can operate a device or a user interface safely.

This article provides an overview of the regulatory requirements for the “summative evaluation” (sometimes also referred to as “summative assessment“) and gives you tips on how to meet them.

What does “summative evaluation” mean?

IEC 62366-1:2015 + AMD1:2020 define summative evaluation as the evaluation of the user-product interface that is carried out at the end of the development of the user-product interface. Objective evidence must be obtained to determine whether the user-product interface can be used safely. The original English version of the standard reads:

“USER INTERFACE EVALUATION conducted at the end of the USER INTERFACE development to obtain OBJECTIVE EVIDENCE that the USER INTERFACE can be used safely.”

In accordance with the standard, manufacturers must check whether use errors or difficulties occur that could

• lead to hazards or
• to hazardous situations, or
• even lead to harm to the patient or user.

The standard notes that the summative evaluation also corresponds to the validation of the user-device interface:

SUMMATIVE EVALUATION relates to validating the safe use of the USER INTERFACE.

The FDA does not use the term “summative evaluation”, but refers to “human factors validation testing“:

Human factors validation testing is sometimes referred to as “summative usability testing.”

Regulatory requirements

a) Requirements of IEC 62366-1

The standard requires manufacturers to consider the following points for a summative evaluation:

• Selecting safety-related us scenarios
  In doing so, the manufacturer should either include all safety-related use scenarios for summative evaluation or select them based on the severity of potential harm (see also FDA: “Critical tasks”).
• Specifying users/ user profiles
  Manufacturers shall document the users/ user profiles involved in the final evaluation. Profiles must be created specifically for “usability tests”.
• Creating a plan for the summative evaluation
  In the plan, manufacturers shall specify
  • which methods they want to use for the final evaluation and justify them (usability tests are not mandatory but provide objective data),
  • which part of the user-device interface they will evaluate,
  • which criteria shall be met to decide whether users can use the device safely (acceptance criteria or “correct use”), and
  • (in the case of usability tests) which use environment and methods are used to collect and evaluate the data, particularly the use errors.
• Performing a summative evaluation
  Finally, manufacturers are required to carry out the final evaluation. If safety-related user problems are found, it must be investigated whether the user interface is a cause. Manufacturers must then decide to revise or rather improve it to reduce or avoid the use problems and errors. Manufacturers must justify all decisions, considering and evaluating the risks and the residual risk.

b) FDA requirements

The FDA formulates its idea of professional usability engineering – calling it human factors engineering – in the corresponding guidance documents:

• Draft Guidance for Industry and Food and Drug Administration Staff – Applying Human Factors and Usability Engineering to Medical Devices
• Medical Device Use-Safety: Incorporating Human Factors Engineering into Risk Management

c) Conclusion

Unlike the FDA, IEC 62366-1:2015 + AMD1:2020 do not require mandatory usability validation in the form of a usability test, i.e., a participatory observation with representative users in a representative (also simulated) use environment.

However, the non-binding “IEC 62366-2:2016” states:

“Summative evaluation generally involves performing a usability test under conditions of simulated use.”

Methods for summative evaluation

Several methods are available for summative evaluation, such as:

• “usability tests,” with or without a participating observer (e.g., through a one-way mirror).
  As mentioned, this method is recommended and is also prescribed by the FDA.
• cognitive walkthrough, e.g., with intended users:
  This method is more interesting for the formative evaluation. More communication can take place, e.g. between users and moderator, especially when testing with several users at the same time.
• inspection by a usability expert:
  This method is also more suitable for formative evaluation. Usability experts identify design weaknesses in relation to possible use errors on the basis of standards and experience. However, they are not intended users and may think and act differently.

Summative evaluation can also include several methods at any time. As a rule, however, only one is chosen for cost reasons.

Conclusion

Manufacturers must prove that their user product interfaces do not lead to (unacceptable) risks and harm for patients or users. The latter is possible if use errors and difficulties occur precisely when performing hazard-related tasks.

The best way to prove this is with usability tests.

Because IEC 62366-1 also interprets this way, manufacturers must provide reasons for their different approaches.

---

Change history

• 2024-04-12: Article largely revised and updated
• 2016-01-22: First version of the article created