Formative evaluation: What the FDA and IEC 62366 require

The regulations relevant to usability engineering require formative evaluations.

1. Formative evaluation: Definition and differentiation

a) Definitions

In its guidance document, the FDA has published its own definition of the term:

Thus, it replaces the old definition in the guideline Applying Human Factors and Usability Engineering to Medical Devices:

„Process of assessing, at one or more stages during the device development process, a user interface or user interactions with the user interface to identify the interface’s strengths and weaknesses and to identify potential use errors that would or could result in harm to the patient or user.“

The formative evaluation must, therefore, be correctly understood as a development-accompanying evaluation of the user-product interface. It is carried out to discover the strengths and weaknesses of the user-product interface as well as unforeseen use errors.

b) Differentiation: Formative and summative evaluation versus verification and validation

IEC 62366-1:2016 requires a formative (during development) and summative (final) evaluation instead of verification and validation of usability.

The two pairs of terms have little to do with each other – apart from the fact that they are all related to the usability inspection. Instead, the two pairs of terms refer to different aspects: Usability validation and usability verification differ in the objective of the inspection. On the other hand, the terms formative evaluation and summative evaluation describe the points in time of the inspection.

2. Formative evaluation: Regulatory requirements

a) IEC 62366-1:2015

IEC 62366-1:2015 requires the planning and implementation of formative (during development) and summative (final) evaluations.

Although the standard does not specify methods, it does require manufacturers to plan,

• which methods they will use for the evaluation (with a justification of the choice),
• at which points in time they will carry out these assessments,
• which parts of the user-device interface will be evaluated,
• which criteria they test, and
• with which users and in which user environment these evaluations are to take place.

Of course, these planning activities, the execution of the evaluation, and the results and their evaluations must be documented.

b) FDA

The FDA formulates its requirements in the guidance document “Applying Human Factors and Usability Engineering to Medical Devices.” Chapter 6.4.3 provides detailed information on formative evaluation.

Objectives

According to the FDA, manufacturers can use the formative evaluation to

• identify unknown use-related hazards, use errors, and critical tasks,
• implement changes to the user device interface in a timely manner,
• review the effectiveness of risk minimization measures on the User Interface (UI),
• determine the need for training measures and
• derive content for the validation plan.

Methods

The US authority does not require a specific method, but writes:

„Formative evaluation complements and refines the analytical approaches described in Section 6.3, revealing use issues that can only be identified through observing user interaction with the device.“

Thus, the FDA associates formative evaluation with methods in which users are to be involved and not with pure verification activities. The FDA does not exclude company employees as users but writes, “their performance and opinions could be misleading or incomplete.”

In the same document, however, it becomes clear that the FDA does not recognize participant observation as the only permitted method. The authority also mentions “cognitive walk-through, observation, discussion, interview.”

Planning

According to the FDA, the formative evaluation plan should specify:

• Objectives and priorities of the evaluation
• Elements of the user-device interface to be evaluated
• Use scenarios and tasks
• Participants/users
• Methods for data collection and data analysis
• How the results will be further used

These requirements are almost completely in line with those of IEC 62366-1:2015.

Documentation

The documents that should be created during the formative evaluation are defined in the FDA guidance “Content of Human Factors Information in Medical Device Submissions.”

3. Methods for formative evaluation

Many methods are suitable for development accompanying (formative) evaluation. These include, for example

• cognitive walkthrough,
• verification, e.g., against style guides, heuristics, and design patterns,
• usability test (participant observation) with prototypes of different quality (low, high fidelity),
• interviews,
• questionnaires.

For formative evaluation, it is advisable to

• use several of these methods and
• to do this early, iteratively, and repeatedly in the development process.

Almost cynically, the FDA notes that formative evaluations should be used to identify usability problems at an early stage because otherwise the summative evaluation (the usability test) could fail; this would then itself become a formative evaluation because subsequent improvements to the user-device interface would be necessary.

4. Summary and conclusion

Formative evaluation is both mandatory and useful for medical devices. It helps to identify and eliminate usability problems more quickly and with less effort. As a result, manufacturers save money and bring their devices to market faster. There is a high probability that risks due to a lack of usability will have been sufficiently minimized.

---

Change history

• 2023-02-13: References to the new FDA guidance document regarding the content of approval documents added