Usability Lab: Things to consider during selection

A usability lab is a facility where usability experts can simulate user environments to review the usability of products using usability tests. This involves asking test subjects (test participants) to complete tasks using these products.

In this article, you will learn

• what are the objectives of a usability lab
• what you should look for when choosing a usability lab
• what determines the costs of a usability lab and usability tests
• what role can the usability lab play

1. Three important objectives of usability labs

A usability lab should achieve the following objectives for medical device manufacturers:

• Create a legally compliant usability engineering file
• Identify usability problems and propose solutions to remedy them
• Compensate for capacity and competence bottlenecks

2. Criteria for selecting usability labs

Qualitative selection criteria can be used to assess whether a usability lab meets these objectives and quality standards.

Selection criterion	Test criteria
1. Legally compliant usability files Experience in conducting and documenting formative and summative assessments of medical devices and IVDs	Usability engineering file Profiles of the persons conducting the usability tests Detailed knowledge of IEC 62366-1 and FDA requirements
2. Representative usage environment An app may work in any office environment, but other products may require the equipment found in an operating room or other clinical setting.	Pictures, floor planPossibility to install the lab at the test subjects’ premises, if necessary
3. Recruitment of test subjects Ability to recruit representative participants.	Test subject databases of the Usability Lab Experience of recruiters “Screeners” for selecting test subjects
4. Internationality The FDA requires test subjects to be US residents.	The Usability Lab is able to conduct tests in the US and other countriesand recruit test subjects in these countries.
5. Organisation, duration and time management A usability lab requires extensive experience and standardized processes to achieve consistent quality and stay on schedule, even when the unexpected happens.	Number of employees. If labs are too small, the absence of one colleague can delay the project.Written documentation of organizational processes
6. Equipment, Technology Complete, reliable, and mastered technology and equipment are essential for smooth and fully documented usability tests.	Cameras, microphones, recording equipment Test room and observation room of sufficient size, separated by a one-way mirror Utilities (electricity, water, internet, gas if necessary) Equipment for connecting remote participants
7.Costs The detailed breakdown of all laboratory costs helps to keep track of the total costs of tests.	The more transparently the lab breaks down the cost to you, the better you can check whether everything has been taken into account and where there is potential for savings.
8. Professional competence The team’s competence determines its ability to design and conduct usability studies, identify usability problems, and develop solutions.	Profiles and experience of the usability team, which indicate experience … in conducting tests in dealing with participants before and during the tests in handling the laboratory technology and, if applicable, the product to be tested with interaction and UI design The following also provides information relevant publications education and training references

3. Factors that determine the costs of the usability lab

Numerous parameters influence the time and costs of a study in a usability lab:

1. Cost factor: Test subjects

• Number and type of user groups: Healthcare professionals (physicians, nurses, installation staff), patients
• Number of test subjects per user group
• Language(s) of the test subjects
• Do test subjects need to be recruited, or will the manufacturer provide them?
• How easy are they to recruit? Recruiting a group of 15 pediatric heart surgeons can be a challenge. Contact us to find out how this can be done..

2. Cost factor: Product

• Complexity: user training, time required for installation, setup, cleaning, etc.
• Size, shipping, and storage
• Necessity and duration of instruction and training. Does the manufacturer provide train-the-trainer?
• Duration of use
• Hazards posed by the product

3. Cost factor: planning, implementation, and documentation

• Objective of the study: Identify usability problems, develop suggestions for improvement, and comply with regulatory requirements
• Success factors: Completeness, correctness, duration, number of errors, comments, surveys
• Methods: Formative versus summative assessment, heuristic evaluation, surveys, interviews, cognitive walkthrough
• Materials for test subjects
• Other persons involved in the study, such as patients (e.g., children, unconscious persons, disabled persons), installation staff
• Number and scope of documentation (multiple target audiences, such as management, authorities, development), evaluations, statistics
• One or more iterations (e.g., after product improvements)

4. Cost factor: usage environment

• Usability Lab, Hospital, Simulator
• Other Products and Systems, Consumables
• Other People such as Installation Staff

5. Cost factor: technical equipment

• Number of microphones, cameras
• Recording
• Remote study versus onsite study in the field

6. Cost factor: Legal considerations

• Data protection requirements
• Special declarations of consent
• Compliance requirements (e.g. regarding payment of test subjects)

4. Role of the Usability Lab

The ideal case

A usability lab should be a service provider involved in the specification and review of the usability of medical devices. It can help to

• specify usable user interfaces
• identify risks posed by unusable user interfaces, and
• evaluate usability formatively and summatively, e.g., through inspections (verification) and usability tests (validation) to detect possible use errors that could lead to hazards for the patient or user.

The reality: Whenever the Usability Lab is needed

In our consulting practice, we regularly encounter manufacturers who only “discover” the topic of usability once the product has already been partially or even completely developed. Then, they must catch up on the missing activities and retrospectively create the corresponding documentation. Often, there is not enough time to optimally revise the tested user interface or to define measures to avoid or at least reduce the probability of use errors by the IEC 62366-1 standard.

However, late product reviews in a usability lab are still better than continuing to market products whose usability risks have not been systematically determined and controlled.

---

Change history

• 2024-04-01: Criteria for selecting usability labs updated, objectives of the labs added, editorial changes