Unit testing and IEC 62304

In software development, unit testing refers to the testing of software units. However, there is no common understanding of

• what a software unit is,
• what characterizes a good unit test, and
• which regulatory/legal requirements must be met.

This article provides clarity.

1. Test objects: What is tested during unit testing

a) Usual understanding of software units

Unit testing involves testing delimited parts of a software system. Depending on the size of these parts, these unit tests are also referred to as

• component tests,
• module tests, or
• class tests.

This means that the units correspond to components, modules, or classes.

Regardless of their type or size, these “units” must provide interfaces such as the

• methods of a Java class,
• REST API of a micro service,
• API of a C++ library, or
• Java Script API of a web assembly.

b) IEC 62304’s special understanding of unit (tests)

The IEC 62304 standard is relevant for the development of medical devices and IVD medical devices. It also recognizes software units. It defines these software units as:

This means that the software units’ granularity depends on the software system’s decomposition in the software architecture (see Fig. 1).

Fig. 1 marks the software units  (in the sense of the standard) as red components. On the other hand, developers usually understand units (in unit testing) to be the components at the lowest level (shown in white).

2. Moment of testing: When unit testing takes place

Manufacturers should implement and carry out unit tests during, or at the latest immediately after, the development of each software unit. Only once the individual software units have been successfully tested should they be integrated piece by piece and subjected to integration tests (see Fig. 2).

Only at the end is the complete software system tested (e.g., subjected to load tests) before validation (e.g., usability tests) takes place.

3. Test objectives: What unit testing can test

Unit testing is intended to contribute to “good software”. ISO 25010 specifies the software quality characteristics (quality objectives).

Quality objectives according to ISO 25010	Contribution of unit tests to achieving the quality objectives
Functionality	Unit tests can check functional correctness (e.g., whether a formula calculates the correct output). Unit tests only determine whether software covers the functional requirements if these functions have been defined and implemented.
Robustness	Unit tests can review whether software units are stable against incorrect entries.
Usability	Usability is not usually a test objective in unit testing.
IT Security	Unit tests can check selected aspects of IT security, such as the correct encryption and decryption of content.
Performance	Unit tests should also identify performance problems (long calculation times, high resource consumption) at the software unit level. This is because these problems are more difficult to locate in extensive software systems.
Maintainability	The fact that the components (software units) can be tested individually and independently in unit tests already speaks for the weak coupling of the components (units) and thus for the maintainability of the software system.
Portability	In most cases, the unit tests do not test  portability. Manufacturers usually test the portability to other hardware and software runtime environments with the entire software system.
Interoperability	Interoperability refers to the interaction between two or more software systems or components. Therefore, interoperability is usually the test objective of integration and system tests.

4. Test methods: What characterizes good unit tests

Good unit testing is characterized by the fact that

• (almost) all errors (i.e., deviations from the quality objectives) are found,
• these errors are found as early as possible, and
• the effort required for testing is as low as possible.

A good test methodology helps to achieve these objectives (Tab. 2).

Objective	Methods for achieving the objective (examples)
Find all errors	The developers should systematically derive the test cases. Black box test methods such as equivalence class and boundary value-based testing are available for this purpose. In addition, the development team should benefit from their knowledge of the source code when defining the test cases. This includes determining and achieving a minimum level of code coverage.
Find errors as early as possible	Errors are found particularly early on if the unit tests are written in parallel with the code to be tested. This leads to the rule that the corresponding unit tests must always be developed for each software unit, software module, or software item, and the automated and regular execution of unit tests as regression tests helps to find and eliminate errors introduced by changes to the code as quickly as possible.
Minimize the effort for testing	There are software tools that not only generate skeletons of test methods but even simple implementations of unit tests. In model-based development, e.g., with Domain Specific Languages, this automatic generation can be taken particularly far, as automation saves the repeated manual invoking of unit tests and the writing of error logs.

5. Regulatory requirements for unit testing

a) Requirements of IEC 62304

IEC 62304 does not recognize “unit testing.” Instead, it expects an inspection to determine whether the software units meet predefined acceptance criteria.

Depending on the safety class (more on this here (German))

• manufacturers can even dispense with this verification (class A) or
• determine the acceptance criteria in such a way that they can only be verified with an analysis of the source code (e.g., code review, static code analysis) (class B),
• or they must, in fact, test the software unit, i.e., carry out unit testing.

In either case, manufacturers are obliged to plan, specify, and document these inspections.

b) FDA requirements for unit tests

The FDA formulates its ideas on unit testing in the guidance documents, e.g., in the “Software Validation Guidance.” It calls these tests “Tests for Modules and Functions,” which must be traceable to the source code on the one hand and back to the architecture on the other.

The FDA formulates the requirements for “Verification and Validation” in similarly vague terms in the guidance document “Content of the Premarket Submissions for Software contained in Medical Devices.”

Given such imprecise requirements, it is all the more surprising that 100% branch coverage “is considered to be a minimum level of coverage for most software products, but decision coverage alone is insufficient for high-integrity applications.” The software experts at the Johner Institute have not yet found this requirement to be met by any company.

6. Tips for unit testing

Tip 1: Choose the right granularity

Avoid the typical trap mentioned above of performing countless unit tests but not explicitly testing the software units. However, this requires that you have isolated testable software units in the first place. And this, in turn, is a question of the quality of your architecture.

So, supplement your unit tests (according to your definition) with the tests of the probably more coarse-grained software units.

Tip 2: In the worst case, use acceptance criteria instead of unit testing

If you cannot (automatically) test your software units due to time constraints or messed-up software architecture, then at least determine formal acceptance criteria for class A and B software units, such as coding guidelines, metrics, or naming conventions. Although this has little to do with professional software engineering, it at least fulfills the few requirements of IEC 62304. This trick is not available to you for class C software.

With class C, you must also check performance aspects, error handling, memory utilization, etc., as part of the acceptance criteria (where applicable). In general, the acceptance criteria of unit tests should be the requirements specified in the software architecture.

Tip 3: Use tools for unit testing

Automate these tests so that you can repeat them as regression tests (ideally daily).

This requires that you have unit test tools such as JUnit, NUnit, or CppUnit in use, as well as tools for continuous integration such as combinations of ANT, make, Maven, Grunt in conjunction with continuous integration tools such as Jenkins/Hudson.

There are also numerous tools for static code analysis, e.g., analysis of code metrics, as well as for determining code coverage.

Version management systems such as git are helpful for managing unit test code and test results.

7. Conclusion, summary

The requirements of IEC 62304 for unit testing are straightforward and tend to be below what is considered the state of the art.

Unit tests are the “insurance policy” for every software manufacturer. Thanks to many tools, they can be implemented and executed quickly. However, they are no substitute for methodical knowledge of how to write unit tests.

---

Change history

• 2024-08-14: Article completely rewritten
• 2016-11-16: First revised version of the article published