Both IEC 62304 and the FDA require integration tests.
1. What are integration tests?
In software development, integration tests are a sequence of individual tests with the objective of finding errors in the interaction of different, interdependent (and already tested) software components of a software system.
Source: Johner Institute, based on Wikipedia
Integration tests verify whether software components that use interfaces of other components, i. e. call its methods, properly handle their returns (return values, exceptions).
2. Why are integration tests necessary?
a) Improve the quality of the software
Integration tests, like all software tests, are intended to improve the quality of the software by finding errors. To do this, they check whether the interfaces of two or more software components (also called software modules or software units) fit together.
These integration tests are necessary, although professional software developers have already tested the public methods of all components. Because component, module, or unit tests are not sufficient in order to find errors in the interaction of these components. This shows the following example very drastically:
One spacecraft crashed even though all modules were fully tested. However, one module (component) used SI units and another IU (Imperial Units). That the two interfaces did not fit could have found integration tests.
b) Comply with regulatory requirements
Many standards require integration testing and set requirements for it. For example, IEC 62304 requires integration testing and a verification whether the selected integration test method is suitable.
IEC 62304 requirements for integration tests
IEC 62304 sets the following requirements for integration testing:
- The existence of an integration plan
- Integration testing according to the plan
- Documentation of the integration tests
- The initiation of the problem-solving process if errors are found.
Many manufacturers do not meet the requirement of the problem resolution process. Instead, they modify the software until it behaves according to specifications. This corresponds neither to the requirements nor to the idea of IEC 62304.
Requirements of IEC 62304 for the evaluation of the integration test procedure
In addition, manufacturers must evaluate and ensure that the chosen integration test method is appropriate.
The manufacturer must evaluate the adequacy of the integration test procedure.
IEC 62304, Chapter 5.6.5
Since the 2015 version of IEC 62304, Section 5.6.5 has been worded somewhat more comprehensibly, as the term “verify” has been replaced by “evaluate” and the term “correctness” by “adequacy”.
It is, therefore, not a matter of reviewing the correct application of the test procedure but rather the suitability of the procedure.
What the authors of IEC 62304 mean by evaluation can also be derived from Annex B.5.6:
- The integration strategy is suitable (see below).
- The manufacturer has determined the correct and necessary test objectives, e.g., functional correctness, timing, and maintainability (for example, in terms of weak coupling).
- The test procedure is suitable; this includes the methods to specify the test cases (see below).
- The manufacturer has selected suitable tools, e.g., for creating stubs/mock objects to replace hardware and software components that are not (yet) available or are to be tested.
This evaluation of the test procedure is usually done by a review of the integration plan.
3. How to perform integration tests?
Step 1: Determine the objectives of the integration tests
The first step is to define the objectives of the integration tests. The taxonomy of ISO 25010 provides a good checklist for the completeness of these objectives.
The selection and prioritization of these objectives should be risk-based. For example, if the timing of the software is critical, it should be evaluated during integration testing.
Step 2: Determine the integration strategy
Determining the integration strategy is part of the test plan. Manufacturers should already determine the integration strategy as part of the software architecture. This determines the order in which the components are to be integrated. Typical representatives of integration strategies are:
- Bottom-up strategy, which is popular with developers and requires many test- drivers
- Top-down strategy, which allows an early impression of the final product and requires many mock objects
- Big-bang strategy, which does not actually mean piecemeal integration at all but rather testing the entire system
In the bottom-up strategy, testers first test the hardware or operating system-related layers and then the higher layers (e.g., UI). This means that in the course of integration testing, they add the next higher layer each time, reviewing whether the test objectives are met.
In the Medical Device University, you will learn more integration strategies and how to meet the requirements of IEC 62304 (not only) for integration testing.
Step 3: Specify test cases
The specification of test cases is part of the test plan. In order to specify the test cases and, thus, the test data, manufacturers should use methods such as the black box methods. These help to systematically determine the test cases so that they find defects with a particularly high probability. This is what distinguishes good tests from bad ones.
The specification of the test cases also includes determining how the tests are to be carried out, documented, and their outputs evaluated.
Read more about blackbox testing techniques, such as equivalence class-based testing, and about software testing in general.
Step 4: Perform tests and log outputs
In order to perform the tests, the testers must develop mock objects or/and test drivers depending on the selected test strategy. With this, they perform the tests as intended in the test plan.
The documentation also includes an evaluation of the test result. In the event of an error, this documentation serves as input for the problem-resolution process.
4. What should you pay attention to?
a) Selection of the persons involved
In order to specify and perform integration tests, both programming experience and competence in software testing are required.
Software developers who have undergone further training to become software testers have this competence.
b) Correct timing
Manufacturers should not start software integration testing only when programming is complete. Instead, it is advisable to test the interaction of the available components following the integration plan from an early stage. In this way, errors can be found earlier and eliminated more easily.
During the maintenance phase, manufacturers should also perform the integration tests as regression tests. This way, they ensure that the changes do not reintroduce bugs that have already been fixed.
c) Compliance with regulatory requirements
IEC 62304 requires manufacturers to review, as part of the software release, whether the software integration tests have been performed as specified in the design and development respectively integration test plan. This assumes that these plans exist.
5. Conclusion & summary
Integration tests are an important instrument of software quality assurance.
The Johner Institute recommends that these tests be performed as a separate step and not together with the software system tests.
The integration tests help prove that the system can be assembled piece by piece from the components. This, in turn, confirms the desired weak coupling of the components and, thus, of a good architecture.
An architecture that fails to meet the “weak coupling and strong cohesion” paradigm becomes an architectural debt for manufacturers that becomes increasingly difficult to resolve. The expenditure for this “debt reduction” is a multiple higher than that for consistently accomplished integration tests.
