Black box testing is when test cases are derived solely from the specification of the object to be tested (product, component). White box testing, on the other hand, derives the test cases from the internal structure of the object, e.g., from its source code or software architecture.
Unfortunately, many medical device manufacturers neither specify the test cases nor derive them systematically with black box test methods. Instead, a tester simply clicks through the application. This is neither effective (errors are only found with an insufficient probability) nor efficient. This regularly becomes a problem in the audit.
Black box testing
Objectives of black box tests
Tests cannot prove correctness (of software in this case), i.e. the absence of errors. They can only confirm the existence of errors.
The objective of black box test methods is, therefore, to specify test cases that have a particularly high probability of finding errors. Which test cases would you use to test the following screen template?
Black box test methods help precisely in this respect to quickly derive the test cases that have a high probability of detecting faults.
Overview of black box test methods
There are several black box test methods, for example
- Equivalence class-based testing
- Boundary-based testing
- Error-based testing
- State-based testing
- Testing with decision tables
- etc.
Example of a black box test procedure: Equivalence class-based tests
In equivalence class-based testing, one considers value ranges in which the product or software behaves equivalently. For example, the following value ranges could be distinguished for the date of birth:
- Valid date of birth, e.g., in the interval from today to 120 years ago.
- Invalid date of birth because it is in the future.
- Invalid date of birth because it is more than 120 years ago (we assume not to have a patient older than that).
What valid and invalid birth dates are should be inferred from the specification. Unfortunately, manufacturers fail to provide sufficiently precise specifications.
Specify your tests while you are still developing the specification of the product. Thus, black box test methods help you achieve precise and indisputable specifications.
Once you have identified the equivalence classes, select one representative of each for the black box test. In the above example, this would be the following birth dates:
- March 27, 1966 (representative of 1st equivalence class)
- December 18, 2045 (representative of 2nd equivalence class)
- February 03, 1843 (representative of 3rd equivalence class)
Black box testing: Other methods
Competent software testers are capable of selecting and combining the appropriate black box test methods depending on, among other things
- on the type of object to be tested,
- on time available,
- on the risk of the product, and
- on the safety class of the software or software item.
Regulatory requirements for black box testing
IEC 62304: Development plan
IEC 62304 does not require a specific test method. However, it does require a development plan (IEC 62304 Chapter 5.1), which must also specify how the software will be tested. I.e., medical device manufacturers must specify the test method(s).
IEC 62304: Integration and system tests
IEC 62304 requires manufacturers to specify and perform software integration tests (IEC 62304 Chapter 5.6) and software system tests (IEC 62304 Chapter 5.7) for safety class B and C software. Since the software system tests must verify that the software requirements (IEC 62304 Chapter 5.2) are met, and since software requirements should be specified as the software’s black box behavior, software should be tested by black box tests.
FDA requirement for software testing
The FDA requirements do not go beyond those of IEC 62304 with regard to (black box) testing.
Requirements of ISO 13485
Requirements for black box testing also arise at least indirectly from ISO 13485: The standard requires adequate training and competence of the team. Dedicated and trained software testers are indispensable for professional software design and development. This cannot be another task of the developers.
