Software testing is part of software verification alongside code reviews. Laws and standards place requirements on the type and scope of software testing.
Content
This page provides a quick overview of software testing and links to articles for further information.
The objective of testing software is to find errors in the software. Software testing is, therefore, part of analytical quality assurance (see Fig. 1).
Fig. 1: Software tests are part of analytical quality assurance
It is neither possible nor permissible to achieve software quality with software tests alone.
Constructive quality assurance includes
At the meta-level, there are laws and standards as well as training and further training programs that relate to the three aspects mentioned above, among others.
Analytical quality assurance attempts to find defects in various test objects using various processes, methods, and tools. A distinction is made between:
Software tests can be sorted according to many criteria (see Fig. 2).
Fig. 2: Taxonomy of methods for software testing
IEC 62304 divides its requirements according to the life-cycle phases (see Fig. 3).
The MDR and IVDR require state-of-the-art software life-cycle processes (see MDR, Annex I, Section 17.2) and documentation of the “verification and validation of the software” (see Annex II, Section 6.1.b).
The harmonized standard IEC 62304 for these EU regulations requires software testing in the following chapters:
During software release (Chapter 5.8), manufacturers must review whether the software tests have been carried out as described in the software development plan.
Fig. 3: In Chapters 5.5 to 5.7, IEC 62304 requires verification of the software using software tests, among other things.
The standard allows exceptions for legacy software.
Software validation does not fall within the scope of IEC 62304 but rather that of IEC 82304-1, although its requirements are very unspecific.
Essential methods for validating software are clinical evaluation and usability validation, typically as a summative evaluation or a usability test.
The term software validation is understood differently in different contexts. For example, Computerized Systems Validation does not only include validation at the end of development, as shown in Fig. 3.
This validation page helps to avoid misunderstandings.
The FDA acknowledges IEC 62304 as a “recognized standard”. However, the authority formulates the most relevant requirements for the software architecture in its guideline General Principles of Software Validation.
The guideline Content of the premarket submission specifies which documents manufacturers must submit. In it, the FDA also sets out requirements for software testing.
Manufacturers that comply with FDA and EU requirements for software testing are also likely to comply with requirements in other markets largely.
Sometimes, it helps to consider other standards. These include:
Developers understand unit testing differently from auditors: For the former, unit tests are the tests of granular units, e.g., methods and classes. In contrast, from a regulatory perspective, unit tests must test the software units that can represent larger components.
Read more in this article.
Depending on the test objective, manufacturers should choose the appropriate methods. A fuzz test has a different objective than a penetration test.
Benefit from black-box test procedures. This allows you to systematically determine the test cases with the highest probability of finding errors, which is not only applicable to software system tests.
Tests can only find errors if the test code runs through the software to be tested. Tools can be used to determine the code coverage automatically. Manufacturers should define this metric for themselves.
Each additional test increases the probability of errors being found and eliminated in time. This is why it is a good idea to repeat all tests as regression tests after changes have been made. Not only during development but also during software maintenance.
If manufacturers allow users to customize the software (for example, by parameterizing the software or writing their own scripts), then these customizations should also be reviewed through software tests.
Beta tests aim to make a product (in this case, software) available to a selected group of users before the official release to obtain feedback on the device.
Medical device manufacturers must be aware that such a beta test may already constitute illegal placing the device on the market.
Software testing is a competence that manufacturers must define and guarantee. ISO 13485, among others, obliges them to do this specifically for each development project.
Benefit from the support of the Johner Institute:
Contact us right away so that we can discuss the next steps. This will ensure that the “approval” is a success and that your software or device is quickly launched on the market.
In software development, unit testing refers to the testing of software units. However, there is no common understanding of This article provides clarity. 1. Test objects: What is tested during unit testing a) Usual understanding of software units Unit testing involves testing delimited parts of a software system. Depending on the size of these parts,…
DetailsThe incorporation of AI in medical devices has made great strides, for example, in the diagnosis of disease. Manufacturers of devices with machine learning face the challenge of having to demonstrate compliance of their devices with the regulations. Even if you know the law – what are the standards and best practices to consider in…
DetailsWhat is the difference between verification and validation, and how are these terms defined? Even standards and regulations use the terms incorrectly or misleadingly. This article 1. Verification a) Definition This definition does not explain what type of “requirements” need to be confirmed by verification. Limiting these requirements to product or component requirements is recommended to avoid…
DetailsSoftware maintenance is the phase in which software is further developed, e.g., with the objective of According to the FDA, 79% of all bugs occur during software maintenance. Accordingly, some regulations address this topic. Update: Addition to software maintenance during development Regulatory requirements for software maintenance Requirements of the Medical Device Regulation MDR (2017/745) The…
DetailsMDCG published guideline MDCG 2023-4 in October 2023 entitled “Medical Device Software (MDSW) – Hardware combinations – Guidance on MDSW intended to work in combination with hardware or hardware components.” Criticism 1: Unclear scope of MDCG 2023-4 The title gives the impression that (all) types of combinations of hardware and Medical Device Software (MDSW) are…
DetailsThe TIR45 (“Guidance on the use of AGILE practices in the development of medical device software”) is a Technical Information Report (hence TIR) of AAMI, the Association for the Advancement of Medical Instrumentation. First published in 2012, TIR45 has one primary objective: To guide medical device manufacturers on developing software compliant with FDA requirements while…
DetailsMedical software manufacturers must meet the legal requirements for software components in order to “approve” their devices. This article presents these requirements and gives seven tips on how to fulfill them quickly and easily. 1. What software components / items are There are different definitions of “software component”, also referred to as software items as…
DetailsBoth IEC 62304 and the FDA require integration tests. 1. What are integration tests? Integration tests verify whether software components that use interfaces of other components, i. e. call its methods, properly handle their returns (return values, exceptions). 2. Why are integration tests necessary? a) Improve the quality of the software Integration tests, like all…
DetailsWe frequently get asked, “Do you also offer Computerized Systems Validation?” One of the reasons for the interest is certainly: Authorities and notified bodies increasingly address the Computerized Systems Validation (CSV) in audits. This article introduces regulatory requirements regarding “Computerized Systems Validation” and provides guidance on how you can best meet these requirements. Computerized Systems…
DetailsMore and more manufacturers are using machine learning libraries, such as scikit-learn, Tensorflow, and Keras, in their devices as a way to accelerate their research and development projects. However, not all manufacturers are fully aware of the regulatory requirements that they have to demonstrate compliance with when using machine learning libraries or how best to…
Details