Objective Evidence: Objective evidence in the audit

Laws and standards (e.g., FDA, ISO 13485) require “objective evidence.”

• How conclusive must this evidence be for authorities and notified bodies to be satisfied?
• How can manufacturers strike a balance between over-documentation and problems in the audit?
• This article provides answers to these questions.

1. Definitions of terms

The FDA uses objective evidence in the definitions of verification and validation in 21 CFR part 820 (the “Quality System Regulations”). Unfortunately, however, the FDA does not define the term itself.

IEC 60601-1 also uses the term “objective evidence”. It refers to the definition of the term in 2.10 of ISO 14971:2007. This standard relates, in turn, to the definition in ISO 9000.

Data should, therefore, provide this evidence. The MDR also expresses this in its definition of the term ‘clinical evidence’.

2. Regulatory requirements according to “Objective Evidence”

a) FDA

The FDA requires “objective evidence”, particularly in the context of the design history file and the verification and validation activities, e.g. in design validation. Process validation must also be carried out using “objective evidence”.

b) MDR / IVDR

The MDR and IVDR require proof of, among other things, the product’s conformity, e.g., compliance with the general safety and performance requirements.

However, the EU regulations do not translate the German term “Nachweis” as “evidence”, but as “demonstration”. The MDR speaks of “evidence” primarily in the context of “clinical evidence”.

The regulations do not specify the type of evidence required..

c) ISO 13485

The English version of ISO 13485 uses the term “evidence”, but without adding “objective”. This objectivity is probably assumed. The standard requires evidence of the following:

• conformity of the product
• effectiveness of the QM system
• commitment of top management

The standard, therefore, requires “evidence” but does not specify how this evidence is to be provided.

3. Demonstration of “objective evidence”

a) Definition of the problem

Many manufacturers of medical devices are unsure how granular this documentation needs to be.

What is the right level?

The objective evidence must be such that an auditor can use only the documents (a video also counts as a document here) to verify whether the tests in the above example were successful (or not).

The first level is not sufficient for this. The mere mention of “passed” does not allow one to understand whether the test was carried out. Even at the second level, traceability is limited. From the third level onwards, auditors and reviewers should have sufficient evidence.

b) Mind experiment: finding the right level of detail

There is no general answer to which of the four levels mentioned meets the requirement for “objective evidence.”” The following thought experiment may help in the search for an answer to this question:

What proof would you expect to be relatively sure that your colleague is doing his job correctly, that he is not cheating, or that he is inadvertently making mistakes? In this case, the documentation you would like to see provides “objective evidence.”

So, it is not about a person acting with malicious intent or falsifying data with a high degree of criminal energy.

c) Specific information

Necessity of screenshots

There is no legal requirement for screenshots, not even from the FDA. Inspectors want to be able to understand which verification and validation activities a manufacturer has carried out and with which results. A simple “Pass/Fail” is therefore not accepted on a regular basis.

Screenshots are one way of meeting these requirements, i.e. of providing this proof. It is not about documenting every mouse click (please understand this as a metaphor).

The example of the login is taken from a post on the topic of “objective evidence” by a former FDA employee.

System-Log

A system log can also help provide “objective evidence” depending on the test.

Videos

Sometimes videos are helpful, for example, if you want to document a dynamic so that “objective evidence” is provided. The disadvantages of videos should be noted:

1. A video only documents the execution and, if applicable, the results. However, it does not evaluate the result, i.e., whether the product requirements to be reviewed have been met. Following up on videos can be time-consuming.
2. Videos are very storage-intensive. This should be considered if manufacturers want to store the records in a version management system automatically.
3. The requirements for the automation of video recordings are high, which would be particularly helpful for regression tests.

Alternatives to screenshots and videos

Most companies resist screenshots and videos because both involve a lot of work. Therefore, professional companies automate the verification of their products in other ways:

• Automated UI tests check the UI. This is how the manufacturer ensures the correctness of outputs, positions, and the layout of user interfaces.
• A manufacturer builds test robots that simulate user actions, evaluate them, and document their reactions together with the system’s reactions. AI helps evaluate these reactions.

4.    Conclusion and summary

Opinions differ on how conclusive the evidence (the “objective evidence”) must be.

• The evidence should be documented in sufficient detail to enable an uninvolved person to understand what has been done and how the manufacturer justifies the product’s compliance with the requirements.
• The granularity of these records should be risk-based: in the case of high regulatory risks or risks to the health of patients, users, and third parties, the evidence should be complete. However, there is no legal requirement for screenshots or videos.
• It is helpful to provide the evidence automatically, just as it is helpful to automate the tests.

---

Change history

• 2024-11-06: Article extensively revised and restructured
• 2016-02-10: The first version of the article was published