When notified bodies identify discrepancies in clinical evaluations, in many cases, these relate to the clinical endpoints.
It is, therefore, essential for medical device manufacturers to specify the clinical endpoints precisely and to demonstrate that they have actually been achieved.
This article explains what manufacturers should pay particular attention to.
1. Clinical endpoints: The basics
a) Background and objective
Manufacturers are required to demonstrate the safety, performance, and clinical benefit of their medical devices based on clinical data. To this end, manufacturers must specify the clinically relevant outcome parameters (“endpoints”) with which they consider this evidence to be provided.
If a manufacturer claims that its medical device is capable of reducing pain, then he can quantify the pain reduction with the VAS score. If the clinical data shows that the score actually decreases by the claimed value, then the clinical benefit is proven.
b) Definitions and synonyms
Endpoints
The German Federal Institute for Drugs and Medical Devices defines (clinical) endpoints as follows:
Endpoints are the target variables of the study. All endpoints must be determined before the start of the study and written down in the study protocol. It must be defined which parameter (WHAT), at which time (WHEN), with which method, which device or survey instrument, etc. (HOW) will be recorded.
Source: DIMDI
The definition of ISO 14155:2020:12 is similar:
Endpoint: <primary> major parameter used to provide evidence of clinical performance (3.11), effectiveness (3.20), or safety in a clinical investigation (3.8)
ISO 14155:2020:12, 3.22
Clinical endpoints are also referred to as clinically relevant (outcome) parameters.
Primary and secondary endpoints
A distinction is made between primary and secondary endpoints.
The primary endpoint is the main objective of the study, and the secondary endpoints are the secondary objective.
Source: DIMDI
The primary endpoint is also referred to as the “hard” endpoint because it must be objective and measurable.
Secondary endpoints are “softer criteria” used to characterize the treatment under study further. For example, data on quality of life, pain perception, further interventions, and medications can be used.
c) Importance of clinical endpoints in clinical investigations
The choice of endpoints and parameters is also critical in the planning of a study: study endpoints are those criteria used to measure the achievement of clinical research objectives, such as evidence of safety, performance, and patient-related benefit.
If safety of a device is to be demonstrated, the test plan must specify the metrics that will document and demonstrate safety. These could be, for example, the number of serious complications or the number and duration of specific deaths or hospital readmissions.
If safety of a device is to be demonstrated, the test plan must specify the metrics that will document and demonstrate safety. These could be, for example, the number of serious complications or the number and duration of specific deaths or hospital readmissions.
The term “parameter” in this context is to be understood as an umbrella term for the term “endpoint.” There are parameters that are not endpoints, e.g., material resistance, which would be determined as part of a bench test.
2. Regulatory requirements
All relevant regulatory requirements require manufacturers to define clinically relevant endpoints. However, regulations differ in their definitions of terms and the granularity of requirements.
a) MDR requirements
The determination of clinically relevant parameters is required by the MDR during the planning of the clinical evaluation:
It says:
MDR, ANNEX XIV, PART A – CLINICAL EVALUATION
- detailed description of intended clinical benefits to patients with relevant and specified clinical outcome parameters
- non-exhaustive list and specification of parameters for the determination, based on the latest medical knowledge, of the reasonableness of the benefit-risk balance for the various indications and intended purpose(s) of the device;
- The primary endpoint must be meaningful to the investigational device and should be clinically relevant.
Notified bodies review in their Clinical Evaluation Assessment Report (CEAR) whether a list of parameters is provided and whether the clinical benefit is evaluated using parameters.
Often the reasonableness of these parameters is not reviewed! Many auditors only review whether the manufacturers have defined the parameters. However, they do not question the meaningfulness of these parameters.
On the other hand, auditors insist that the choice of parameters MUST follow the state of the art. However, this does not make sense in some cases, as explained below (see surrogate endpoints).
The requirements can be found in, among other places
- MEDDEV, Chapter 8.2 (current knowledge/state of the art […] justify the validity of surrogate endpoints (if used))
- MDR, Annex XIV, Part A 1a, indent 6
- MDCG 2020-13, Section C
Many manufacturers are not aware that the MDR requirements for relevant concrete parameters for clinical output may be synonymous with trial endpoints. In terms of clinical investigations, the MDR states:
“The endpoints of the clinical investigation shall address the intended purpose, clinical benefits, performance and safety of the device. The endpoints shall be determined and assessed using scientifically valid methodologies. The primary endpoint shall be appropriate to the device and clinically relevant.”
ANNEX XV CLINICAL INVESTIGATIONS CHAPTER I GENERAL REQUIREMENTS
b) Requirements of the MDCG 2020-13 guideline
Notified bodies review parameters against MDCG 2020-13 guidance, which requires:
“Has the manufacturer adequately described an indicative list and specification of parameters used to determine, based on the state of the art in medicine, the acceptability of the benefit-risk ratio for the various indications and for the intended purpose or purposes of the device?“
MDCG 2020-13
The fact that manufacturers do not adequately specify these parameters is one of the most frequent points in defect reports.
c) Requirements of MEDDEV 2.7/1 rev. 4
MEDDEV 2.7/1 rev.4 also requires that manufacturers define endpoints:
c. Quantification of benefit(s) to the patients
MEDDEV 2.7/1 rev.4 A7.2.
- Defining specified endpoints is indispensable for setting up clinical investigations and properly performing the identification, appraisal, and analysis of the clinical data.
- Benefit(s) are often evaluated along a scale or according to specific endpoints or criteria (types of benefits), or by evaluating whether a pre-identified health threshold was achieved.
- The change in subjects’ condition or clinical management as measured on that scale, or as determined by an improvement or worsening of the endpoint, determines the magnitude of
- the benefit(s) in subjects. Variation in the magnitude of the benefit across a population may also be considered.
- The clinical relevance of these changes should be discussed and justified.
- Ideally, these parameters should be directly clinically relevant.
- In certain cases benefits can be assumed when validated surrogate endpoints are met (such as obtaining certain results with laboratory tests or measurements of anatomical or physiological properties).
- Based on the current state of medical knowledge, the evaluators shall justify and document the clinical relevance of endpoints used for the clinical evaluation of a device and demonstrate the validity of all surrogate endpoints (if surrogate endpoints have been used).
However, MEDDEV 2.7/1 rev.4 does not provide a definition or a more specific description of what the guideline means by an endpoint.
d) Requirements of ISO 14155:2020:12
ISO 14155, the standard on clinical investigations, also requires specified endpoints:
(c) Primary and secondary endpoints with the rationale for their selection and measurement. Combined endpoints, if any, with the rationale for their choice and measurement. […]
ISO 14155:2020:12
3. Selection of the endpoints / parameters
a) Challenge in the selection of clinical endpoints
Challenge 1: Determine appropriate endpoints
The MDR only requires that parameters supporting clinical benefit must be clinically relevant and should be measurable:
The primary endpoint shall be appropriate to the device and clinically relevant.
ANNEX XV CLINICAL INVESTIGATIONS CHAPTER I GENERAL REQUIREMENTS
It is up to the manufacturer to decide which endpoints are considered reasonable.
Challenge 2: Conflicts of interest
Conflicts of interest may arise in the selection of endpoints.
| Role | Interest |
|---|---|
| Manufacturer | Get device to market as quickly as possible |
| Product manager | Explain as many benefits as possible |
| Responsible person for clinical evaluations | Demonstrate benefit to patients |
| Doctor | Improve a (e.g., surgical) method |
Challenge 3: Medical devices without intended medical purpose
There are medical devices that do not have an intended medical purpose or where clinical data does not make sense. Examples include accessories and some class I medical devices:
- phantom used to calibrate a CT scanner
- the workstation used to record examination results
- bedpan rinsing apparatus for cleaning bedpans
- turbines for dental drills
- control systems for dialysis fluid and dialysis water in dialysis centers
In these cases, no clinically relevant endpoints need to be collected, only technical parameters (performance parameters). This is why the MDR speaks of parameters and not endpoints.
b) Procedure for the selection of endpoints (general)
The manufacturer should identify the relevant concrete parameters that allow the clinical output to be assessed and their acceptance criteria from the state of the art using a systematic literature search.
The acceptance criteria set the limit values within which the endpoint is considered to be met. The manufacturers determine these from the literature, norms, and other standards.
| Example device | Example of promised benefit | Examples of parameters for the measurement of this benefit | Examples of acceptance criteria for the parameters |
|---|---|---|---|
| Cooling pack | Pain reduction | Pain reduction / pain intensity using VAS score | -3.5 points after 6 weeks or -30 % after 24 h |
| Cooling pack | Reduction of swelling | Time and extent to reduce the swelling | Reduction by 2 points (from 10 to 8) |
| Standalone software | Improvement of sleep quality | Objective sleep quality: efficiency Subjective sleep quality: Pittsburgh Sleep Quality Index | 80 % 8,0 |
Helpful for this activity is the article on systematic literature search.
c) Selection of clinical endpoints (parameters related to clinical benefit)
General
According to Article 2 of the MDR, the “clinical performance” of a device […] must meet the intended purpose stated by the manufacturer so that, when used as intended, a clinical benefit is achieved for patients according to the manufacturer’s claims.
The MDR defines what it means by clinical benefit:
Refers to the positive impact of a device on a person’s health as indicated by appropriate, measurable, and patient-relevant clinical outputs, including diagnostic results, or a positive impact on patient management or public health.
The clinical benefit must be considered individually for each device.
According to MDR Annex I, Chapter III, 23.4 (c), the clinical benefit must be stated by the manufacturer in the instructions for use.
Examples for parameters related to clinical benefit (clinical endpoints)
There are different classes of clinical benefits and, thus, clinical endpoints:
| Clinical benefit, clinical endpoint | Examples |
|---|---|
| Positive impact on clinical output | Reduced likelihood of adverse outputs, e.g., reduction in blood loss resulting in reduced mortality and morbidity; improvement in impaired body function; reduction in duration of surgery |
| Patient’s quality of life | Faster recovery (significant improvements, including simplification of care or improvement in the clinical management of patients, improvement in body function, reduction in symptoms) |
| Better diagnosis | Enabling correct diagnosis, faster diagnosis; identifying patients who are more likely to respond to a particular therapy |
| Therapeutic success and long-term effectiveness, public health implications | Capability of a medical diagnostic device to detect a specific disease and thus prevent its spread, to identify phases, stages, locations, severity or variants of the disease; prediction of future onset of disease |
While parameters should be appropriate and directly clinically relevant, in certain cases, a benefit can be assumed if validated surrogate endpoints are met (e.g., obtaining specific outputs from laboratory tests or measurements of anatomic or physiologic characteristics).
For drugs, one also chooses as classes for clinical endpoints:
- mortality
- morbidity
- health-related quality of life
- side effects
d) Selection of performance and safety parameters
Step 1: Decide for which parameters clinical data are necessary
Manufacturers must also determine safety and performance parameters, some of which are demonstrated through clinical investigation.
| Parameter type | To be proven with clinical data | Detectable with bench tests |
|---|---|---|
| Performance parameters | Quality of Life (QOL) Image quality (bench tests) Measurement accuracy of diagnostic products (bench tests) Sensitivity and specificity of tests (needs both) | Image quality Measurement accuracy of diagnostic products Sensitivity and specificity of tests (needs both) Material resistance Early defect detection (poka yoke) |
| Safety parameters | Mortality Malfunction of the device per year Number of adverse events per year | Usability Electrical safety Biocompatibility Cleanability Sterility Alarming |
Not all parameters that need to be captured in the clinical evaluation for the review of the safety and performance of a device are clinically relevant.
In addition, there are safety and performance parameters that are not linked to endpoints in a clinical investigation with patients but require other experimental studies. These are nonetheless important for providing evidence in clinical evaluation.
Step 2: Determine acceptance criteria
For parameters for which clinical data are required, the manufacturer must determine the acceptance criteria. As with the clinical parameters (endpoints), he uses the literature, norms, and other standards for this purpose.
| Parameter type | Device example | Example for parameter | Examples of acceptance criteria |
|---|---|---|---|
| Performance parameters | Cooling pack | Minimum achievable skin temperature | < 13,6 °C |
| Performance parameters | Cooling pack | Maintaining the temperature for a certain time | 20 minutes application time |
| Performance parameters | Standalone software | Measurement of sleep duration | 7 h |
| Safety parameters | Cooling pack | Rate of undesirable side effects such as cold burns | 1 % Cold burns |
| Safety parameters | Standalone software | Number of malfunctions | 0,5 % |
| Safety parameters | Implant | Infection rate | 2,5 % |
4. Common mistakes
Error 1: Non-specific or non-measurable endpoints
If clinically relevant outcome parameters are not specifically and measurably defined, it is not possible to design a study or provide evidence in a clinical evaluation that these parameters have been achieved.
| Parameter | Poor example | Good example |
|---|---|---|
| Pain | Eases pain | Reduces pain on the numerical rating scale (NRS) by 10% |
| Muscle building | Promotes muscle building | Improves the “mean peak toque” by 30% in a muscle strength measurement |
Error 2: Non-relevant endpoints
Problems with non-relevant surrogate endpoints
Instead of patient-relevant endpoints, so-called surrogate endpoints are often determined. These can usually be determined more quickly.
Surrogate endpoints are often physiological or biochemical measurements that are not of direct relevance to the patient. However, this often allows the number of cases to be reduced so that a smaller patient population can be selected and the costs of the studies can be reduced. However, the ethical justifiability is questionable.
Advantages of patient-relevant endpoints
Patient-relevant endpoints can be experienced directly by the patient. These include parameters such as mortality, morbidity, adverse events, and health-related quality of life.
| Device, Method | Surrogate endpoint | Patient-relevant endpoint |
|---|---|---|
| Operation method | Reduction of blood loss | Time to recovery or a quality of life |
| Implants | Hospitalization (length of stay of the patient) | Number of regression interventions |
| Autoinjector (e.g., for insulin) | Minimal system errors, misinjections | Adherence (adherence to therapy) |
Those responsible for clinical evaluation must critique endpoints and determine them from the state of the art.
Error 3: Endpoints in the wrong study environment
If the design of a study is not planned correctly, the power of the measured endpoints may be limited. Examples that affect the determination of endpoints include:
- Measurement at the wrong time or on the wrong patient population
- A too small number of cases
- Wrong statistics (method)
- Wrong control group (bias)
- Confounding factors (such as additional administration of analgesics, even though pain reduction is measured by the device)
5. Support
a) Determine clinical strategy
The Johner Institute’s clinical experts will help you determine your clinical strategy and formulate it into a Clinical Evaluation Plan (CEP). This includes:
- Determine the state of the art
- Derive clinically relevant parameters to derive clinical benefit
- Determine acceptance criteria
- Determine a method for demonstrating that these acceptance criteria are met
This will provide you with the necessary input for the next steps while meeting all regulatory requirements for the CEP.
b) Assist in study design and perform case number planning
The clinical investigation can be planned (“designed”) on the basis of the clinical strategy. The experts at the Johner Institutes will support you in this process. They also calculate the number of cases and ensure that the costs of the study remain as low as possible but that as many patients as necessary are included to prove the clinical benefit of the medical device.
This gives you a clear roadmap and allows you to estimate the duration and cost of clinical investigations. You avoid the worst-case scenario, namely that a notified body or authority rejects the outputs as insufficient after the study has ended.
Take advantage of the Johner Institute’s support to help you achieve accurate clinical evaluations quickly and in a targeted manner, setting the stage for the quick commercialization of your devices.
Feel free to contact us here to clarify the next steps for this.
6. Conclusion and summary
Determining the right parameters and endpoints is challenging. Therefore, it is not surprising that in the many inspections by notified bodies, the choice of these endpoints and their acceptance criteria leads to deficiency reports. This, in turn, delays or prevents the marketing of many medical devices.
Without the choice of relevant endpoints, no clinical investigation can be planned that is suitable for demonstrating the clinical benefit, safety, and performance of medical devices.
Therefore, as a manufacturer, you should plan the time necessary to do this and build or consult the necessary competencies. You owe this not only to yourself and the notified bodies but, above all, to your patients.
