This part of the tool has been designed with the aim of supporting actors in need of evaluating their existing evidence.
Like the rest of the tool Evidence4MedTech, it has been developed based on an extensive mapping exercise of the Swedish HTA system. This included the review of published regional HTA reports in Sweden; as well as discussions, feedback and evaluation in collaboration with relevant experts and stakeholders.
We have formulated the essential guidelines that evidence (data) needs to comply with in order to be used to validate a medical technologies.
Evidence must be generated through one of the following designs
Randomised controlled trial
Cohort trial
Case series with sufficient sample size
During health technology assessment (HTA), evidence is usually graded, and selected, according to criteria based on the Cochrane pyramid.
Information on evidence grading and previously published HTA reports can be useful in guiding clinical study design.
How is Evidence Generated?
According to SBU's Method Book, evidence is graded according to the Cochrane pyramid, among others. This is also reflected in the inclusion and exclusion criteria for data collection during regional HTA. Commonly included study designs are systematic reviews, randomised controlled trials, non-randomised controlled studies, cohort studies and case series. Healthcare registry data has also been used more recently. In many cases the minimum number of study participants is also stated. This varies between five and 5,000 individuals, or a median of approximately 20 study participants per study or study arm depending on the study design. Figure. Left: The Cochrane pyramid for evidence grading. Right: Frequencies of included study designs during health technology assessment.
Reasons for Excluding Study Data
The figure shows the most common reasons for excluding study data during HTA. These include instances where the PICO (patient, intervention, control, outcome) does not correspond to the defined PICO in the overall aim of the HTA report. Other common reasons for exclusion of evidence include the wrong type of study design, wrong publication type, low study quality and too few study participants. The category 'other' mainly includes studies where the data of interest could not be extracted from the study for the purpose of comparing study groups or interventions, or meta-analysis. Figure. Most common reasons for excluding study data during HTA.
The Database
A database was collated of analysed HTA reports. The database contains summarised information from over 60 HTA reports published during 2016-2021. In the database, it is possible to search for summary information from reports. The table below shows an example of an extract of the collected data, with the report title, inclusion criteria, reasons for excluding study data, and link to reference. Table. Example of extracted information on excluded data from the HTA analysis database.
Reported study outcomes must include one of the following:
Clinical measurements of outcome(s)
Health economic measure(s)
Organisational measure(s)
Patient ethics measure(s)
If more than one study is included, outcomes across studies must be comparable.
Health technology assessment (HTA) is carried out based on defined clinical research questions, health economics, organisation and patient ethics.
Information on evidence grading and previously published HTA reports can be useful for ideation and verification during clinical study design in anticipation of HTA.
Clinical Research Questions
During HTA, medical technologies are evaluated based on a set of questions across the categories: clinical research question, health economics, organisation and patient ethics. The figure below shows the most common elements in the structure of clinical research questions for HTA. These often focus on treatment specific outcome measures, a technology or method, comparator, and relevant patient group. This may be of use when designing research questions in clinical studies. Figure. Prevalence of elements in defined clinical research questions for HTA.
Themes and Aspects Considered During HTA
The network map (developed using kumu.io) below gives a summary of themes and aspects considered during health economic evaluation, organisational assessment and patient ethical analysis during HTA. This based on thematised data from 66 HTA reports published during 2016-2021.
The themes and aspects considered during HTA are presented in the page HTA Aspects
The Database
A database was collated of analysed HTA reports. The database contains summarised information from over 60 HTA reports published during 2016-2021. In the database, it is possible to search for summary information from reports. The table below shows an example of an extract of the collected data, with the report title, inclusion criteria, reasons for excluding study data, and link to reference. Table. Example of extracted information on research questions from the HTA analysis database (note: translated from Swedish).
If more than one study is included, outcomes across studies must be comparable.
The PICO (Patient Group, Intervention, Control and Outcome) method is widely used for the design of systematic literature reviews.
PICO may be of use when reviewing your study data to ensure comparability between studies.
PICO
PICO is widely a used method to define Patient group or population, Intervention, Control group and Measurements of outcome for clinical research questions e.g., for health technology assessment (HTA). This is used to aid the selection of study data. The Figure below shows the most common reasons for excluding study data during HTA. PICO can be a useful method for the review of study data and to ensure that studies are comparable. Figure. Most common reasons for excluding study data during HTA.
The Database
The HTA analysis database contains summary information from 66 regional HTA reports published during 2016-2021. In the database, it is possible to search information on PICO. The table below illustrates an example of summary data on PICO. Example of extracted information on PICO from the HTA analysis database (note: translated from Swedish).
Measurements of outcome must include one of the following:
Intermediary measurements of outcome
Clinical measurements of outcome
Outcomes can be linked to health economics
If more than one study is included, outcomes across studies must be comparable.
Defining measurements of outcome is an important step in study design.
Here we provide resources to inform the ideation and design of measurements of outcome.
During HTA, measurements of outcome of interest are identified. These could be categorised as: critical or important for decision-making, intermediary outcomes, or less important for decision-making.
The measurements of outcome are typically highly specific to the treatment area and intervention. However, to exemplify, critical outcomes could include aspects, such as: survival, health-related quality of life, or serious adverse effects. Less important outcome measures could include, e.g., less severe adverse effects.
PICO
PICO is widely a used method to define Patient group or population, Intervention, Control group and Measurements of outcome for clinical research questions e.g., for health technology assessment (HTA). This is used to aid the selection of study data. The Figure below shows the most common reasons for excluding study data during HTA. PICO can be a useful method for the review of study data and to ensure that studies are comparable. Figure. Most common reasons for excluding study data during HTA.
The Database
The HTA analysis database contains summary information from over 60 regional HTA reports published during 2016-2021. In the database it is possible to retrieve information on assessed measurement of outcome in these HTA reports. An example is given in the table below. Example search result on measurements of outcome from the HTA analysis database (note: translated from Swedish).
