In a session at the International Headache Congress 2023, Professor Pozo-Rosich, as Honorary Secretary of the International Headache Society, chaired the debate titled ‘Does real-world evidence (RWE) have a role in headache medicine.’ On the ‘Pro’ side was Professor Paolo Martelletti, Editor-in-Chief of The Journal of Headache and Pain, and Professor Marta Torres-Ferrús, from Vall d’Hebron University Hospital, Barcelona, Spain, representing an editor’s and a clinician’s point-of view, respectively. They argued that RWE can complement evidence from randomised controlled trials (RCT) and even be included in drug authorisation applications. RWEs typically have a more heterogenous population than RCTs. This necessitates careful planning to define endpoints similar to those used in an RCT and proper analysis of results with consideration of bias due to confounding effects. On the ‘Con’ side was Dr Amy Gelfand, as Editor-in-Chief of the journal Headache, and Professor Dimos Mitsikostas, as a clinician from the National and Kapodistrian University of Athens, Greece. They argued that many RWE studies miss data, do not have proper control measures or standardised reporting, and fail to report co-interventions. RWE studies also often do not assess placebo responses and don’t take into account such factors as a patient’s treatment expectations and personality traits. These can confound a RWE study too much for it to be valid.
Editors View: Pro
Professor Martelletti started his argument by discussing how randomised clinical trials (RCTs) may be seen as the gold standard for evidence-based medicine but that findings of false or ‘fatally flawed’ data in some RCTs means such studies may need backing from other types of data.1 For example, while in 2019 results from a single RCT investigating a treatment for episodic cluster headaches was used for approval by the US Food & Drugs Administration,2 in 2020, the European Medicines Agency (EMA) found this same trial lacking in enough evidence for them to approve the same medicine following analysis of individual participant data.3
Real-world evidence studies can complement data from randomised controlled trials
In cases such as the above, Professor Martelletti discussed, “real-world evidence (RWE) can complement evidence from other sources including RCTs.” Accordingly, the EMA recently published a review concerning such studies and how they are utilised in regulatory decision making.4 “RWE can support both pre-authorisation and post-approval assessment.” Professor Martelletti continued “however, more effort is needed to better anticipate the need for such studies.” He concluded that, given that most RCTs are robust, RWE that originated from registration studies and those caried out in primary care could support such trials.
Editors View: Con
Dr Gelfand provided 10 common problems that, in her experience, could be wrong with RWE studies. These included:
- Non-standard outcome measures and timing of outcome assessments
- Lack of: control group; quality control on recording outcomes; systematic collection of adverse events
- Missing data, misclassification, and limited measures to assess treatment adherence
- Failure to register the analysis plan prior to study start; failure to report or assess co-interventions
Common problems with real-world evidence studies include lack of quality control, systematic data collection, and a clear analysis plan
With regard to what she considered the two most important points, missing data or non-standard outcome measures and timing, Dr Gelfand described how this includes information not being collected, or being collected at non-standardised timepoints following intervention; non-standardised inputting of information, and loss to follow-up. In many RWE studies, explained Dr Gelfand, “it’s simply what the patient reported to the clinician and what they wrote down. It’s an utterly subjective outcome.” She concluded that “RWE has simply too many problems to make it useful in headache medicine.”
Clinician's View: Pro
While RCTs are thought to provide the highest level of evidence due to standardisation of methodology and outcomes, RWEs have lower credibility due to confounding factors and control issues.5 In headache research, Professor Torres-Ferrús explained, such confounding factors can include comorbidities; medication; psychological and sociodemographic differences; and a patient’s health system. If these are not controllable in an RCT, the patient will be excluded. In RWE though, these patients can be included and confounding factors examined as to how they affect outcomes. This should happen with caution however, and only following adjustment for bias using methodology developed by a statistician with knowledge in the field being studied.6
Real-world evidence studies often include a more heterogenous population than randomised controlled trials
Another criticism of RWE studies is regarding data quality. This can be addressed by having prospective studies with well-defined protocols said Professor Torres-Ferrús.5 In studies where data collected is not that originally intended for research, such as medication orders and hospital records,7 “it is imperative,” he explained, “that we pre-define clinically significant endpoints that are measurable, accurate and suitable for research purposes.” Endpoints for RWE studies can be adapted from those for RCTs, for example, using patient reported outcomes and standardised diaries to assess the onset of a drug’s effect on symptom occurrence and severity.7
Clinician's View: Con
“Migraine is a variable intensity condition,” said Professor Mitsikostas, “so you are not absolutely sure that the good outcome you are getting is due to the medication.” Evidence-based medicine should, he said, come from the best available current research. In which case, he argued, “RWE is not the best available research because placebo responses are not evaluated.” Such a response is not just an affect attributed to placebo mechanisms, including expectation and conditioning, but also to the regression of a medical condition to the mean in some patients.8
Placebo responses are often not evaluated in real-world evidence studies, making them invalid
While there are some modifiable placebo/nocebo factors, such as a patient’s expectations and cognitive traits, drug packaging, and verbal/nonverbal suggestions, there are also non-modifiable ones, such as the patient’s previous experiences and their personality, culture, gender, education, and use of social media.8 “Because life is much more complicated, the help RWE gives us is small,” concluded Professor Mitsikostas.
Conclusion
At the end of the debate, Professor Patricia Pozo-Rosich concluded that “I hope that we will be able to work as a society to be better at collecting RWE.” She added that “I think it makes you a better physician if you are aware of these things when practicing medicine.”
Our correspondent’s highlights from the symposium are meant as a fair representation of the scientific content presented. The views and opinions expressed on this page do not necessarily reflect those of Lundbeck.
