Pros and Cons of real-word evidence in headache medicine: A debate

In a session at the International Headache Congress 2023, Professor Pozo-Rosich, as Honorary Secretary of the International Headache Society, chaired the debate titled ‘Does real-world evidence (RWE) have a role in headache medicine.’ On the ‘Pro’ side was Professor Paolo Martelletti, Editor-in-Chief of The Journal of Headache and Pain, and Professor Marta Torres-Ferrús, from Vall d’Hebron University Hospital, Barcelona, Spain, representing an editor’s and a clinician’s point-of view, respectively. They argued that RWE can complement evidence from randomised controlled trials (RCT) and even be included in drug authorisation applications. RWEs typically have a more heterogenous population than RCTs. This necessitates careful planning to define endpoints similar to those used in an RCT and proper analysis of results with consideration of bias due to confounding effects. On the ‘Con’ side was Dr Amy Gelfand, as Editor-in-Chief of the journal Headache, and Professor Dimos Mitsikostas, as a clinician from the National and Kapodistrian University of Athens, Greece. They argued that many RWE studies miss data, do not have proper control measures or standardised reporting, and fail to report co-interventions. RWE studies also often do not assess placebo responses and don’t take into account such factors as a patient’s treatment expectations and personality traits. These can confound a RWE study too much for it to be valid.

Editors View: Pro 

Professor Martelletti started his argument by discussing how randomised clinical trials (RCTs) may be seen as the gold standard for evidence-based medicine but that findings of false or ‘fatally flawed’ data in some RCTs means such studies may need backing from other types of data.For example, while in 2019 results from a single RCT investigating a treatment for episodic cluster headaches was used for approval by the US Food & Drugs Administration,2 in 2020, the European Medicines Agency (EMA) found this same trial lacking in enough evidence for them to approve the same medicine following analysis of individual participant data.3

Real-world evidence studies can complement data from randomised controlled trials

In cases such as the above, Professor Martelletti discussed, “real-world evidence (RWE) can complement evidence from other sources including RCTs.” Accordingly, the EMA recently published a review concerning such studies and how they are utilised in regulatory decision making.4 “RWE can support both pre-authorisation and post-approval assessment.” Professor Martelletti continued “however, more effort is needed to better anticipate the need for such studies.” He concluded that, given that most RCTs are robust, RWE that originated from registration studies and those caried out in primary care could support such trials.

 

Editors View: Con 

Dr Gelfand provided 10 common problems that, in her experience, could be wrong with RWE studies. These included:

  • Non-standard outcome measures and timing of outcome assessments
  • Lack of: control group; quality control on recording outcomes; systematic collection of adverse events
  • Missing data, misclassification, and limited measures to assess treatment adherence
  • Failure to register the analysis plan prior to study start; failure to report or assess co-interventions

Common problems with real-world evidence studies include lack of quality control, systematic data collection, and a clear analysis plan

With regard to what she considered the two most important points, missing data or non-standard outcome measures and timing, Dr Gelfand described how this includes information not being collected, or being collected at non-standardised timepoints following intervention; non-standardised inputting of information, and loss to follow-up. In many RWE studies, explained Dr Gelfand, “it’s simply what the patient reported to the clinician and what they wrote down. It’s an utterly subjective outcome.” She concluded that “RWE has simply too many problems to make it useful in headache medicine.”

 

Clinician's View: Pro 

While RCTs are thought to provide the highest level of evidence due to standardisation of methodology and outcomes, RWEs have lower credibility due to confounding factors and control issues.5 In headache research, Professor Torres-Ferrús explained, such confounding factors can include comorbidities; medication; psychological and sociodemographic differences; and a patient’s health system. If these are not controllable in an RCT, the patient will be excluded. In RWE though, these patients can be included and confounding factors examined as to how they affect outcomes. This should happen with caution however, and only following adjustment for bias using methodology developed by a statistician with knowledge in the field being studied.6

Real-world evidence studies often include a more heterogenous population than randomised controlled trials

Another criticism of RWE studies is regarding data quality. This can be addressed by having prospective studies with well-defined protocols said Professor Torres-Ferrús.5 In studies where data collected is not that originally intended for research, such as medication orders and hospital records,7 “it is imperative,” he explained, “that we pre-define clinically significant endpoints that are measurable, accurate and suitable for research purposes.” Endpoints for RWE studies can be adapted from those for RCTs, for example, using patient reported outcomes and standardised diaries to assess the onset of a drug’s effect on symptom occurrence and severity.7

 

Clinician's View: Con

“Migraine is a variable intensity condition,” said Professor Mitsikostas, “so you are not absolutely sure that the good outcome you are getting is due to the medication.” Evidence-based medicine should, he said, come from the best available current research. In which case, he argued, “RWE is not the best available research because placebo responses are not evaluated.” Such a response is not just an affect attributed to placebo mechanisms, including expectation and conditioning, but also to the regression of a medical condition to the mean in some patients.8

Placebo responses are often not evaluated in real-world evidence studies, making them invalid

While there are some modifiable placebo/nocebo factors, such as a patient’s expectations and cognitive traits, drug packaging, and verbal/nonverbal suggestions, there are also non-modifiable ones, such as the patient’s previous experiences and their personality, culture, gender, education, and use of social media.8 “Because life is much more complicated, the help RWE gives us is small,” concluded Professor Mitsikostas.

 

Conclusion

At the end of the debate, Professor Patricia Pozo-Rosich concluded that “I hope that we will be able to work as a society to be better at collecting RWE.” She added that “I think it makes you a better physician if you are aware of these things when practicing medicine.”

Our correspondent’s highlights from the symposium are meant as a fair representation of the scientific content presented. The views and opinions expressed on this page do not necessarily reflect those of Lundbeck.

References

  1. Carlisle JB. False individual patient data and zombie randomised controlled trials submitted to Anaesthesia. Anaesthesia 2021;76472−9.
  2. US Food & Drug Administration. FDA approaves first treatment for episodic cluster headache that reduces the frequency of attacks. Available at https://www.fda.gov/news-events/press-announcements/fda-approves-first-t.... Published 2019. Accessed: 25.Sept.2023.
  3. European Medicines Agency. Refusal of a change to the marketing authorisation for Emgality (galcanezumab) [press release]. 2020. Available at: https://www.ema.europa.eu/en/documents/smop/questions-answers-refusal-ch.... Accessed: 25.Sept.2023.
  4. European Medicines Agency. Real-world evidence framework to support EU regulatory decision-making. 2023. Available at: https://www.ema.europa.eu/en/documents/report/real-world-evidence-framew.... Accessed: 25.Sept.2023.
  5. Kim HS, et al. Real-world evidence versus randomized controlled trial: Clinical research based on electronic medical records. J Korean Med Sci 2018;33:e213.
  6. Liu F, Panagiotakos D. Real-world data: A brief review of the methods, applications, challenges and opportunities. BMC Med Res Methodol 2022;22:287.
  7. Swift B, et al. Innovation at the intersection of clinical trials and real-world data science to advance patient care. Clin Transl Sci 2018;11:450−60.
  8. Mitsikostas DD, et al. European Headache Federation recommendations for placebo and nocebo terminology. J Headache Pain 2020;21:117.