In short Article

Level of evidence: randomised or observational studies?


Randomised controlled trials (RCT’s) are regarded as the gold standard in medical research to prove whether presumed effects of medicinal or other treatments are of a causal nature. There may however be financial, practical or ethical issues which make it undesirable or impossible to carry out an RCT for each and every relevant treatment. As a result, observational research has become a much-used alternative method to estimate the effects and adverse effects of, especially, drugs. This has been partly inspired by the ever growing availability of large quantities of data. From the point of view of the philosophy of science, observational research should only be used to generate hypotheses. Hence, in evaluating findings that are being presented, one always needs to be aware of methodological limitations such as ‘confounding by indication’, the ‘healthy cohort effect’ and the ‘immortal time bias’. One will always have to consider whether it is possible to correct sufficiently for these issues to enable the findings of observational studies to be regarded as valid in terms of establishing a causal effect or association.

  • Randomised double-blind trials are regarded as the gold standard in estimating a causal effect of a drug.
  • Randomised research is not possible for all research questions regarding the effect of therapies; it is therefore relevant to consider if, and under what conditions, observational research may be a valid alternative. 
  • The main scientific objection to using observational studies to estimate a causal effect of  a therapy is that the groups to be compared often differ. It is often not possible to adequately correct for these differences, leading to a biased result. 
  • For some research questions, the risk of confounding is limited, and in these cases observational research can be a good alternative to randomised research; this is particularly true for research into rare adverse effects of drugs. It is also relevant since the follow-up in randomised studies is often too short, and they include too few participants, to be able to detect rare adverse effects.

  1. Van Deventer KR, Janssens JEM. Klinische relevantie van onderzoeksuitkomsten. Gebu. 2019;53(1):1-11
  2. RVS Zonder context geen bewijs. Over de illusie van evidence-based practice in de zorg. Publicatie 17-05. RVS, Den Haag 2017. ISBN: 987-90-5732-2679.
  3. Fervenza FC, Appel GB, Barbour SJ, Rovin BH, Lafayette RA, Aslam N, et al. Rituximab or Cyclosporine in the Treatment of Membranous Nephropathy. The New England journal of medicine. 2019;381(1):36-46.
  4. Ruggenenti P, Remuzzi G. A First Step toward a New Approach to Treating Membranous Nephropathy. N Engl J Med. 2019; 381(1):86-88.
  5. Remuzzi G, Chiurchiu C, Abbate M, Brusegan V, Bontempelli M, Reggenenti P. Rituximab for idiopathic membranous nephropathy. Lancet. 2002; 360(9337):923-924.
  6. Senn S. Seven myths of randomisation in clinical trials. Statistics in medicine. 2013;32(9):1439-50.
  7. Mansournia MA, Higgins JP, Sterne JA, Hernan MA. Biases in Randomized Trials: A Conversation Between Trialists and Epidemiologists. Epidemiology (Cambridge, Mass). 2017;28(1):54-9.
  8. Dekkers OM, von Elm E, Algra A, Romijn JA, Vandenbroucke JP. How to assess the external validity of therapeutic trials: a conceptual approach. International journal of epidemiology. 2010;39(1):89-94.
  9. Lodi S, Phillips A, Lundgren J, Logan R, Sharma S, Cole SR, et al. Effect Estimates in Randomized Trials and Observational Studies: Comparing Apples With Apples. American journal of epidemiology. 2019;188(8):1569-77.
  10. Rossouw JE, Anderson GL, Prentice RL, LaCroix AZ, Kooperberg C, Stefanick ML, et al. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results From the Women's Health Initiative randomized controlled trial. JAMA. 2002; Jul 17;288(3):321-33.
  11. Mendelsohn ME, Karas RH. The Protective Effects of Estrogen on the Cardiovascular System. N Engl J Med. 1999; 340(23):1801-1811.
  12. Stampfer MJ, Colditz GA, Willett WC, Manson JE, Rosner B, Speizer FE, et al. Postmenopausal estrogen therapy and cardiovascular disease. Ten-year follow-up from the Nurses’ Health Study. N Engl J Med 1991;325(11):756-62.
  13. Grady D, Rubin SM, Petitti DB, Fox CS, Black D, Ettinger B, et al. Hormone therapy to prevent disease and prolong life in postmenopausal women. Ann Intern Med 1992;117:1016-37.
  14. Hernán MA, Alonso A, Logan R, Grodstein F, Michels KB, Willett WC, et al. Observational studies analyzed like randomized experiments: an application to postmenopausal hormone therapy and coronary heart disease. Epidemiology 2008;19(6):766–79.
  15. Vandenbroucke JP. The HRT controversy: observational studies and RCTs fall in line. Lancet. 2009;373(9671):1233-5.
  16. Rosén M, Axelsson S, Lindblom J. Observational studies versus RCTs: what about socioeconomic factors? Lancet. 2009;373(9680):2026.
  17. Vandenbroucke JP. When are observational studies as credible as randomised trials? Lancet (London, England). 2004;363(9422):1728-31.
  18. Papanikolaou PN, Christidi GD, Ioannidis JP. Comparison of evidence on harms of medical interventions in randomized and nonrandomized studies. CMAJ : Canadian Medical Association journal = journal de l'Association medicale canadienne. 2006;174(5):635-41.
  19. Psaty BM, Prentice RL. Minimizing bias in randomized trials: the importance of blinding. Jama. 2010;304(7):793-4.
  20. Dwan K, Gamble C, Williamson PR, Kirkham JJ. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PloS one. 2013;8(7):e66844.
  21. Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet (London, England). 2005;365(9453):82-93.
  22. Nissen SE, Wolski K. Effect of rosiglitazone on the risk of myocardial infarction and death from cardiovascular causes. The New England journal of medicine. 2007;356(24):2457-71.
  23. Suissa S. Immortal time bias in pharmaco-epidemiology. American journal of epidemiology. 2008;167(4):492-9.
  24. Via: https://www.ncbi.nlm.nih.gov/pubmed/16954361
  25. Rennemo E, Zatterstrom U, Boysen M. Impact of second primary tumors on survival in head and neck cancer: an analysis of 2,063 cases. The Laryngoscope. 2008;118(8):1350-6.
  26. Rochon PA, Tu JV, Anderson GM, Gurwitz JH, Clark JP, Lau P, et al. Rate of heart failure and 1-year survival for older people receiving low-dose beta-blocker therapy after myocardial infarction. Lancet (London, England). 2000;356(9230):639-44.
  27. Petersen I, Douglas I, Whitaker H. Self controlled case series methods: an alternative to standard epidemiological study designs. BMJ (Clinical research ed). 2016;354:i4515.
  28. Douglas IJ, Evans SJ, Hingorani AD, Grosso AM, Timmis A, Hemingway H, et al. Clopidogrel and interaction with proton pump inhibitors: comparison between cohort and within person study designs. BMJ (Clinical research ed). 2012;345:e4388.
  29. Burgess S, Butterworth A, Malarstig A, Thompson SG. Use of Mendelian randomisation to assess potential benefit of clinical intervention. BMJ (Clinical research ed). 2012;345:e7325.

The literature refers to the Dutch text

Authors

  • Olaf M. Dekkers, MD, PhD, MSc, MA