SCCM is updating its SCCM Connect Community. Access to SCCM Connect may be limited until April 23.

How the Pandemic Impacted Evidence-Based Medicine

visual bubble
visual bubble
visual bubble
visual bubble

Is This the New Normal?

Judi Jacobi, PharmD, BCCCP, MCCM

The scope of the COVID-19 pandemic is beyond anything most could imagine. The impact on healthcare institutions and clinicians is still being defined, but in the heat of the initial wave, there was a feeling that the plane was being constructed while flying. Critical thinking skills were challenged by fear and lack of time, minimizing the ability to question assumptions and apply reason through logic (rather than wishful thinking). Theoretical therapies were used, based on cellular or animal data.1 These unprecedented challenges led clinicians who were being rapidly overwhelmed to consider any available therapy that theoretically could help, erring on the side of commission rather than omission.
After a long career at the bedside in critical care pharmacy, I have witnessed therapeutic wishful thinking—that suppression of premature ventricular contractions would prevent severe tachyarrhythmias, high-dose vitamin C would reduce sepsis mortality, and many others. Admittedly, I learned best through my own mistakes and hopefully was able to change perspective when needed. I retired from that role in 2019 and watched the COVID-19 pandemic developments with interest and concern for my colleagues worldwide.
Early papers evaluated potential therapies based on a variety of cell-based assessments of drug ac­tivity, often using Vero E6 or FRhK cells because the virus replicated readily, but these monkey cell lines were not good models for human respiratory tissue. In July, recognition that the viral inhibition shown by hydroxychloroquine or chloroquine in monkey kidney cells was not duplicated in human lung cells explained the many clinical failures seen by that time.2 It is difficult to predict how many other agents identified as potentially effective by the monkey cell model are similarly incorrect. Lopinavir/ritonavir failures may be the best similar example, although it did not garner the amount of public attention of hydroxychloroquine.
Based on this type of data, hydroxychloroquine and many other agents were used empirically in China, and early papers suggested benefit in small case series and retrospective reports that were clouded by a morass of other drug therapies (antivirals, herbals, vitamins, antimicrobials, immunoglobulins, and antiinflammatories) that quickly became defined as a local standard of care. Combination therapy may have masked therapies with potential benefit. The definition of standard of care varied from hospital to hospital, and other important interventions (e.g., oxygen support, intubation timing, ventilation strategies, prone positioning) were evolving. Further, while there is no proof that any data were falsified, ongoing concerns with “paper mills” in China producing papers using falsified datasets and images have not yet been fully addressed.3 There have been other concerns with data quality throughout this evolving pandemic.
As I sat on the sidelines with time to evaluate these papers, it was apparent that many confound­ers were influencing observational results and, in a few cases, very dubious methods. One paper written about hydroxychloroquine was actually registered to study a Chinese herbal regimen. French data that were incomplete and improperly excluded patients from evaluation were published without adequate peer review by a previously discredited author. Reports were rushed to pub­lication to be the first to report when a significant number of patients were still in critical condition. Surrogate study end points (fever reduction, viral clearance) often included large numbers of patients who were not critically ill.4
Simultaneously, the preprint servers emerged to bring data to clinicians prior to peer review, and press releases have joined that information over­load, aptly named a “paperdemic” or “infodemic.”5-8 As of August 18, 2020, there were over 6100 papers on COVID-19 in the medRxiv preprint list. Other papers published in prominent journals were retracted due to concern for a large dataset that could not be verified.9 As of August, PubMed had almost 45,000 papers in the COVID-19 category representing accepted papers.
On a positive note, the editors of professional journals are to be praised for allowing open access to COVID-19 publications. Access barriers were replaced by the challenge to keep up with this tidal wave of data, likely beyond the ability of most clinicians, who were actively providing patient care. Further, the lay press have been highly engaged and desperate to report anything with a positive message, often misinterpreting statistics, surrogate outcomes, applicable populations, and other components. Some politicians also promoted anything that suggested benefit. Nevertheless, hospitals developed treatment guidelines, restructured the physical plant, established new safety measures, and addressed a myriad of challenges to persevere on a day-to-day basis. The level of innovation has been excep­tional but has come with the risk that practices are not optimal, are not current in a rapidly changing environment, or may be wasteful.
One treatment decision that was based on prior experience was an early suggestion to avoid corticosteroids, based on influenza data. The early World Health Organization guidelines suggested avoiding cortico­steroids.10 However, the Surviving Sepsis Campaign guidelines suggested their potential role in patients with acute respiratory distress syndrome.11 Using an adaptive trial design, the UK RECOVERY group was able to perform a series of trials showing benefit with corticosteroids and lack of benefit with hydroxychloroquine.12-14 Unfortunately, these and other trial results came through press releases before even preprint versions of the manuscript, leaving clinicians with the quandary about how to utilize the information in a responsible way.
Clinicians under pressure responded admirably. Creative solutions to patient care problems emerged and were shared. IV pumps were moved outside the room with long extension tubing, plastic drapes and boxes were used to reduce aerosol generation with intubation, and the model was changed from early and preemptive intubation (to avoid urgent and emergent intubation) to delayed intubation and utilization of awake proning, noninvasive oxygenation methods, and avoidance of mechanical ventilation when possible. Clearly the sharing of this experiential knowl­edge has been beneficial overall to patients and clinicians. Unfortunately, some of this creativity may have distracted from our core mission of consistent supportive care, based on decades of understanding of pul­monary pathophysiology. The importance of a well-functioning critical care team, where everyone knows what needs to be done and ensures it happens, remains essential.
Making some therapies popular and prominent has led to high demand and intermittent shortages, at least regionally.15 Additional supply chain challenges remain significant, with sporadic and regional inadequacy of supplies of personal protective equipment, ventilators, medications, and other standard equipment. While the problem of drug shortages is not new, health systems have been challenged to utilize their ethical frameworks to allocate limited resources and make enhancements to include larger populations.16,17 Surge plans have been shared to assist other sites with their preparations.18,19
Organizations responded with amazing amounts of targeted education. The Society of Critical Care Medicine (SCCM) has provided webinars, podcasts, frequently asked questions, microlearning presentations, news streams, and publications. These resources remain available on SCCM’s COVID-19 Rapid Resource Center, but clinicians must recognize when older content is no longer as applicable.20 Education has been directed toward both ICU staff and the non-ICU personnel who have stepped forward to assist during the surge. The Strategic Education Task Force was created to direct program development and assist the SCCM staff.
So, what has happened with evidence-based medicine principles during the COVID-19 pandemic? My early fear that the pandemic would cause its demise was premature, and I believe that clinicians and researchers are now gaining perspective, using a more systematic approach and continuing to rely on years of solid knowledge. Research groups have been organized throughout the world and have completed large studies in the face of great personal challenges.12 The worldwide SCCM Discovery VIRUS COVID-19 Registry will hopefully provide perspective for the future.21
Finally, as we gain perspective on what has transpired, I am gratified to see that standards for research and quality improvement have been published with a goal to consistently frame data and analysis.22,23 The challenge is now to meet those standards.
Our talented and committed critical care personnel and teams must continue to provide evidence-based care, do the little things well (e.g., checklists, consistent care and monitoring, following protocols) and not feel pressured to just do something—trusting foundational knowledge while cautiously assimilating new data to maintain the important role of evidence-based medicine.24

  1. Costa A, Weinstein ES, Sahoo DR, Thompson SC, Faccincani R, Ragazzoni L. How to build the plane while flying: VTE/PE thromboprophylaxis clinical guidelines for COVID-19 patients. Disaster Med Pub Health Preparedness 2020 Jun 16;1-15. doi: 10.1017/dmp.2020.195. Online ahead of print.
  2. Hoffmann M, Mösbauer K, Hofmann-Winkler H, et al. Chloroquine does not inhibit infection of human lung cells with SARS-CoV-2. Nature. 2020 Jul 22. Online ahead of print.
  3. Mallapaty S. China’s research-misconduct rules target ‘paper mills’ that churn out fake studies. Nature. doi: 10.1038/d41586-020-02445-8. Online ahead of print.
  4. Ingraham NE, Tignanelli CJ. Fact versus science fiction: fighting coronavirus disease 2019 requires the wisdom to know the difference. Crit Care Explor. 2020 Apr 29;2(4):e0108.
  5. medRxiv. The preprint server for health sciences. COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv. Accessed September 14, 2020.
  6. JMIR Publications. JMIR Preprints. Accessed September 14, 2020.
  7. Dinis-Oliveira RJ. COVID-19 research: pandemic versus “paperdemic,” integrity, values and risks of the “speed science.” Forensic Sci Res. 2020;5:174-187.
  8. Tuccori M, Convertino I, Ferraro S, et al. The impact of the COIVD-19 “infodemic” on drug-utilization behaviors: implications for pharmacovigilance. Drug Saf. 2020 Aug;43(8):699-709.
  9. Piller C, Servick K. Two elite medical journals retract coronavirus papers over integrity questions. Science. June 4, 2020. Accessed September 14, 2020.
  10. World Health Organization. Clinical management of COVID-19. Interim guidance. May 27, 2020. Accessed September 14, 2020. 
  11. Alhazzani W, Møller MH, Arabi YM, et al. Surviving Sepsis Campaign: guidelines on the management of critically ill adults with coronavirus disease 2019 (COVID-19). Crit Care Med. 2020 Jun;48(6):e440-e469.
  12. Recovery. Randomised evaluation of COVID-19 therapy. Accessed September 14, 2020.
  13. RECOVERY Collaborative Group; Horby P, Lim WS, et al. Dexamethasone in hospitalized patients with COVID-19: preliminary report. N Eng J Med 2020 Jul 17;doi: 10.1056/NEJMoa2021436. Online ahead of print.
  14. Horby P, Mafham M, Linsell L, et al. Effect of hydroxychloroquine in hospitalized patients with COVID-19: preliminary results from a multi-centre, randomized, controlled trial. medRxiv. Preprint posted July 15, 2020.
  15. Kanji S, Burry L, Williamson D, et al; Ontario COVID-19 Drug Task Force (Appendix). Therapeutic alternatives and strategies for drug conservation in the intensive care unit during times of drug shortages: a report of the Ontario COVID-19 ICU drug task force. Can J Anesth. 2020 Oct;67(10)1405-1416.
  16. White DB, Angus DC. A proposed lottery system to allocate scarce COVID-19 medications: promoting fairness and generating knowledge. JAMA. 2020 Jul 28;324(4):329-330.
  17. Satomi E, Rodrigues de Souza PM, da Costa Thomé B, et al. Fair allocation of scarce medical resources during COVID-19 pandemic: ethical considerations. Einstein (Sao Paulo). 2020 Apr 30;18:eAE5775.
  18. Harris GH, Baldisseri MR, Reynolds BR, Orsino AS, Sackrowitz R, Bishop JM. Design for implementation of a system-level ICU pandemic surge staffing plan. Crit Care Expl. 2020 Jun 15;2(6):e0136.
  19. Halpern NA, Kaplan LJ, Rausen M, Yang JJ. Configuring ICUs in the COVID-19 Era. Society of Critical Care Medicine. Updated June 15, 2020. Accessed August 18, 2020.
  20. Society of Critical Care Medicine. COVID-19 Rapid Resource Center. Accessed August 18, 2020.
  21. Society of Critical Care Medicine. Discovery VIRUS COVID-19 Registry. Accessed September 14, 2020.