The COVID-19 pandemic has been associated with a storm of information by social media and with an increase in publications with high percentages of retractions.1,2 There is a clear need for reproducible randomised controlled trials (RCTs) on the topic. A discussion by independent authors with expertise in research methodology has been proposed for probing inferential reproducibility and for addressing issues in discussion sections of evaluated papers.3
The pandemic has also triggered unprecedented use of tools supposedly intended to prevent invasive mechanical ventilation among individuals with COVID-19 associated acute respiratory failure.4,5 Quality data, in particular reproducible RCTs regarding modes of non-invasive respiratory support (RS), are greatly needed.6 High-flow nasal oxygen (HFNO) is one such tool.7,8
In a recently published RCT, the COVID-HIGH trial,9 individuals with COVID-19 and mild hypoxaemia were randomised to treatment with either HFNO or conventional oxygen therapy (COT).9 The trial is particularly relevant as HFNO is increasingly being used in enviroments with a lower level of monitoring where such patients are often treated.10,11 Using a structured independent discussion,3 two authors with expertise in research methodology consider the findings and the inferential reproducibility of this RCT. Below, original and independent discussions for each section of the paper 9 are compared.3
Main findingsOriginal Discussion: The authors report that individuals with COVID-19 pneumonia and mild hypoxaemia randomised to HFNO versus COT had similar rates of RS escalation within 28 days.9
Independent discussionThe study and control groups did not differ in the rate of the primary outcome. The sample size for the study 9 was calculated based on a retrospective study, the best evidence at the time.12 However, effect size is often inflated in retrospective studies,13,14 and indeed the observed rate of RS escalation was lower than expected. COVID-HIGH has an 80% power to identify only a 35% relative difference in event rate (i.e. a 15% absolute risk reduction).9 In order to identify a 27% relative difference, which takes into consideration the actual event rate (i.e. an 11% absolute risk reduction), 580 individuals would need to be recruited. Therefore the study, with 364 participants, only has a 60% power to refute the baseline hypothesis of no difference between the two modes of treatment. The likelihood of a type II error (i.e. a false negative finding) is high.
Commentary: The original and independent discussions are concordant in their interpretation of the main study findings in COVID-HIGH. The independent discussion also highlights that study underpowering may limit the validity of the main study finding.
Relationship of main findings to previous studiesOriginal discussion: The authors compare their findings to those of the RECOVERY-RS multicentre trial which showed no difference between HFNO and COT for the primary outcome of intubation or 30-day mortality.15 They highlight that the study15 recruited individuals with greater disease severity, with a SpO2 ≤94% despite receiving an inspiratory oxygen fraction (FiO2) of at least 40% and point out that RECOVERY-RS15 had continuous positive airway pressure (CPAP) as a third study arm. The authors also cite a RCT by Ospina-Tascon et al.16 in which HFNO significantly reduced the intubation risk and time to clinical recovery in individuals with FiO2 <200. They propose that, taken together, the findings of these trials suggest different clinical effects of HFNO versus COT in individuals with different disease severity.9,15,16
Independent discussion: Like COVID-HIGH,9 the RCT by Ospina-Tascon et al., was underpowered for the primary outcome (220 participants).16 RECOVERY-RS was an adaptive (group-sequential) cohort nested within a pragmatic trial. Hence it should have been adequately powered to show different treatment effects, but was stopped early for futility.15
The effect of any intervention depends on several variables, including baseline load of comorbidity, disease severity, treatment timing (late vs. early) and dose. The COVID-HIGH trial9 included participants with moderate Charlson comorbidity scores. Markers of inflammation were not used as inclusion criteria,17 but these data are presented and indicate moderate disease.9 Both Ospina-Tascon et al.16 and RECOVERY-RS15 included individuals with worse disease than COVID-HIGH.9 Ospina-Tascon et al.16 described comorbidity rates similar to those described in COVID-HIGH.9 Although the ROX indices18,19 of individuals under HFNO were higher than those under COT, the levels of inflammation markers suggest more severe disease than in COVID-HIGH.9 RECOVERY-RS decribes more heart and lung diseases but provides neither the overall weight of comorbidity nor data on inflammation markers.15
In COVID-HIGH the time from symptom onset to randomization averaged seven9 versus eleven days in the Ospina-Tascon trial16 and nine days in RECOVERY–RS.15 Finally, in COVID-HIGH9 the treatment protocol (i.e. “dose”) was preset as were the criteria for treatment escalation. The average duration of treatment was three days. Ospina-Tascon16 also protocolized treatment, including the criteria for intubation. Treatment duration was planned as six days but ultimately averaged only one day. In RECOVERY-RS15 treatment with HFNO was not protocolized and the duration of treatment was not described.
Commentary: Both discussions refer to the same two studies,15,16 however, the independent discussion identified more differences between the studies.
Secondary findingsOriginal discussion: The authors report that the secondary clinical outcomes (i.e. likelihood of clinical recovery, time to first RS escalation, rate of intensive care unit [ICU] admission and 28- and 60-day mortality) did not differ between the treatment arms.9
Independent discussion: No difference was found in secondary outcomes between study and controls.
Commentary: The original and independent discussions are concordant in their interpretation of the secondary study findings in COVID-HIGH study.9
Relationship of additional (secondary) findings to previous studiesOriginal discussion: The relationship of secondary findings to previous studies was not discussed by the authors.
Independent discussion: The COVID-HIGH trial was not powered for any of the secondary outcomes examined although these were preplanned and were registered in the study protocol.9
Ospina-Tascon16 noted earlier (but not more) recovery among participants treated with HFNO. ICU admission rates were similarly unaffected by treatment with HFNO in COVID-HIGH9 and RECOVERY-RS.15 Thirty day mortality was part of the primary composite outcome of RECOVERY-RS15 and was not related to treatment. It was a secondary outcome in the Ospina-Tascon trial16 where it was also unrelated to treatment. Neither trial15,16 reported the time to first RS escalation or 60 day mortality.
Commentary: HFNO does not seem to have a consistent effect on any of the objective secondary outcomes studied. Indications for ICU admission and length of stay are also dependent on local bed availability and practices. Long term outcomes were not studied in any of the trials but short-term mortality rates (28- or 30- day mortality in all three trials) seem consistently unaffected.9,15,16
LimitationsOriginal discussion: The authors admit their study has several limitations. Due to the nature of interventions, blinding was not possible. However, clinical criteria used to decide on RS escalation were standardised. Subjectivity in clinical judgement could not be excluded. In selected cases, clinicians may have considered HFNO as a form of RS and been less likely to escalate to CPAP/NIV compared with COT. This may partly explain the higher protocol violation rate in the control group. The trial was underpowered. However a clinically meaningful benefit from HFNO in this population could not be definitely ruled out. The COVID-HIGH cohort included 64% male participants, which may limit the generalisability of the findings.9 However, the adjusted odds-ratio for sex showed no significant effect on the association between occurrence of the primary outcome and study interventions. Due to the multinational and multicentre nature of the study, different pandemic surges may have had different indirect consequences on the care level at study sites. Data on SARS-CoV-2 variants or vaccination status of participants were not registered. Finally, the results of the subgroup analyses should be considered exploratory as positive findings may be attributed to repeated testing.9
Independent discussion: Lack of power is the most important study limitation in COVID-HIGH.9 Despite the difference between the expected and observed RS escalation rates, the investigators chose to close the study with the preplanned number of participants as prolonging the study would have increased population variability (e.g. COVID variants) and co-treatment effects (e.g. vaccine effects, local practice). The COVID-HIGH investigators provide no data on the use of ancillary respiratory support therapies such as self-proning,20 physiotherapy and mobilisation.21 While individuals with limitation of care before randomization were not included, this status may change during treatment. Like previous RCTs, information regarding with-holding/withdrawal of care is missing. Finally, the rates of specific COVID phenotypes may have differred in the two study groups and specific phenotypes may respond differently to different management strategies.22
Commentary:
Both discussions agree on the lack of power, the independent discussion highlighted several issues that were not mentioned in the original discussion.
Future directionsOriginal discussion: The original discussion did not consider future research directions.
Independent discussion: Future physiological work should include comparative data on work of breathing with HFNO versus COT. Clinical data should include adequately powered RCTs with more detailed information on the effects of HFNO, if such exist, in different COVID-19 disease phenotypes and data on the long term effect of HFNO. More data is also required on the human- and health-resource costs of using HFNO and on the risks of caregiver contamination.23
Commentary: Only the independent discussion suggests directions for future research and how these may be informed by the findings of COVID-HIGH.
ConclusionOriginal discussion: The authors concluded that HFNO did not significantly decrease the escalation of RS compared with COT among hospitalised individuals with COVID-19 pneumonia with mild hypoxaemia.9
Independent discussion: The current evidence shows no proof either for or against indiscriminate use of HFNO in hospitalised individuals with COVID-19 pneumonia with mild hypoxaemia.
Commentary: The independent conclusion highlights the need for additional research.
Inferential reproducibilityThere was acceptable inferential reproducibility between the two discussions. The independent discussion provides a more detailed description and analysis of the primary and secondary findings in relation to current literature than the original discussion and offers more in-depth explanations for the lack of effect and methodological issues in COVID-HIGH. However, both discussions agree on the key findings and their interpretation.
