The patients were stratified based on admission data and assigned to risk classes for each score, as validated by previous studies.8,9 Patients were assigned to low-, intermediate-, and high-risk classes as follows: PSI score: low risk, classes I through III; intermediate risk, class IV; and high risk, class V; CURB-65: low risk, classes 0 to 1; intermediate risk, class 2; and high risk, classes 3 through 5; SCAP score: low risk, classes 0–1 (0–9 points); intermediate risk, class 2 (10–19 points); and high risk, classes 3–4 (≥20 points) and SMART-COP score: low risk, 0–2 points; intermediate risk, 3–4 points; and high risk, ≥5 points.1,3–9 The following four adverse outcomes were evaluated: death, ICU admission, need for invasive mechanical ventilation or vasopressor support.

The study was conducted in a tertiary public hospital which had 580 inpatient beds (27 in the pulmonology department, five of which were in isolation rooms) and 12 polivalent ICU beds plus 18 cardiothoracic and cardiology/coronary ICU beds. It is the reference hospital for around 400000 citizens and second line for over 600000 citizens. Each year, around 150–200 patients with pneumonia are admitted to the pulmonology department.

The diagnosis of pneumonia was made according to the standardized best clinical practice – acute illness associated with ≥1 of the respiratory symptoms: pain, dyspnea, fever or hypothermia, altered breath sounds on auscultation, and the presence of a new infiltrate on a chest radiograph. Pneumoccocal pneumonia was defined as a pneumonia associated with positive sputum, bronchoalveolar lavage, blood or pleural fluid culture and/or rapid urinary antigen testing and an absence of alternative cultural microbiological agent.10–12

All patients received antibiotic regimens with expected efficacy in accordance with the accepted national and international guidelines within the first hours of ER admittance.10,11

Treatment failure was defined as a lack of response or clinical deterioration (hemodynamic instability, impairment of respiratory failure, need for mechanical ventilation, radiographic progression, or appearance of new metastatic infectious foci) with appropriate drug dosage and delivery system, which led to a change in antibiotic regimen.13

Descriptive statistical measures of the studied variables included mean and standard deviation for continuous variables, and absolute frequency and percentage for each category of a categorical variable. For each major adverse event and for each score, the ROC curve plotting sensitivity against one minus specificity was considered, and the area under the curve (AUC) was calculated. This is a measure of the global diagnostic exactness of the score. Estimates for sensitivity, specificity and negative predictive value for each score separated into low versus non-low risk classes were then presented.

All statistical analyses were performed with SPSS version 19 and MedCalc version 12. Significance level was fixed at 0.05.

Table 2 shows the predictive accuracy of the PSI, CURB-65, SCAP and SMART-COP score for ICU admission, need for mechanical ventilation, need for vasopressor support and mortality. The adverse outcome rate increased steadily from low- to high risk classes for all four instruments.

The classification as high risk patients in the 2 new scores shows higher accuracy for predicting all adverse outcomes, compared to the classic scoring systems.

For each risk class we separated the low versus non-low risk classes and for each group we assessed the sensitivity, specificity and negative predicted value for the anticipating major adverse outcomes defined by the classification of a certain patient in a non-low risk class. The importance of a correct distinction for these two groups is consistent with the conviction that patients within the low risk classes can be safely treated at home as they have low mortality risk.

We calculated the area under the ROC curve (AUC) for each severity score and for each major adverse event. Results are summarized in Table 3.

When applying the Hanley and McNeil14 method for comparing paired AUC we found that for the mortality there was no statistically significant difference between all the scores. For ICU admittance both SCAP and SMART-COP are superior to PSI and CURB-65, but there are no statistically significant differences between the former pair or the latter. Regarding the need for mechanical ventilation, SCAP is superior to PSI and CURB-65, SMART-COP is superior only to PSI and there are no differences between SCAP and SMART-COP. Concerning the need for vasopressor support both SCAP and SMART-COP are superior to PSI but not to CURB-65, without any difference between SCAP and SMART COP.

For each adverse advent a ROC curve for each score was produced (Figs. 1–4).

Figure 1.

ROC curve for the mortality prediction for each score.

(0.06MB).

Figure 2.

ROC curve for the need for ICU prediction for each score.

(0.06MB).

Figure 3.

ROC curve for the need for vasopressor support prediction for each score.

(0.07MB).

Figure 4.

ROC curve for the need for mechanical ventilation prediction for each score.

(0.07MB).

We found a high percentage of patients with an initial stratification of low risk that were admitted to the hospital (50.7–68.3%). Even though there is a majority of patients classified as low-risk in all scoring systems, the attending physicians considered nevertheless that the patients had a clinical condition or social problem that made for hospital treatment (the most common reported causes of admission despite the low risk scores were the presence of respiratory failure, multilobar opacities and/or pleural effusion, decompensation of comorbidities and lack of social support).

All the scores in the study presented very high negative predictive values for studied adverse events. This means that when considered as being a low-risk patients the probability of developing an adverse event is quite low. This is consistent with the idea that these patients can potentially be treated at home.

When addressing the question of which of the severity scores was the best at establishing the low risk, the score that had the highest negative predictive value in all adverse outcomes was the SCAP score.

Compared to the other scores, SCAP significantly stratified more patients in the medium risk class. The differences might be due to the underestimation, by the classic scores, of the radiological appearance and the gravity of respiratory insufficiency at the time of admission (the latter being a major drawback of the CURB-65) and the overestimation of stable co morbidities and the greater number of variables addressed by the PSI.

Most of the adverse outcomes are reported in the high-risk class patients – this is more significant in the new scores (SCAP and SMART-COP) where percentage of patients in the high risk class and ICU admission, mechanical ventilation or vasopressor support was quite high. This is consistent with the purpose for which they were developed (to accurately select the high risk patients, mostly the ones that will need close monitoring and more aggressive treatment or ICU admission).

All scores had very high sensitivity and negative predictive values for assessing the risk of mortality.

When applying the Hanley and McNeil14 method for comparing paired AUC we observed that SCAP score was statistically superior to PSI and CURB-65 for predicting the need for ICU admission and mechanical ventilation, but only superior to PSI regarding vasopressor support.

The SMART-COP score proved statistically superior to PSI and CURB-65 for predicting the need for ICU admission, but only superior to PSI for mechanical ventilation and vasopressor support.

There were no statistical differences between SCAP and SMART-COP comparing their AUC for each score.

In the literature we found some studies which compared different severity scores, most commonly for community acquired pneumonia. The majority address the mortality rate and ICU admission and only a few address other variables such as need for vasopressor support, mechanical ventilation or total length of stay as independent outcomes in their studies.

The rate of treatment failure in community acquired pneumonia is estimated to be between 10 and 15%.14 In a Portuguese university hospital study of mortality in patients admitted with pneumococcal pneumonia, the authors observed a need for antibiotic change in 15.4% characterized by clinical worsening and persistence indications of infection and fever.15 Our results were consistent with these previous studies and since the population was the same for all the scoring systems, the authors do not consider the rate of treatment failure to have influenced the outcomes of this study.

In a Spanish study of patients hospitalized for pneumonia, Rosón described that 43% of patients were admitted even though they were low-risk patients according to the PSI.16 They found that these patients had respiratory failure, pleural effusions or unstable vital sounds and that these parameters could not be correctly assessed by relying solely on the PSI score.16

Previous studies have shown that both PSI and CURB-65 perform consistently as predictors for mortality with AUC values ranging from 0.74 to 0.83 for PSI and from 0.73 to 0.83 for CURB-65. When predicting other adverse events such as need for mechanical ventilation or vasopressor support these values are less impressive – AUC between 0.69 and 0.79 for PSI and between 0.59 and 0.77 for CURB-65.17–23 The SMART-COP has been shown to outperform these previous scores with AUC of 0.87 for these last adverse events.7

A Swedish study of prognostic scores in community acquired bacteriaemic pneumococcal pneumonia showed that the classification of patients in the PSI risk classes of IV–V had sensitivity/specificity/PPV/NPV for mortality of 100/60/25/100 and ICU admission of 95/64/36/98. The CURB-65 classification in the classes 3–5 had poorer results respectively 62/86/36/95 for mortality and 71/87/55/91 for ICU admission.24 Our study showed similar results regarding mortality but slightly less impressive values for ICU admission in these classic scores.

España et al. found that the SCAP score was as accurate or better than other scoring systems (CURB-65 and PSI) in predicting adverse outcomes in patients hospitalized with CAP while helping classify patients into different categories of increasing risk for potentially closer monitoring.6 They also found that the SCAP score classified a significantly higher proportion of patients as low risk than the PSI and CURB-65, with lower rates of all adverse outcomes.6 These results were different in our study, where SCAP stratified a lesser number of patients compared to the other 3 scores and a higher number in the medium risk.

This study has some limitations. Firstly, all included patients were admitted to the hospital and the data did not address the great number of patients that were treated as outpatients and their outcomes or complications. Secondly, the sample size and the number of adverse events (most importantly the low number of deaths) observed in this study can be a limiting factor for its conclusions. Thirdly, not all data were available. Nevertheless, the amount of values found to be missing from the admission records was low (<2%) and most commonly associated with the respiratory rate – for the purposes of this study the authors consider it did not represent an important limitation since the cut off for the respiratory rate values are similar (the SCAP being age related).

The sensitivity for the other major adverse outcomes is considerably higher in the new severity scores (>80% for all major adverse outcomes) as compared with the more classic scores. The SCAP score while stratifying a greater number of patients as medium risk has a lower specificity for the prediction of adverse events. Nevertheless, it has the best AUC for all adverse events and so the highest diagnostic accuracy.