The predictive performance of SAPS 2 and SAPS 3 in an intermediate care unit for internal medicine at a German university transplant center; A retrospective analysis

Summary

Objective

To analyze and compare the performance of the Simplified-Acute-Physiology-Score (SAPS) 2 and SAPS 3 among intermediate care patients with internal disorders.

Materials and methods

We conducted a retrospective single-center analysis in patients (n = 305) admitted to an intermediate-care-unit (ImCU) for internal medicine at the University Hospital Essen, Germany. We employed and compared the SAPS 2 vs. the SAPS 3 scoring system for the assessment of disease severity and prediction of mortality rates among patients admitted to the ImCU within an 18-month period. Both scores, which utilize parameters recorded at admission to the intensive-care-unit (ICU), represent the most widely applied scoring systems in European intensive care medicine. The area-under-the-receiver-operating-characteristic-curve (AUROC) was used to evaluate the SAPS 2 and SAPS 3 discrimination performance. Ultimately, standardized-mortality-ratios (SMRs) were calculated alongside their respective 95%-confidence-intervals (95% CI) in order to determine the observed-to-expected death ratio and calibration belt plots were generated to evaluate the SAPS 2 and SAPS 3 calibration performance.

Results

Both scores provided acceptable discrimination performance, i.e., the AUROC was 0.71 (95% CI, 0.65–0.77) for SAPS 2 and 0.77 (95% CI, 0.72–0.82) for SAPS 3. Against the observed in-hospital mortality of 30.2%, SAPS 2 showed a weak performance with a predicted mortality of 17.4% and a SMR of 1.74 (95% CI, 1.38–2.09), especially in association with liver diseases and/or sepsis. SAPS 3 performed accurately, resulting in a predicted mortality of 29.9% and a SMR of 1.01 (95% CI, 0.8–1.21). Based on Calibration belt plots, SAPS 2 showed a poor calibration-performance especially in patients with low mortality risk (P<0.001), while SAPS 3 exhibited a highly accurate calibration performance (P = 0.906) across all risk levels.

Conclusions

In our study, the SAPS 3 exhibited high accuracy in prediction of mortality in ImCU patients with internal disorders. In contrast, the SAPS 2 underestimated mortality particularly in patients with liver diseases and sepsis.

Keywords:

Death rates – Cirrhosis – kidneys – chronic kidney disease – Intensive care units – liver transplantation – sepsis – Trauma surgery

Introduction

The use of in-hospital mortality prediction scores at admission to an intensive care unit (ICU) has become a viable evaluation method for treatment outcomes in critically ill patients [1]. To date, the Simplified Acute Physiology Score (SAPS) 2 and SAPS 3 are two of the most extensively validated risk prediction scores worldwide, and represent the most frequently applied scoring systems in European intensive care medicine [2–7]. In contrast to crude mortality data, such scores provide a risk-adjusted mortality assessment by considering various grades of disease severity and other prerequisites or predisposing conditions. Thus, standardized mortality ratios (SMRs) can be reliably calculated by comparing observed versus expected mortality rates [8]. Given an acceptable calibration performance, SMRs are useful tools, which can be used to evaluate interventions and/or quality of clinical management within an ICU or across a group of ICUs with comparable configurations over time. SMRs can also act as benchmarking parameters for performance assessment and improvement in an ICU with evolving cost-containment policies and medical practices or structures, as (for example) a decreasing SMR may indicate the presence of some particular change that is beneficially affecting mortality [9].

To date, in contrast to ICUs [2–4, 6, 7, 10–17], reports on routine application of mortality prediction scores in intermediate care units (ImCUs) are scarce [18–21]. Although such scores are considered important in decision-making processes in critical care, they are not yet standardly integrated in routine algorithms (e.g., via the hospital information system) in German ImCUs. Moreover, due to quite heterogeneous ImCU characteristics/settings, uniform validation of these scores remains challenging.

Meanwhile, significantly reduced in-hospital mortality rates have been reported for patients admitted to ICUs in hospitals with an ImCU relative to those admitted to ICUs in hospitals with no ImCUs [22]. These findings have added to the rising popularity of ImCUs in Germany, especially, for patients, which are too sick for the normal ward but also too stable to be admitted to the ICU [23]. The still relatively constrained ICU bed availability and the growing need to treat low/moderate-risk critical care patients in a more cost-effective manner have primarily driven this development. In the German hospital sector, ImCUs can be either integrated and/or adjacent to an ICU or run separately and independently from an ICU [24]. In fact, ImCUs may serve as “step-up” or “step-down” units between general wards and ICUs or as “subintensive” wards for patients coming from emergency or recovery rooms [25]. Although the majority of ImCUs treat surgical patients [24], the spectrum of medical support provided in each ImCU depends on the specific infrastructures and/or areas of expertise of the hosting clinics. However, continuous monitoring, non-invasive mechanical ventilation and application of vasoactive medications on demand represent common treatment modalities among ImCUs. Therefore, it is reasonable to further pursue evaluation of ICU scoring systems like the SAPS2 and/or the SAPS 3 in ImCU patients [24].

Taken together, this study aimed to analyze and compare performance of the SAPS 2 vs. the SAPS 3 in patients with internal disorders—who were admitted to an ImCU at a German university hospital´s transplant center—in order to improve the quality of local care evaluation and management. In addition, our data should be cross-interpreted with results from previous studies that evaluated the performance of both scores in non-surgical ImCU patients [18–21].

Materials and methods

Design, setting and patients

This study was conducted in a 22-bed ImCU for internal medicine at the University Hospital Essen, Germany, an academic clinical institution with a nearly 1300-bed capacity. This ImCU is headed by a team of gastroenterologists/hepatologists and nephrologists and runs independently from other intensive care units. Patients with acute or acute-on-chronic liver failure, acute kidney injury or chronic kidney disease (including patients requiring hemodialysis), patients receiving immunosuppressive therapy after solid organ transplantation (SOT; especially kidney and liver transplantation), and transplant candidates with various acute conditions accounted for the majority of ImCU admissions.

The nurse-to-patient ratio has been 1:4. The ImCU medical team involves 9 physicians (of those, 5 critical care specialists), i.e., 2 senior physicians and 7 residents, who work in 8- to 12-hour shifts. All beds are standardly equipped with continuous telemetry, pulse-oxymetry, body temperature as well as non-invasive and invasive monitoring of arterial blood pressure and central venous blood pressure. The bedside monitors are linked to a central unit that processes all relayed data and/or alarm signals.

Within a 18-month period, from January 2014 to June 2015, patients´ characteristics, medical history, reasons for admission as well as the worst clinical conditions and laboratory values were recorded (independently from medical interventions) within the first hour after ImCU-admission for SAPS 3 [26] or within the first 24 hours after ImCU-admission for SAPS 2 [27]. For both scores, predicted mortality rates have been calculated using the equations shown in Table 1. While SAPS 2 uses one general equation [27], customized formulas are available in different major geographic regions for SAPS 3; the present analysis has been carried out by using the SAPS 3 customized equation for North Europe [26].

During the study period, 73 out of 379 ImCU-admissions were identified as readmissions, and have therefore been withdrawn from further consideration. In addition, one patient had to be excluded due to incomplete data. Thus, data from 305 patients (all aged over 18 years) were subjected to final statistical analysis. In an effort to increase data reliability, we undertook quality control tests of data collected during a one-month pilot run, before launching the regular data acquisition phase.

The SAPS 2 includes 15 variables, i.e., 12 physiology variables, age, type of admission, and one variable related to underlying disease, which should be recorded in a 24-hour time window after admission [27]. The SAPS 3 utilizes 20 variables, i.e., 5 variables regarding patient characteristics prior to admission, 5 variables regarding the circumstances of the admission, and 10 physiology variables, which should be recorded within 1 hour before or after admission [26]. Table 1 provides a side-by-side overview of both scoring systems along with a detailed description of parameters included in each score. In-hospital mortality was the endpoint of this study.

Statistical analysis

Statistical analysis was performed using SPSS (version 21.0; SPSS Inc, Chicago, IL) and SAS software (version 9.4; SAS Institute Inc., Cary, NC). For descriptive statistics, absolute and relative frequencies were calculated for categorical parameters, whereas continuous parameters were characterized using the median (MD) as well as the first and third quartile (Q1, Q3). For SAPS 2 and SAPS 3 mean and standard deviation (Mean ± SD) were also calculated. Inferential statistics to compare deceased with non-deceased patients included Fisher’s Exact Test for categorical variables and the Mann-Whitney U test for continuous variables. Results were considered statistically significant when P≤0.05.

The performance of prognostic models such as SAPS 2 and SAPS 3 encompasses two measures: discrimination and calibration. While the area under the receiver operating characteristic curve (AUROC) was calculated as measure for discrimination [28]), calibration was assessed using the Calibration belt [29, 30]. Note that a high p value (P>0.05) indicates a well-calibrated model. Calibration curves for the Calibration belt (Figs 1 & 2) were created by plotting the predicted mortality (x-axis) against observed mortality (y-axis). SMRs alongside their respective 95% confidence intervals (95% CIs) were determined for both SAPS 2 and SAPS 3 by dividing the observed mortality rate through the mean value of predicted mortality rates. Thus, we allow identifying over- (SMR < 1) or underestimation (SMR > 1) of the scores regarding mortality in our ImCU population.

Ethics statement

This non-interventional study has been performed in accordance with the ethical principles and standards of the 1964 Helsinki declaration and its later amendments. The study protocol was approved by the local institutional review board of the University Hospital Essen (IRB: Ethik-Kommission am Universitätsklinikum Essen, 15-6436-BO). Because of the observational design of this cohort study, the institutional review board waived the requirement for patients’ informed consent.

Results

Patient characteristics

Of all patients (n = 305), one third (n = 92; 30.2%) was admitted via the department of gastroenterology and hepatology, and two thirds (n = 213; 69.8%) via the department of nephrology. Sources of patients’ admission were regular internal medicine wards at the University Hospital Essen or other hospitals (n = 226; 74.1%), the emergency department (n = 33; 10.8%), and the ICU (n = 45; 14.7%; Table 2). Median patient age was 60 years, the majority of patients were male (n = 184; 60.3%) and the median BMI was 25.7 kg/m² (Table 3). The median stay in the ImCU was 6 days.

Among all chronic diseases taken into account by the SAPS 2 or the SAPS 3, liver cirrhosis (n = 68; 22.3%) showed the highest frequency in our cohort. Further co-morbidities like cancer and cancer therapy (n = 15; 4.9%), chronic heart failure at stage NYHA IV (n = 10; 3.3%), hematological cancer (n = 6; 2.0%) or AIDS (n = 4; 1.3%) were markedly less common. Moreover, we observed a high rate of patients with chronic kidney diseases (CKD) based on the KDIGO-criteria [31] (n = 131; 43.0%, of those, n = 39 with end-stage CKD) and patients with systemic immunosuppression after solid organ transplantation (n = 55, 18.0%; of those, n = 22 post liver transplantation, n = 28 post kidney transplantation, and n = 5 post combined pancreas-kidney transplantation, Table 3).

Regarding the SAPS 3 framework, the most frequent acute conditions at admission were cardiac arrhythmias (n = 52; 17.1%), followed by acute or acute-on-chronic liver failure (n = 50, 16.5%), nosocomial infections (n = 26; 8.5%), vigilance disturbances, e.g. coma, stupor, obtunded patient, agitation, confusion, delirium (n = 26; 8.5%), septic shock (n = 15; 4.9%), gastrointestinal emergencies, e.g. severe pancreatitis, acute abdomen (n = 11; 3.6%), and various focal neurologic deficits (n = 3; 1.0%). We also registered high rates of acute kidney injury based on the KDIGO-criteria [32] (n = 97; 31.8%) or sepsis according to Sepsis-2 criteria [33] (n = 43; 14.1%), and (less frequently) respiratory insufficiency requiring non-invasive ventilation (n = 17; 5.6%) or gastrointestinal bleeding (n = 10; 3.3%, Table 3).

Over half of all patients had to be treated with vasoactive drugs (noradrenaline, dobutamine, adrenaline or hemopressin; n = 188, 61.6%) at admission, while approximately one fourth of all patients (n = 70; 23.0%) received renal replacement therapy in terms of either intermittent hemodialysis (n = 62; 20.4%) or continuous cyclic or ambulatory peritoneal dialysis (n = 8; 2.6%).

Performance of SAPS 2 vs SAPS 3

The median score (MD [Q1; Q3]) was 30 [22; 39] for SAPS 2 and 55 [46; 64] for SAPS 3, resulting in a median predicted mortality of 10.6% [4.7%; 23.0%] and 26.0% [12.0%; 44.0%], respectively (Table 3). Corresponding mean values and standard deviations were 31.4 ± 12.9 for SAPS 2 and 55.4 ± 12.9 for SAPS 3, causing mean predicted mortalities of 17.4% ± 17.6 and 29.9% ± 21.1 (Table 4). The observed in-hospital mortality was 30.2% (n = 92 patients; of those, n = 56 were men and n = 36 women), resulting in an SMR of 1.74 (95% CI, 1.38–2.09) for SAPS 2 and 1.01 (95% CI, 0.8–1.21) for SAPS 3, respectively (Tables 4 and 5). Hence, while SAPS 2 heavily underestimated the true mortality rate, SAPS 3 provided a markedly accurate prediction.

With an AUROC of 0.71 (95% CI, 0.65–0.77) for SAPS 2 and 0.77 (95% CI, 0.72–0.82) for SAPS 3 (Table 4; Fig 3), both scores revealed an acceptable discrimination performance. However, calibration performance differed significantly. Based on the Calibration belt, SAPS 3 presented a good calibration (P = 0.906) while SAPS 2 showed a poor calibration, which was pronounced among patients with low risk of mortality (P<0.001, Table 4, Figs 1 and 2).

When viewed separately, the SAPS 2 and the SAPS 3 demonstrated an equally consistent performance in either nephrological or gastroenterological/hepatological patients. However, the SAPS 2 exhibited a poor performance in terms of SMRs in association with liver failure, cirrhosis and sepsis, which represented leading causes of mortality. In fact, the observed mortality for liver failure, cirrhosis and sepsis was 62.0%, 55.9%, and 62.8% respectively. Here, the SAPS 2 predicted mortality rates were 24.0% for liver failure, 22.3% for cirrhosis, and 17.6% for sepsis, resulting in a SMR of 2.59, 2.51, and 3.57, respectively (Table 5). Overall, the SAPS 3 demonstrated a more accurate SMR for each of these diagnoses along with a predicted mortality of 51.8% for liver failure, 50.0% for cirrhosis, and 39.8% for sepsis resulting in a SMR of 1.2, 1.12, and 1.58, respectively (Table 5).

Discussion

In intermediate care patients, previous studies revealed an overestimation of mortality by either the SAPS 2 [18–20] or the SAPS 3 [18], which is largely consistent with analyses that evaluated these tools in critical care patients [6, 7, 14–16]. In the present work, we report a significant underestimation of mortality by the SAPS 2 in intermediate care patients with prolonged or end-stage liver and/or kidney diseases. To date, the SAPS 2 has often been suggested as most suitable tool for mortality prediction not only by earlier investigations in different ICU settings [14, 16] but also by a recent mixed population study conducted on a multi-purpose Spanish ImCU [18]. Contrary to these results, we found a slightly better discrimination performance along with a remarkably more accurate mortality prediction capacity for the SAPS 3 scoring system.

These conflicting results may be particularly attributable to the unique case mix of our study population. In contrast to three preexisting ImCU studies [18–20] or the original investigations that introduced the SAPS 2 and SAPS 3 scoring systems [26, 27], our study enrolled predominantly patients with chronic or terminal renal and hepatic diseases. Among those, transplant candidates or patients after SOT were represented in high percentages. The SAPS 3 comprises more variables linked to gastrointestinal and hepatic disorders (GHD), i.e., cirrhosis, liver failure, severe pancreatitis, acute abdomen, total bilirubin, platelets, and Glasgow Coma Scale (GCS), compared to the SAPS 2, which takes into account only three GHD-related parameters, i.e., total bilirubin, GCS, and sodium. Thus, it is not surprisingly that the SAPS 3 showed superior accuracy in mortality prediction for liver failure and cirrhosis than the SAPS 2, in our cohort. Even though Dupont et al. reported controversial findings in cirrhotic patients, our data are in line with a recent study by Alegre et al. that also attested superior test performance for the SAPS 3 compared to the SAPS 2 in patients with liver cirrhosis [21, 34]. Furthermore, differences between present study results and previous research are emphasized in septic patients; here, we observed the most pronounced underestimation of mortality risk using either score. Even though the SAPS 3 revealed better performance in terms of SMR relative to the SAPS 2, both scores were deemed unsuitable to adequately assess the risk of death among septic patients in the present cohort. However, due to the restricted sample size of this subgroup, this result should be considered with caution.

In terms of evaluating calibration performance, we consciously did not use the Hosmer-Lemeshow goodness-of-fit C-test [35]. Even though this test has been frequently used to validate mortality risk prediction scores [5–7, 10–12, 14, 15, 17, 18, 20, 26, 27], it has also been repeatedly criticized for different limitations related to its interpretation. For example, the Hosmer-Lemeshow goodness-of-fit C-test describes only deviations between observed and expected mortality, but neither contains information about their direction or dimension nor identifies characteristics of subgroups or individual patients that cause such deviations [36, 37]. Furthermore, high P-values do not necessarily provide evidence for a good fit, as numerous conditions can generate high P-values, including poor test power [36, 38]. In contrast to the Hosmer-Lemeshow goodness-of-fit C-test, the Calibration belt relates the logits of predicted and observed mortality in a polynomial function, whereby confidence intervals can be computed and plotted as the eponymous Calibration belt [29, 30]. The graphic illustration can assess the goodness of fit without any supervised categorization. Moreover, directions and ranges of poor calibrations from the perfect fit of a model can be directly visualized. The Calibration belt has already been tested against the Hosmer-Lemeshow goodness-of-fit C-test with similar type I error rates [30] and has been previously used in a similar study context [16]. For comparison purposes, we still applied the Hosmer-Lemeshow goodness-of-fit C-test and attached these results as supporting information (S1 and S2 Figs) below. In fact, using this method, we calculated an acceptable calibration performance for both prediction models.

Clearly, we agree with Lucena et al. that differences in case mix exert a strong influence on calibration and discrimination performance of the SAPS 2 and the SAPS 3 [18]. This consideration may gain additional support from our previous work that detected superior performance for the SAPS 2 relative to the SAPS 3 in ICU patients with internistic (predominantly cardiovascular) disorders at the same university hospital [14]. Of note, recent research analyzing data from three ICUs with over 2500 patients concluded that scoring systems with more predictor variables (such as the SAPS 3 when compared to the SAPS 2) are likely to achieve better overall performance relative to those with fewer [13].

Finally yet importantly, several characteristics and limitations of the present research warrant special attention. First, both the relatively small sample size and the unique case mix of this retrospective single-center study may generate substantial differences in terms of reproducibility of the SAPS 2 and the SAPS 3 based assessments between present and previous cohorts [18, 26, 27]. Moreover, different admission and discharge criteria as well as non-uniform staff- and/or shift- work patterns might skew or strongly particularize our findings, thus rendering them likely not extrapolable to other ImCU populations and settings. Just in this respect, it deserves attention that our ImCU had a less favorable nurse-to-patient ratio compared with the study conducted by Lucena et al [18], i.e., 1:4 vs. 1:3, respectively. In view of almost equal scores for the SAPS 2 (31.4 ± 12.9 vs. 36.6 ± 11.9) and the SAPS 3 (55.4 ± 12.9 vs. 58.4 ± 15.4), this condition may have been at least co-responsible for higher in-hospital mortality rates (30.2% vs. 20.1%) reported in the present study compared with the study by Lucena et al [18].

Conclusions

The SAPS 3 (North Europe Logit) provided good discrimination and satisfactory calibration and was highly accurate in predicting mortality in our cohort of adults admitted to the ImCU due to chronic or terminal renal or gastrointestinal/hepatic disorders. The SAPS 2 appeared less suitable for risk evaluation in this setting due to a marked underestimation of mortality in comparison to the SAPS 3. Despite a large number of preexisting validation studies for SAPS 2 and SAPS 3 in ICU patients and (to a lesser degree also in) ImCU patients, predictive performance of these tools remains inconsistent and seems to be strongly influenced by variable case mix indices and other specific characteristics of the attending wards [10, 14, 18, 26, 27]. Furthermore, scoring systems are prone to be outdated over time as ImCU populations and structures change; this fate is driven by the evolution of diagnostic, therapeutic, technological, and economic means or strategies, which influence treatment outcomes and mortality. Thus, the present findings may not be easily generalizable to other populations. Especially, the poor performance of the SAPS 2 for sepsis and acute and/or chronic liver disorders seen in our analysis stresses the need for further and broader validation of these and other predictive ICU scores in different ImCU settings and populations during larger, prospective and well-designed studies.

Supporting information

S1 Fig [tif]
Calibration curve based in the Hosmer and Lemeshow goodness-of-fit C test for SAPS 2.

S2 Fig [tif]
Calibration curve based in the Hosmer and Lemeshow goodness-of-fit C test for SAPS 3.