Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist

1. ReillyBM, EvansAT (2006) Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med 144: 201–209.

2. BouwmeesterW, ZuithoffNP, MallettS, GeerlingsMI, VergouweY, et al. (2012) Reporting and methods in clinical prediction research: a systematic review. PLoS Med 9: 1–12.

3. SteyerbergEW, MoonsKG, van der WindtDA, HaydenJA, PerelP, et al. (2013) Prognosis Research Strategy (PROGRESS) 3: prognostic model research. PLoS Med 10: e1001381.

4. WellsPS, HirshJ, AndersonDR, LensingAW, FosterG, et al. (1998) A simple clinical model for the diagnosis of deep-vein thrombosis combined with impedance plethysmography: potential for an improvement in the diagnostic process. J Intern Med 243: 15–23.

5. OudegaR, MoonsKG, HoesAW (2005) Ruling out deep venous thrombosis in primary care. A simple diagnostic algorithm including D-dimer testing. Thromb Haemost 94: 200–205.

6. StiellIG, GreenbergGH, McKnightRD, NairRC, McDowellI, et al. (1993) Decision rules for the use of radiography in acute ankle injuries. Refinement and prospective validation. JAMA 269: 1127–1132.

7. RietveldRP, ter RietG, BindelsPJ, SloosJH, van WeertHC (2004) Predicting bacterial cause in infectious conjunctivitis: cohort study on informativeness of combinations of signs and symptoms. BMJ 329: 206–210.

8. GaleaMH, BlameyRW, ElstonCE, EllisIO (1992) The Nottingham Prognostic Index in primary breast cancer. Breast Cancer Res Treat 22: 207–219.

9. NashefSA, RoquesF, MichelP, GauducheauE, LemeshowS, et al. (1999) European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 16: 9–13.

10. WilsonPW, D'AgostinoRB, LevyD, BelangerAM, SilbershatzH, et al. (1998) Prediction of coronary heart disease using risk factor categories. Circulation 97: 1837–1847.

11. LindstromJ, TuomilehtoJ (2003) The diabetes risk score: a practical tool to predict type 2 diabetes risk. Diabetes Care 26: 725–731.

12. PerelP, EdwardsP, WentzR, RobertsI (2006) Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak 6: 38.

13. Altman (2007) Prognostic models: a methodological framework and review of models for breast cancer. In: Lyman GH, Burstein HJ, editor. Breast cancer Translational therapeutic strategies. New York Informa Healtcare. pp. 11–25.

14. van DierenS, BeulensJW, KengneAP, PeelenLM, RuttenGE, et al. (2012) Prediction models for the risk of cardiovascular disease in patients with type 2 diabetes: a systematic review. Heart 98: 360–369.

15. CollinsGS, MallettS, OmarO, YuLM (2011) Developing risk prediction models for type 2 diabetes: a systematic review of methodology and reporting. BMC Med 9: 103.

16. EttemaRG, PeelenLM, SchuurmansMJ, NierichAP, KalkmanCJ, et al. (2010) Prediction models for prolonged intensive care unit stay after cardiac surgery: systematic review and validation study. Circulation 122: 682–689.

17. RabarS, LauR, O'FlynnN, LiL, BarryP (2012) Risk assessment of fragility fractures: summary of NICE guidance. BMJ 345: e3698.

18. GoffDCJr, Lloyd-JonesDM, BennettG, CoadyS, D'AgostinoRBSr, et al. (2013) 2013 ACC/AHA Guideline on the Assessment of Cardiovascular Risk: A Report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. Circulation 129: S49–S73.

19. Riley RD, Ridley G, Williams K, Altman DG, Hayden J, et al.. (2007) Prognosis research: toward evidence-based results and a Cochrane methods group. J Clin Epidemiol 60 : 863–865; author reply 865–866.

20. HemingwayH (2006) Prognosis research: why is Dr. Lydgate still waiting? J Clin Epidemiol 59: 1229–1238.

21. InguiBJ, RogersMA (2001) Searching for clinical prediction rules in MEDLINE. J Am Med Inform Assoc 8: 391–397.

22. WongSS, WilczynskiNL, HaynesRB, RamkissoonsinghR (2003) Hedges Team (2003) Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE. AMIA Annu Symp Proc 2003: 728–732.

23. KeoghC, WallaceE, O'BrienKK, MurphyPJ, TeljeurC, et al. (2011) Optimized retrieval of primary care clinical prediction rules from MEDLINE to establish a Web-based register. J Clin Epidemiol 64: 848–860.

24. GeersingGJ, BouwmeesterW, ZuithoffP, SpijkerR, LeeflangM, et al. (2012) Search filters for finding prognostic and diagnostic prediction studies in Medline to enhance systematic reviews. PLoS ONE 7: e32844.

25. HaydenJA, CoteP, BombardierC (2006) Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med 144: 427–437.

26. HaydenJA, van der WindtDA, CartwrightJL, CoteP, BombardierC (2013) Assessing bias in studies of prognostic factors. Ann Intern Med 158: 280–286.

27. CounsellC, DennisM (2001) Systematic review of prognostic models in patients with acute stroke. Cerebrovasc Dis 12: 159–170.

28. TamarizLJ, EngJ, SegalJB, KrishnanJA, BolgerDT, et al. (2004) Usefulness of clinical prediction rules for the diagnosis of venous thromboembolism: A systematic review. Am J Med 117: 676.

29. LeushuisE, van der SteegJW, SteuresP, BossuytPM, EijkemansMJ, et al. (2009) Prediction models in reproductive medicine: a critical appraisal. Hum Reprod Update 15: 537–552.

30. MallettS, TimmerA, SauerbreiW, AltmanDG (2010) Reporting of prognostic studies of tumour markers: a review of published articles in relation to REMARK guidelines. Br J Cancer 102: 173–180.

31. MoonsKG, AltmanDG, VergouweY, RoystonP (2009) Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338: 1487–1490.

32. MoonsKG, KengneAP, GrobbeeDE, RoystonP, VergouweY, et al. (2012) Risk prediction models: II. External validation, model updating, and impact assessment. Heart 98: 691–698.

33. MoherD, HopewellS, SchulzKF, MontoriV, GotzschePC, et al. (2010) CONSORT 2010 Explanation and Elaboration: Updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol 63: e1–37.

34. McShaneLM, AltmanDG, SauerbreiW, TaubeSE, GionM, et al. (2005) REporting recommendations for tumour MARKer prognostic studies (REMARK). Br J Cancer 93: 387–391.

35. BossuytPM, ReitsmaJB, BrunsDE, GatsonisCA, GlasziouPP, et al. (2003) The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Clin Chem 49: 7–18.

36. von ElmE, AltmanDG, EggerM, PocockSJ, GotzschePC, et al. (2007) The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Ann Intern Med 147: 573–577.

37. JanssensAC, IoannidisJP, BedrosianS, BoffettaP, DolanSM, et al. (2011) Strengthening the reporting of genetic risk prediction studies (GRIPS): explanation and elaboration. Eur J Epidemiol 26: 313–337.

38. LiberatiA, AltmanDG, TetzlaffJ, MulrowC, GotzschePC, et al. (2009) The PRISMA Statement for Reporting Systematic Reviews and Meta-Analyses of Studies That Evaluate Health Care Interventions: Explanation and Elaboration. Ann Intern Med 151: W65–W94.

39. HigginsJP, AltmanDG, GotzschePC, JuniP, MoherD, et al. (2011) The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 343: d5928.

40. WhitingP, RutjesAW, ReitsmaJB, BossuytPM, KleijnenJ (2003) The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol 3: 25.

41. WhitingPF, RutjesAW, WestwoodME, MallettS, DeeksJJ, et al. (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155: 529–536.

42. AltmanDG (2001) Systematic reviews of evaluations of prognostic variables. BMJ 323: 224–228.

43. KyzasPA, Denaxa-KyzaD, IoannidisJP (2007) Quality of reporting of cancer prognostic marker studies: association with reported prognostic effect. J Natl Cancer Inst 99: 236–243.

44. SiontisGC, TzoulakiI, SiontisKC, IoannidisJP (2012) Comparisons of established risk prediction models for cardiovascular disease: systematic review. BMJ 344: e3318.

45. ShariatSF, KarakiewiczPI, MargulisV, KattanMW (2008) Inventory of prostate cancer predictive tools. Curr Opin Urol 18: 279–296.

46. VeerbeekJM, KwakkelG, van WegenEE, KetJC, HeymansMW (2011) Early prediction of outcome of activities of daily living after stroke: a systematic review. Stroke 42: 1482–1488.

47. MoonsKG, RoystonP, VergouweY, GrobbeeDE, AltmanDG (2009) Prognosis and prognostic research: what, why, and how? BMJ 338: 1317–1320.

48. LaupacisA, SekarN, StiellIG (1997) Clinical prediction rules. A review and suggested modifications of methodological standards. JAMA 277: 488–494.

49. AltmanDG, RoystonP (2000) What do we mean by validating a prognostic model? Stat Med 19: 453–473.

50. McGinnTG, GuyattGH, WyerPC, NaylorCD, StiellIG, et al. (2000) Users' guides to the medical literature: XXII: how to use articles about clinical decision rules. Evidence-Based Medicine Working Group. JAMA 284: 79–84.

51. Harrell FE (2001) Regression Modeling Strategies. New York: Springer-Verlag.

52. MoonsKG, GrobbeeDE (2002) Diagnostic studies as multivariable, prediction research. J Epidemiol Community Health 56: 337–338.

53. JanssenKJ, MoonsKG, KalkmanCJ, GrobbeeDE, VergouweY (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61: 76–86.

54. Grobbee DE, Hoes AW (2009) Clinical Epidemiology - Principles, Methods and Applications for Clinical Research. London: Jones and Bartlett Publishers. 413 pp.

55. RoystonP, MoonsKG, AltmanDG, VergouweY (2009) Prognosis and prognostic research: Developing a prognostic model. BMJ 338: b604.

56. AltmanDG, VergouweY, RoystonP, MoonsKG (2009) Prognosis and prognostic research: validating a prognostic model. BMJ 338: 1432–1435.

57. McGeechanK, MacaskillP, IrwigL, LiewG, WongTY (2008) Assessing new biomarkers and predictive models for use in clinical practice: a clinician's guide. Arch Intern Med 168: 2304–2310.

58. Steyerberg EW (2009) Clinical prediction models: A practical approach to development, validation, and updating; Gail M, Krickeberg K, Samet J, Tsiati A, Wong W, editors. Rotterdam: Springer. 497 p.

59. MoonsKG, KengneAP, WoodwardM, RoystonP, VergouweY, et al. (2012) Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker. Heart 98: 683–690.

60. MallettS, RoystonP, DuttonS, WatersR, AltmanDG (2010) Reporting methods in studies developing prognostic models in cancer: a review. BMC Med 8: 20.

61. JacobM, BrueggerD, ConzenP, BeckerBF, FinstererU, et al. (2005) Development and validation of a mathematical algorithm for quantifying preoperative blood volume by means of the decrease in hematocrit resulting from acute normovolemic hemodilution. Transfusion 45: 562–571.

62. CollinsGS, OmarO, ShanyindeM, YuLM (2013) A systematic review finds prediction models for chronic kidney disease were poorly reported and often developed using inappropriate methods. J Clin Epidemiol 66: 268–277.

63. VickersAJ, CroninAM (2010) Everything you always wanted to know about evaluating prediction models (but were too afraid to ask). Urology 76: 1298–1301.

64. BollandMJ, JacksonR, GambleGD, GreyA (2013) Discrepancies in predicted fracture risk in elderly people. BMJ 346: e8669.

65. GannaA, ReillyM, de FaireU, PedersenN, MagnussonP, et al. (2012) Risk prediction measures for case-cohort and nested case-control designs: an application to cardiovascular disease. Am J Epidemiol 175: 715–724.

66. BiesheuvelCJ, VergouweY, OudegaR, HoesAW, GrobbeeDE, et al. (2008) Advantages of the nested case-control design in diagnostic research. BMC Med Res Methodol 8: 48.

67. RutjesAW, ReitsmaJB, VandenbrouckeJP, GlasAS, BossuytPM (2005) Case-control and two-gate designs in diagnostic accuracy studies. Clin Chem 51: 1335–1341.

68. van ZaaneB, VergouweY, DondersAR, MoonsKG (2012) Comparison of approaches to estimate confidence intervals of post-test probabilities of diagnostic test results in a nested case-control study. BMC Med Res Methodol 12: 166.

69. OostenbrinkR, MoonsKG, BleekerSE, MollHA, GrobbeeDE (2003) Diagnostic research on routine care data: prospects and problems. J Clin Epidemiol 56: 501–506.

70. GeominiP, KruitwagenR, BremerGL, CnossenJ, MolBW (2009) The accuracy of risk scores in predicting ovarian malignancy: a systematic review. Obstet Gynecol 113: 384–394.

71. StiellIG, WellsGA (1999) Methodologic standards for the development of clinical decision rules in emergency medicine. Ann Emerg Med 33: 437–447.

72. AltmanDG, SchulzKF, MoherD, EggerM, DavidoffF, et al. (2001) The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med 134: 663–694.

73. KnottnerusJA (2002) Challenges in dia-prognostic research. J Epidemiol Community Health 56: 340–341.

74. OudegaR, MoonsKG, HoesAW (2005) Limited value of patient history and physical examination in diagnosing deep vein thrombosis in primary care. Fam Pract 22: 86–91.

75. BeneciukJM, BishopMD, GeorgeSZ (2009) Clinical prediction rules for physical therapy interventions: a systematic review. Phys Ther 89: 114–124.

76. MinneL, EslamiS, de KeizerN, de JongeE, de RooijSE, et al. (2012) Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Med 38: 40–46.

77. CollinsGS, AltmanDG (2012) Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2. BMJ 344: e4181.

78. MallettS, RoystonP, WatersR, DuttonS, AltmanDG (2010) Reporting performance of prognostic models in cancer: a review. BMC Med 8: 21.

79. RutjesAW, ReitsmaJB, Di NisioM, SmidtN, van RijnJC, et al. (2006) Evidence of bias and variation in diagnostic accuracy studies. CMAJ 174: 469–476.

80. WhitingP, RutjesAW, ReitsmaJB, GlasAS, BossuytPM, et al. (2004) Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med 140: 189–202.

81. HessEP, ThiruganasambandamoorthyV, WellsGA, ErwinP, JaffeAS, et al. (2008) Diagnostic accuracy of clinical prediction rules to exclude acute coronary syndrome in the emergency department setting: a systematic review. CJEM 10: 373–382.

82. Ferreira-GonzalezI, BusseJW, Heels-AnsdellD, MontoriVM, AklEA, et al. (2007) Problems with use of composite end points in cardiovascular trials: systematic review of randomised controlled trials. Bmj 334: 786.

83. GlynnRJ, RosnerB (2004) Methods to evaluate risks for composite end points and their individual components. J Clin Epidemiol 57: 113–122.

84. GondrieMJ, JanssenKJ, MoonsKG, van der GraafY (2012) A simple adaptation method improved the interpretability of prediction models for composite end points. J Clin Epidemiol 65: 946–953.

85. LijmerJG, MolBW, HeisterkampS, BonselGJ, PrinsMH, et al. (1999) Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 282: 1061–1066.

86. MaguireJL, BoutisK, UlerykEM, LaupacisA, ParkinPC (2009) Should a head-injured child receive a head CT scan? A systematic review of clinical prediction rules. Pediatrics 124: e145–154.

87. Serrano LA, Hess EP, Bellolio MF, Murad MH, Montori VM, et al.. (2010) Accuracy and quality of clinical decision rules for syncope in the emergency department: a systematic review and meta-analysis. Ann Emerg Med 56 : 362–373 e361.

88. ReitsmaJB, RutjesAW, KhanKS, CoomarasamyA, BossuytPM (2009) A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol 62: 797–806.

89. RutjesA, ReitsmaJ, CoomarasamyA, KhanK, BossuytP (2007) Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 50: ix–51.

90. WalravenvC, DavisD, ForsterAJ, WellsGA (2004) Time-dependent bias was common in survival analyses published in leading clinical journals. J Clin Epidemiol 57: 672–682.

91. MoonsKG, GrobbeeDE (2002) When should we remain blind and when should our eyes remain open in diagnostic studies? J Clin Epidemiol 55: 633–636.

92. BennetteC, VickersA (2012) Against quantiles: categorization of continuous variables in epidemiologic research, and its discontents. BMC Med Res Methodol 12: 21.

93. RoystonP, AltmanDG, SauerbreiW (2006) Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med 25: 127–141.

94. AltmanDG, RoystonP (2006) The cost of dichotomising continuous variables. BMJ 332: 1080.

95. LeeflangMM, MoonsKG, ReitsmaJB, ZwindermanAH (2008) Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clin Chem 54: 729–737.

96. RoystonP, SauerbreiW, AltmanDG (2000) Modeling the effects of continuous risk factors. J Clin Epidemiol 53: 219–221.

97. ConcatoJ, PeduzziP, HolfordTR, FeinsteinAR (1995) Importance of events per independent variable in proportional hazards analysis. I. Background, goals, and general strategy. J Clin Epidemiol 48: 1495–1501.

98. PeduzziP, ConcatoJ, FeinsteinAR, HolfordTR (1995) Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol 48: 1503–1510.

99. PeduzziP, ConcatoJ, KemperE, HolfordTR, FeinsteinAR (1996) A simulation study of the number of events per variable in logistic regression analysis. J Clin Epidemiol 49: 1373–1379.

100. VittinghoffE, McCullochCE (2007) Relaxing the rule of ten events per variable in logistic and Cox regression. Am J Epidemiol 165: 710–718.

101. CourvoisierDS, CombescureC, AgoritsasT, Gayet-AgeronA, PernegerTV (2011) Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol 64: 993–1000.

102. VergouweY, SteyerbergEW, EijkemansMJ, HabbemaJD (2005) Substantial effective sample sizes were required for external validation studies of predictive logistic regression models. J Clin Epidemiol 58: 475–483.

103. BurtonA, AltmanDG (2004) Missing covariate data within cancer prognostic studies: a review of current reporting and proposed guidelines. Br J Cancer 91: 4–8.

104. MackinnonA (2010) The use and reporting of multiple imputation in medical research - a review. J Intern Med 268: 586–593.

105. LittleRJA (1992) Regression with missing X's: A review. JASA 87: 1227–1237.

106. MoonsKG, DondersRA, StijnenT, HarrellFEJr (2006) Using the outcome for imputation of missing predictor values was preferred. J Clin Epidemiol 59: 1092–1101.

107. GorelickMH (2006) Bias arising from missing data in predictive models. J Clin Epidemiol 59: 1115–1123.

108. WoodAM, WhiteIR, RoystonP (2008) How should variable selection be performed with multiply imputed data? Stat Med 27: 3227–3246.

109. JanssenKJ, DondersAR, HarrellFEJr, VergouweY, ChenQ, et al. (2010) Missing covariate data in medical research: to impute is better than to ignore. J Clin Epidemiol 63: 721–727.

110. JanssenKJ, VergouweY, DondersAR, HarrellFEJr, ChenQ, et al. (2009) Dealing with missing predictor values when applying clinical prediction models. Clin Chem 55: 994–1001.

111. VergouweY, RoystonP, MoonsKG, AltmanDG (2010) Development and validation of a prediction model with missing predictor data: a practical approach. J Clin Epidemiol 63: 205–214.

112. MarshallA, AltmanDG, RoystonP, HolderRL (2010) Comparison of techniques for handling missing covariate data within prognostic modelling studies: a simulation study. BMC Med Res Methodol 10: 7.

113. DondersAR, van der HeijdenGJ, StijnenT, MoonsKG (2006) Review: a gentle introduction to imputation of missing values. J Clin Epidemiol 59: 1087–1091.

114. SterneJA, WhiteIR, CarlinJB, SprattM, RoystonP, et al. (2009) Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ 338: b2393.

115. WhiteIR, RoystonP, WoodAM (2011) Multiple imputation using chained equations: Issues and guidance for practice. Stat Med 30: 377–399.

116. SunGW, ShookTL, KayGL (1996) Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis. J Clin Epidemiol 49: 907–916.

117. HarrellFE, LeeKL, MarkDB (1996) Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing erros. Stat Med 15: 361–387.

118. Houwelingen vanJC, Le CessieS (1990) Predictive value of statistical models. Stat Med 9: 1303–1325.

119. RoystonP, AltmanDG (2013) External validation of a cox prognostic model: principles and methods. BMC Med Res Methodol 13: 33.

120. SteyerbergEW, EijkemansMJ, HarrellFE, HabbemaJD (2001) Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 21: 45–56.

121. PeekN, ArtsDG, BosmanRJ, van der VoortPH, de KeizerNF (2007) External validation of prognostic models for critically ill patients required substantial sample sizes. J Clin Epidemiol 60: 491–501.

122. SteyerbergEW, VickersAJ, CookNR, GerdsT, GonenM, et al. (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21: 128–138.

123. CookNR (2008) Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 54: 17–23.

124. PencinaMJ, D'AgostinoRBSr, D'AgostinoRBJr, VasanRS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172 discussion 207–112.

125. Pepe MS, Feng Z, Gu JW (2008) Comments on ‘Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond’ by M. J. Pencina et al., Statistics in Medicine (DOI: 10.1002/sim.2929). Stat Med 27: 173–181.

126. PencinaMJ, D'AgostinoRBSr, SteyerbergEW (2011) Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 30: 11–21.

127. PepeMS (2011) Problems with risk reclassification methods for evaluating prediction models. Am J Epidemiol 173: 1327–1335.

128. VickersAJ, ElkinEB (2006) Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making 26: 565–574.

129. JusticeAC, CovinskyKE, BerlinJA (1999) Assessing the generalizability of prognostic information. Ann Intern Med 130: 515–524.

130. VergouweY, MoonsKG, SteyerbergEW (2010) External validity of risk models: Use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 172: 971–980.

131. SteyerbergEW, EijkemansMJ, HarrellFEJr, HabbemaJD (2000) Prognostic modelling with logistic regression analysis: a comparison of selection and estimation methods in small data sets. Stat Med 19: 1059–1079.

132. TollDB, JanssenKJ, VergouweY, MoonsKG (2008) Validation, updating and impact of clinical prediction rules: a review. J Clin Epidemiol 61: 1085–1094.

133. Houwelingen vanJC (2000) Validation, calibration, revision and combination of prognostic survival models. Stat Med 19: 3401–3415.

134. TzoulakiI, LiberopoulosG, IoannidisJP (2009) Assessment of claims of improved prediction beyond the Framingham risk score. JAMA 302: 2345–2352.

135. AltmanDG, LymanGH (1998) Methodological challenges in the evaluation of prognostic factors in breast cancer. Breast Cancer Res Treat 52: 289–303.

136. VergouweY, SteyerbergEW, de WitR, RobertsJT, KeizerHJ, et al. (2003) External validity of a prediction rule for residual mass histology in testicular cancer: an evaluation for good prognosis patients. Br J Cancer 88: 843–847.

137. HukkelhovenCW, RampenAJ, MaasAI, FaraceE, HabbemaJD, et al. (2006) Some prognostic models for traumatic brain injury were not valid. J Clin Epidemiol 59: 132–143.

138. CollinsGS, MoonsKG (2012) Comparing risk prediction models. BMJ 344: e3186.

139. SiregarS, GroenwoldRH, de HeerF, BotsML, van der GraafY, et al. (2012) Performance of the original EuroSCORE. Eur J Cardiothorac Surg 41: 746–754.

140. PetersSA, den RuijterHM, BotsML, MoonsKG (2012) Improvements in risk stratification for the occurrence of cardiovascular disease by imaging subclinical atherosclerosis: a systematic review. Heart 98: 177–184.