Use of Expert Panels to Define the Reference Standard in Diagnostic Research: A Systematic Review of Published Methods and Reporting

English version České info

Background:
In diagnostic studies, a single and error-free test that can be used as the reference (gold) standard often does not exist. One solution is the use of panel diagnosis, i.e., a group of experts who assess the results from multiple tests to reach a final diagnosis in each patient. Although panel diagnosis, also known as consensus or expert diagnosis, is frequently used as the reference standard, guidance on preferred methodology is lacking. The aim of this study is to provide an overview of methods used in panel diagnoses and to provide initial guidance on the use and reporting of panel diagnosis as reference standard.

Methods and Findings:
PubMed was systematically searched for diagnostic studies applying a panel diagnosis as reference standard published up to May 31, 2012. We included diagnostic studies in which the final diagnosis was made by two or more persons based on results from multiple tests. General study characteristics and details of panel methodology were extracted. Eighty-one studies were included, of which most reported on psychiatry (37%) and cardiovascular (21%) diseases. Data extraction was hampered by incomplete reporting; one or more pieces of critical information about panel reference standard methodology was missing in 83% of studies. In most studies (75%), the panel consisted of three or fewer members. Panel members were blinded to the results of the index test results in 31% of studies. Reproducibility of the decision process was assessed in 17 (21%) studies. Reported details on panel constitution, information for diagnosis and methods of decision making varied considerably between studies.

Conclusions:
Methods of panel diagnosis varied substantially across studies and many aspects of the procedure were either unclear or not reported. On the basis of our review, we identified areas for improvement and developed a checklist and flow chart for initial guidance for researchers conducting and reporting of studies involving panel diagnosis.

Please see later in the article for the Editors' Summary

Vyšlo v časopise: Use of Expert Panels to Define the Reference Standard in Diagnostic Research: A Systematic Review of Published Methods and Reporting. PLoS Med 10(10): e32767. doi:10.1371/journal.pmed.1001531
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pmed.1001531

Souhrn

Zdroje

1. ReitsmaJB, RutjesAW, KhanKS, CoomarasamyA, BossuytPM (2009) A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. J Clin Epidemiol 62: 797–806.

2. HadguA, DendukuriN, HildenJ (2005) Evaluation of nucleic acid amplification tests in the absence of a perfect gold-standard test: a review of the statistical and epidemiologic issues. Epidemiology 16: 604–612.

3. AlonzoTA, PepeMS (1999) Using a combination of reference tests to assess the accuracy of a new diagnostic test. Stat Med 18: 2987–3003.

4. PepeMS, JanesH (2007) Insights into latent class analysis of diagnostic test performance. Biostatistics 8: 474–484.

5. BaughmanAL, BisgardKM, CorteseMM, ThompsonWW, SandenGN, et al. (2008) Utility of composite reference standards and latent class analysis in evaluating the clinical accuracy of diagnostic tests for pertussis. Clin Vaccine Immunol 15: 106–114.

6. MoonsKG, GrobbeeDE (2002) When should we remain blind and when should our eyes remain open in diagnostic studies? J Clin Epidemiol 55: 633–636.

7. MagazinerJ, ZimmermanSI, GermanPS, KuhnK, MayC, et al. (1996) Ascertaining dementia by expert panel in epidemiologic studies of nursing home residents. Ann Epidemiol 6: 431–437.

8. LiberatiA, AltmanDG, TetzlaffJ, MulrowC, GotzschePC, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med 6: e1000100 doi:10.1371/journal.pmed.1000100

9. LilfordRJ, RichardsonA, StevensA, FitzpatrickR, EdwardsS, et al. (2001) Issues in methodological research: perspectives from researchers and commissioners. Health Technol Assess 2001 5: 1–57.

10. BossuytPM, ReitsmaJB, BrunsDE, GatsonisCA, GlasziouPP, et al. (2003) The STARD statement for reporting studies of diagnostic accuracy: explanation and elaboration. Ann Intern Med 138: W1–W12.

11. WhitingPF, RutjesAW, WestwoodME, MallettS, Deeks JJJB, et al. (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 155: 529–536.

12. BairdG, SimonoffE, PicklesA, ChandlerS, LoucasT, et al. (2006) Prevalence of disorders of the autism spectrum in a population cohort of children in South Thames: the Special Needs and Autism Project (SNAP). Lancet 368: 210–215.

13. BienvenuOJ, SamuelsJF, RiddleMA, Hoehn-SaricR, LiangKY, et al. (2000) The relationship of obsessive-compulsive disorder to possible spectrum disorders: results from a family study. Biol Psychiatry 48: 287–293.

14. BoustaniM, CallahanCM, UnverzagtFW, AustromMG, PerkinsAJ, et al. (2005) Implementing a screening and diagnosis program for dementia in primary care. J Gen Intern Med 20: 572–577.

15. BrughaTS, McManusS, SmithJ, ScottFJ, MeltzerH, et al. (2012) Validating two survey methods for identifying cases of autism spectrum disorder among adults in the community. Psychol Med 42: 647–656.

16. Carnero-PardoC, Espejo-MartinezB, Lopez-AlcaldeS, Espinosa-GarciaM, Saez-ZeaC, et al. (2011) Diagnostic accuracy, effectiveness and cost for cognitive impairment and dementia screening of three short cognitive tests applicable to illiterates. PLoS One 6: e27069 doi:10.1371/journal.pone.0027069

17. ClassCA, UnverzagtFW, GaoS, HallKS, BaiyewaO, et al. (1996) Psychiatric disorders in African American nursing home residents. Am J Psychiatry 153: 677–681.

18. de KoningHJ, de Ridder-SluiterJG, van AgtHM, Reep-van den BerghCM, van der StegeHA, et al. (2004) A cluster-randomised trial of screening for language disorders in toddlers. J Med Screen 11: 109–116.

19. DrakeRE, OsherFC, NoordsyDL, HurlbutSC, TeagueGB, et al. (1990) Diagnosis of alcohol use disorders in schizophrenia. Schizophr Bull 16: 57–67.

20. DubersteinPR, MaY, ChapmanBP, ConwellY, McGriffJ, et al. (2011) Detection of depression in older adults by family and friends: distinguishing mood disorder signals from the noise of personality and everyday life. Int Psychogeriatr 23: 634–643.

21. FennigS, CraigTJ, Tanenberg-KarantM, BrometEJ (1994) Comparison of facility and research diagnoses in first-admission psychotic patients. Am J Psychiatry 151: 1423–1429.

22. FladbyT, SchusterM, GronliO, SjoholmH, LosethS, et al. (1999) Organic brain disease in psychogeriatric patients: impact of symptoms and screening methods on the diagnostic process. J Geriatr Psychiatry Neurol 12: 16–20.

23. GirardC, SimardM, NoiseuxR, LaplanteL, DugasM, et al. (2011) Late-onset-psychosis: cognition. Int Psychogeriatr 23: 1301–1316.

24. Graff-RadfordNR, FermanTJ, LucasJA, JohnsonHK, ParfittFC, et al. (2006) A cost effective method of identifying and recruiting persons over 80 free of dementia or mild cognitive impairment. Alzheimer Dis Assoc Disord 20: 101–104.

25. GulevichSJ, ConwellTD, LaneJ, LockwoodB, SchwettmannRS, et al. (1997) Stress infrared telethermography is useful in the diagnosis of complex regional pain syndrome, type I (formerly reflex sympathetic dystrophy). Clin J Pain 13: 50–59.

26. HallKS, GaoS, BaiyewuO, LaneKA, GurejeO, et al. (2009) Prevalence rates for dementia and Alzheimer's disease in African Americans: 1992 versus 2001. Alzheimers Dement 5: 227–233.

27. JohnsonS, HollisC, HennessyE, KochharP, WolkeD, et al. (2011) Screening for autism in preterm children: diagnostic utility of the Social Communication Questionnaire. Arch Dis Child 96: 73–77.

28. LaurilaJV, PitkalaKH, StrandbergTE, TilvisRS (2004) Delirium among patients with and without dementia: does the diagnosis according to the DSM-IV differ from the previous classifications? Int J Geriatr Psychiatry 19: 271–277.

29. MagazinerJ, GermanP, ZimmermanSI, HebelJR, BurtonL, et al. (2000) The prevalence of dementia in a statewide sample of new nursing home admissions aged 65 and older: diagnosis by expert panel. Epidemiology of Dementia in Nursing Homes Research Group. Gerontologist 40: 663–672.

30. MillerPR, DasherR, CollinsR, GriffithsP, BrownF (2001) Inpatient diagnostic assessments: 1. Accuracy of structured vs. unstructured interviews. Psychiatry Res 105: 255–264.

31. OgunniyiA, DaifAK, Al-RajehS, AbdulJabbarM, Al-TahanAR, et al. (1998) Dementia in Saudi Arabia: experience from a university hospital. Acta Neurol Scand 98: 116–120.

32. OgunniyiA, LaneKA, BaiyewuO, GaoS, GurejeO, et al. (2011) Hypertension and incident dementia in community-dwelling elderly Yoruba Nigerians. Acta Neurol Scand 124: 396–402.

33. PlassmanBL, LangaKM, FisherGG, HeeringaSG, WeirDR, et al. (2007) Prevalence of dementia in the United States: the aging, demographics, and memory study. Neuroepidemiology 29: 125–132.

34. PlassmanBL, LangaKM, McCammonRJ, FisherGG, PotterGG, et al. (2011) Incidence of dementia and cognitive impairment, not dementia in the United States. Ann Neurol 70: 418–426.

35. PotterGG, PlassmanBL, BurkeJR, KabetoMU, LangaKM, et al. (2009) Cognitive performance and informant reports in the diagnosis of cognitive impairment and dementia in African Americans and whites. Alzheimers Dement 5: 445–453.

36. SachdevPS, BrodatyH, ValenzuelaMJ, LorentzL, LooiJC, et al. (2006) Clinical determinants of dementia and mild cognitive impairment following ischaemic stroke: the Sydney Stroke Study. Dement Geriatr Cogn Disord 21: 275–283.

37. SteenlandNK, AumanCM, PatelPM, BartellSM, GoldsteinFC, et al. (2008) Development of a rapid screening instrument for mild cognitive impairment and undiagnosed dementia. J Alzheimers Dis 15: 419–427.

38. Tanenberg-KarantM, FennigS, RamR, KrishnaJ, JandorfL, et al. (1995) Bizarre delusions and first-rank symptoms in a first-admission sample: a preliminary analysis of prevalence and correlates. Compr Psychiatry 36: 428–434.

39. WeintraubD, RaskinA, RuskinPE, Gruber-BaldiniAL, ZimmermanSI, et al. (2000) Racial differences in the prevalence of dementia among patients admitted to nursing homes. Psychiatr Serv 51: 1259–1264.

40. WienerP, AlexopoulosGS, KakumaT, MeyersBS, RosenthalE, et al. (1997) The limits of history-taking in geriatric depression. Am J Geriatr Psychiatry 5: 116–125.

41. WilliamsJ, ScottF, StottC, AllisonC, BoltonP, et al. (2005) The CAST (Childhood Asperger Syndrome Test: test accuracy. Autism 9: 45–68.

42. AssomullRG, ShakespeareC, KalraPR, LloydG, GulatiA, et al. (2011) Role of cardiovascular magnetic resonance as a gatekeeper to invasive coronary angiography in patients presenting with heart failure of unknown etiology. Circulation 124: 1351–1360.

43. BosnerS, HaasenritterJ, BeckerA, KaratoliosK, VaucherP, et al. (2010) Ruling out coronary artery disease in primary care: development and validation of a simple prediction rule. CMAJ 182: 1295–1300.

44. DaoQ, KrishnaswamyP, KazanegraR, HarrisonA, AmirnovinR, et al. (2001) Utility of B-type natriuretic peptide in the diagnosis of congestive heart failure in an urgent-care setting. J Am Coll Cardiol 37: 379–385.

45. DoubalFN, DennisMS, WardlawJM (2011) Characteristics of patients with minor ischaemic strokes and negative MRI: a cross-sectional study. J Neurol Neurosurg Psychiatry 82: 540–542.

46. GaikwadAB, MudalgiBA, PatankarKB, PatilJK, GhongadeDV (2008) Diagnostic role of 64-slice multidetector row CT scan and CT venogram in cases of cerebral venous thrombosis. Emerg Radiol 15: 325–333.

47. HoffmannR, BorgesAC, KasprzakJD, vonBS, FirschkeC, et al. (2007) Analysis of myocardial perfusion or myocardial function for detection of regional myocardial abnormalities. An echocardiographic multicenter comparison study using myocardial contrast echocardiography and 2D echocardiography. Eur J Echocardiogr 8: 438–448.

48. HoffmannR, vonBS, KasprzakJD, BorgesAC, tenCF, et al. (2006) Analysis of regional left ventricular function by cineventriculography, cardiac magnetic resonance imaging, and unenhanced and contrast-enhanced echocardiography: a multicenter comparison of methods. J Am Coll Cardiol 47: 121–128.

49. HoffmannU, NagurneyJT, MoselewskiF, PenaA, FerencikM, et al. (2006) Coronary multidetector computed tomography in the assessment of patients with acute chest pain. Circulation 114: 2251–2260.

50. HoffmannU, PenaAJ, MoselewskiF, FerencikM, AbbaraS, et al. (2006) MDCT in early triage of patients with acute chest pain. Am J Roentgenol 187: 1240–1247.

51. KantarciM, CevizN, SevimliS, BayraktutanU, CeyhanE, et al. (2007) Diagnostic performance of multidetector computed tomography for detecting aorto-ostial lesions compared with catheter coronary angiography: multidetector computed tomography coronary angiography is superior to catheter angiography in detection of aorto-ostial lesions. J Comput Assist Tomogr 31: 595–599.

52. KelderJC, CramerMJ, VerweijWM, GrobbeeDE, HoesAW (2011) Clinical utility of three B-type natriuretic peptide assays for the initial diagnostic assessment of new slow-onset heart failure. J Card Fail 17: 729–734.

53. KelderJC, CramerMJ, RuttenFH, PlokkerHW, GrobbeeDE, et al. (2011) The furosemide diagnostic test in suspected slow-onset heart failure: popular but not useful. Eur J Heart Fail 13: 513–517.

54. LinnJ, Ertl-WagnerB, SeelosKC, StruppM, ReiserM, et al. (2007) Diagnostic value of multidetector-row CT angiography in the evaluation of thrombosis of the cerebral venous sinuses. AJNR Am J Neuroradiol 28: 946–952.

55. NordenholzKE, ZieskeM, DyerDS, HansonJA, HeardK (2007) Radiologic diagnoses of patients who received imaging for venous thromboembolism despite negative D-dimer tests. Am J Emerg Med 25: 1040–1046.

56. OudejansI, MosterdA, BloemenJA, ValkMJ, van VelzenE, et al. (2011) Clinical evaluation of geriatric outpatients with suspected heart failure: value of symptoms, signs, and additional tests. Eur J Heart Fail 13: 518–527.

57. Remy-JardinM, RemyJ, MassonP, BonnelF, DebatselierP, et al. (2000) CT angiography of thoracic outlet syndrome: evaluation of imaging protocols for the detection of arterial stenosis. J Comput Assist Tomogr 24: 349–361.

58. TrevelyanJ, NeedhamEW, SmithSC, MattuRK (2003) Sources of diagnostic inaccuracy of conventional versus new diagnostic criteria for myocardial infarction in an unselected UK population with suspected cardiac chest pain, and investigation of independent prognostic variables. Heart 89: 1406–1410.

59. BroekhuizenBD, SachsAP, HoesAW, MoonsKG, van den BergJW, et al. (2010) Undetected chronic obstructive pulmonary disease and asthma in people over 50 years with persistent cough. Br J Gen Pract 60: 489–494.

60. BroekhuizenBD, SachsAP, MoonsKG, CheragwandiSA, DamsteHE, et al. (2011) Diagnostic value of oral prednisolone test for chronic obstructive pulmonary disorders. Ann Fam Med 9: 104–109.

61. ChavannesN, SchermerT, AkkermansR, JacobsJE, van de GraafG, et al. (2004) Impact of spirometry on GPs' diagnostic differentiation and decision-making. Respir Med 98: 1124–1130.

62. GauvinF, DassaC, ChaibouM, ProulxF, FarrellCA, et al. (2003) Ventilator-associated pneumonia in intubated children: comparison of different diagnostic methods. Pediatr Crit Care Med 4: 437–443.

63. GuderG, BrennerS, AngermannCE, ErtlG, HeldM, et al. (2012) “GOLD or lower limit of normal definition? A comparison with expert-based diagnosis of chronic obstructive pulmonary disease in a prospective cohort-study”. Respir Res 13: 13.

64. Mohamed HoeseinFA, ZanenP, SachsAP, VerheijTJ, LammersJW, et al. (2012) Spirometric Thresholds for Diagnosing COPD: 0.70 or LLN, Pre- or Post-dilator Values? COPD 9: 338–343.

65. ReinartzP, WildbergerJE, SchaeferW, NowakB, MahnkenAH, et al. (2004) Tomographic imaging in the diagnosis of pulmonary embolism: a comparison between V/Q lung scintigraphy in SPECT technique and multislice spiral CT. J Nucl Med 45: 1501–1508.

66. ReinartzP, KaiserHJ, WildbergerJE, GordjiC, NowakB, et al. (2006) SPECT imaging in the diagnosis of pulmonary embolism: automated detection of match and mismatch defects by means of image-processing techniques. J Nucl Med 47: 968–973.

67. Szucs-FarkasZ, SchallerC, BenslerS, PatakMA, VockP, et al. (2009) Detection of pulmonary emboli with CT angiography at reduced radiation exposure and contrast material volume: comparison of 80 kVp and 120 kVp protocols in a matched cohort. Invest Radiol 44: 793–799.

68. ThiemeSF, GrauteV, NikolaouK, MaxienD, ReiserMF, et al. (2012) Dual Energy CT lung perfusion imaging–correlation with SPECT/CT. Eur J Radiol 81: 360–365.

69. GeirnaerdtMJ, HermansJ, BloemJL, KroonHM, PopeTL, et al. (1997) Usefulness of radiography in differentiating enchondroma from central grade 1 chondrosarcoma. AJR Am J Roentgenol 169: 1097–1104.

70. JorgensenF, FruergaardP, LaunbjergJ, AggestrupS, ElsborgL, et al. (1998) The diagnostic value of oesophageal radionuclide transit in patients admitted for but without acute myocardial infarction. Clin Physiol 18: 89–96.

71. MarshallJK, CawdronR, ZealleyI, RiddellRH, SomersS, et al. (2004) Prospective comparison of small bowel meal with pneumocolon versus ileo-colonoscopy for the diagnosis of ileal Crohn's disease. Am J Gastroenterol 99: 1321–1329.

72. MartinezFJ, StanopoulosI, AceroR, BeckerFS, PickeringR, et al. (1994) Graded comprehensive cardiopulmonary exercise testing in the evaluation of dyspnea unexplained by routine evaluation. Chest 105: 168–174.

73. RayP, BirolleauS, LefortY, BecqueminMH, BeigelmanC, et al. (2006) Acute respiratory failure in the elderly: etiology, emergency diagnosis and prognosis. Crit Care 10: R82.

74. RuttenFH, CramerMJ, GrobbeeDE, SachsAP, KirkelsJH, et al. (2005) Unrecognized heart failure in elderly patients with stable chronic obstructive pulmonary disease. Eur Heart J 26: 1887–1894.

75. WhiteCS, KuoD, KelemenM, JainV, MuskA, et al. (2005) Chest pain evaluation in the emergency department: can MDCT provide a comprehensive evaluation? AJR Am J Roentgenol 185: 533–540.

76. AmourJ, BirenbaumA, LangeronO, LeMY, BertrandM, et al. (2008) Influence of renal dysfunction on the accuracy of procalcitonin for the diagnosis of postoperative infection after vascular surgery. Crit Care Med 36: 1147–1154.

77. BisulliF, VignatelliL, NaldiI, PittauF, ProviniF, et al. (2012) Diagnostic accuracy of a structured interview for nocturnal frontal lobe epilepsy (SINFLE: a proposal for developing diagnostic criteria. Sleep Med 13: 81–87.

78. Gamez-DiazLY, EnriquezLE, MatuteJD, VelasquezS, GomezID, et al. (2011) Diagnostic accuracy of HMGB-1, sTREM-1, and CD64 as markers of sepsis in patients recently admitted to the emergency department. Acad Emerg Med 18: 807–815.

79. HamH, McInnesMD, WooM, LemondeS (2012) Negative predictive value of intravenous contrast-enhanced CT of the abdomen for patients presenting to the emergency department with undifferentiated upper abdominal pain. Emerg Radiol 191: 19–26.

80. HardieAD, NaikM, HechtEM, ChandaranaH, MannelliL, et al. (2010) Diagnosis of liver metastases: value of diffusion-weighted MRI compared with gadolinium-enhanced MRI. Eur Radiol 20: 1431–1441.

81. HumphriesPD, SimpsonJC, CreightonSM, Hall-CraggsMA (2008) MRI in the assessment of congenital vaginal anomalies. Clin Radiol 63: 442–448.

82. LinWC, UppotRN, LiCS, HahnPF, SahaniDV (2007) Value of automated coronal reformations from 64-section multidetector row computerized tomography in the diagnosis of urinary stone disease. J Urol 178: 907–911.

83. O'TooleRV, CoxG, ShanmuganathanK, CastilloRC, TurenCH, et al. (2010) Evaluation of computed tomography for determining the diagnosis of acetabular fractures. J Orthop Trauma 24: 284–290.

84. OtteJA, GeelkerkenRH, OostveenE, MensinkPB, HuismanAB, et al. (2005) Clinical impact of gastric exercise tonometry on diagnosis and management of chronic gastrointestinal ischemia. Clin Gastroenterol Hepatol 3: 660–666.

85. PenzkoferAK, PflugerT, PochmannY, MeissnerO, LeinsingerG (2002) MR imaging of the brain in pediatric patients: diagnostic value of HASTE sequences. AJR Am J Roentgenol 179: 509–514.

86. RobinTA, MullerA, RaitJ, KeeffeJE, TaylorHR, et al. (2005) Performance of community-based glaucoma screening using Frequency Doubling Technology and Heidelberg Retinal Tomography. Ophthalmic Epidemiol 12: 167–178.

87. TadrosAM, LunsjoK, CzechowskiJ, CorrP, bu-ZidanFM (2007) Usefulness of different imaging modalities in the assessment of scapular fractures caused by blunt trauma. Acta Radiol 48: 71–75.

88. TepperSJ, DahlofCG, DowsonA, NewmanL, MansbachH, et al. (2004) Prevalence and diagnosis of migraine in patients consulting their physician with a complaint of headache: data from the Landmark Study. Headache 44: 856–864.

89. ThabutD, D'AmicoG, TanP, De FrancisR, FabriciusS, et al. (2010) Diagnostic performance of Baveno IV criteria in cirrhotic patients with upper gastrointestinal bleeding: analysis of the F7 liver-1288 study population. J Hepatol 53: 1029–1034.

90. van RandenA, LamerisW, van EsHW, van HeesewijkHP, van RamhorstB, et al. (2011) A comparison of the accuracy of ultrasound and computed tomography in common diagnoses causing acute abdominal pain. Eur Radiol 21: 1535–1545.

91. WeihLM, NanjanM, McCartyCA, TaylorHR (2001) Prevalence and predictors of open-angle glaucoma: results from the visual impairment project. Ophthalmology 108: 1966–1972.

92. WhiteleyWN, WardlawJM, DennisMS, SandercockPA (2011) Clinical scores for the identification of stroke and transient ischaemic attack in the emergency department: a cross-sectional study. J Neurol Neurosurg Psychiatry 82: 1006–1010.

93. GabelMJ, ShipanCR (2004) A social choice approach to expert consensus panels. J Health Econ 23: 543–564.

94. de GrootJA, BossuytPM, ReitsmaJB, RutjesAW, DendukuriN, et al. (2011) Verification problems in diagnostic accuracy studies: consequences and solutions. BMJ 343: d4770.

95. BankierAA, LevineD, HalpernEF, KresselHY (2010) Consensus interpretation in imaging research: is there a better way? Radiology 257: 14–17.

96. ObuchowskiNA, ZeppRC (1996) Simple steps for improving multiple-reader studies in radiology. AJR Am J Roentgenol 166: 517–521.

97. RansohoffDF, FeinsteinAR (1978) Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N Engl J Med 299: 926–930.

98. RutjesAW, ReitsmaJB, CoomarasamyA, KhanKS, BossuytPM (2007) Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 11: iii, ix-51.