The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview

English version České info

Background:
There is considerable international interest in exploiting the potential of digital solutions to enhance the quality and safety of health care. Implementations of transformative eHealth technologies are underway globally, often at very considerable cost. In order to assess the impact of eHealth solutions on the quality and safety of health care, and to inform policy decisions on eHealth deployments, we undertook a systematic review of systematic reviews assessing the effectiveness and consequences of various eHealth technologies on the quality and safety of care.

Methods and Findings:
We developed novel search strategies, conceptual maps of health care quality, safety, and eHealth interventions, and then systematically identified, scrutinised, and synthesised the systematic review literature. Major biomedical databases were searched to identify systematic reviews published between 1997 and 2010. Related theoretical, methodological, and technical material was also reviewed. We identified 53 systematic reviews that focused on assessing the impact of eHealth interventions on the quality and/or safety of health care and 55 supplementary systematic reviews providing relevant supportive information. This systematic review literature was found to be generally of substandard quality with regards to methodology, reporting, and utility. We thematically categorised eHealth technologies into three main areas: (1) storing, managing, and transmission of data; (2) clinical decision support; and (3) facilitating care from a distance. We found that despite support from policymakers, there was relatively little empirical evidence to substantiate many of the claims made in relation to these technologies. Whether the success of those relatively few solutions identified to improve quality and safety would continue if these were deployed beyond the contexts in which they were originally developed, has yet to be established. Importantly, best practice guidelines in effective development and deployment strategies are lacking.

Conclusions:
There is a large gap between the postulated and empirically demonstrated benefits of eHealth technologies. In addition, there is a lack of robust research on the risks of implementing these technologies and their cost-effectiveness has yet to be demonstrated, despite being frequently promoted by policymakers and “techno-enthusiasts” as if this was a given. In the light of the paucity of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, it is vital that future eHealth technologies are evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle. Such evaluation should be characterised by careful attention to socio-technical factors to maximise the likelihood of successful implementation and adoption.

: Please see later in the article for the Editors' Summary

Vyšlo v časopise: The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview. PLoS Med 8(1): e32767. doi:10.1371/journal.pmed.1000387
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pmed.1000387

Souhrn

Zdroje

1. CatwellL

SheikhA

2009 Evaluating eHealth interventions. PLoS Med 6 e1000126 doi:10.1371/journal.pmed.1000126

2. Department of Health, Chairman, 2000 An organisation with a memory: report of an expert group of learning from adverse events in the NHS. London The Stationary Office

3. Department of Health, Chief Pharmaceutical Officer 2004 Building a safer NHS for patients: improving medication safety. London The Stationary Office

4. Institute of Medicine 2000 To err is human: building a safer health system. Washington (D.C.) National Academy Press

5. Institute of Medicine, Committee on Quality Health Care in America 2001 Crossing the quality chasm: a new health system for the 21st century. Washington (D.C.) National Academy Press

6. Institute of Medicine 2003 Patient safety: achieving a new standard for care. Washington (D.C.) National Academy Press

7. Institute of Medicine 2007 Preventing medication errors. Washington (D.C.) National Academy Press

8. CarJ

BlackA

AnandanC

CresswellK

PagliariC

2008 The impact of eHealth on the quality and safety of healthcare. Available: http://www.haps.bham.ac.uk/publichealth/cfhep/001.shtml. Accessed 3 December 2010

9. BravataDM

McDonaldKM

ShojaniaKG

SundaramV

OwensDK

2005 Challenges in systematic reviews: synthesis of topics related to the delivery, organization, and financing of health care. Ann Intern Med 142 1056 1065

10. HigginsJPT

GreenS

2009 Cochrane handbook for systematic reviews of interventions 5.0.2. The Cochrane Library. Available: http://www.cochrane-handbook.org. Accessed 3 December 2010

11. PopayP

RobertsH

SowdenA

PetticrewM

AraiL

RodgersM

2006 Guidance on the conduct of narrative synthesis in systematic reviews. A product from the ESRC Methods Programme. Lancaster Institute of Health Research

12. PagliariC

SloanD

GregorP

SullivanF

KahanJP

2004 EH1 E-Health scoping exercise. Review of wider Web-based information sources. Dundee University of Dundee

13. PagliariC

SloanD

GregorP

SullivanF

KahanJP

2004 EH1 E-Health scoping exercise. Review of the traditional research literature. Dundee University of Dundee

14. Critical Appraisal Skills Programme. Available: http://www.phru.nhs.uk/doc_links/s.reviews%20appraisal%20tool.pdf. Accessed 6 July 2010

15. AmmenwerthE

Schnell-InderstP

MachanC

SiebertU

2008 The effect of electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inform Assoc 15 585 600

16. AndersonD

FlynnK

1997 Picture archiving and communication systems: a systematic review of published studies of diagnostic accuracy, radiology work processes, outcomes of care, and cost. Available: http://www.research.va.gov/resources/pubs/docs/pacs.pdf Last accessed 3 December 2010

17. BalasEA

KrishnaS

KretschmerRA

CheekTR

LobachDF

2004 Computerized knowledge management in diabetes care. Med Care 42 610 621

18. BennettJW

GlasziouPP

2003 Computerised reminders and feedback in medication management: a systematic review of randomised controlled trials. Med J Aust 178 217 222

19. BryanC

BorenSA

2008 The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature. Inform Prim Care 16 79 91

20. Charvet-ProtatS

ThoralF

1998 Economic and organizational evaluation of an imaging network (PACS). J Radiol 79 1453 1459

21. ChatellierG

ColombetI

DegouletP

1998 An overview of the effect of computer-assisted management of anticoagulant therapy on the quality of anticoagulation. Int J Med Inform 49 311 320

22. ChaudhryB

WangJ

WuS

MaglioneM

MojicaW

2006 Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 144 742 752

23. ClampS

KeenS

2005 The value of electronic health records. Leeds Yorkshire Centre for Health Informatics, University of Leeds

24. DelpierreC

CuzinL

FillauxJ

AlvarezM

MassipP

2004 A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach? Int J Qual Health Care 16 407 416

25. DexheimerJW

TalbotTR

SandersDL

RosenbloomST

AronskyD

2008 Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. J Am Med Inform Assoc 15 311 320

26. DurieuxP

TrinquartL

ColombetI

NiesJ

WaltonR

2008 Computerized advice on drug dosage to improve prescribing practice. Cochrane Database Syst Rev CD002894

27. EslamiS

Abu-HannaA

de JongeE

de KeizerNF

2009 Tight glycemic control and computerized decision-support systems: a systematic review. Intensive Care Med 35 1505 1517

28. EslamiS

Abu-HannaA

de KeizerNF

2007 Evaluation of outpatient computerized physician medication order entry systems: a systematic review. J Am Med Inform Assoc 14 400 406

29. EslamiS

KeizerNF

Abu-HannaA

2008 The impact of computerized physician medication order entry in hospitalized patients-A systematic review. Int J Med Inform 77 365 376

30. FitzmauriceDA

HobbsFD

DelaneyBC

WilsonS

McManusR

1998 Review of computerized decision support systems for oral anticoagulation management. Br J Haematol 102 907 909

31. GargAX

AdhikariNK

McDonaldH

Rosas-ArellanoMP

DevereauxPJ

2005 Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 293 1223 1238

32. GeorgiouA

WilliamsonM

WestbrookJI

RayS

2007 The impact of computerised physician order entry systems on pathology services: a systematic review. Int J Med Inform 76 514 529

33. HaywardGL

ParnesAJ

SimonSR

2009 Using health information technology to improve drug monitoring: a systematic review. Pharmacoepidemiol Drug Saf 18 1232 1237

34. HenderK

2000 How effective are computer assisted decision support systems (CADSS) in improving clinical outcomes of patients? Available from: cce@southernhealth.org.au. Accessed 3 December 2010

35. HeselmansA

Van Der MeijdenMJ

DonceelP

AertgeertsB

RamaekersD

2009 Effectiveness of electronic guideline-based implementation systems in ambulatory care settings - a systematic review. Implement Sci 4 82

36. HiderP

2002 Electronic prescribing: a critical appraisal of the literature. Available at: http://nzhta.chmeds.ac.nz/publications/elpresc.pdf Accessed 3 December 2010

37. IraniJS

MiddletonJL

MarfatiaR

OmanaET

D'AmicoF

2009 The use of electronic health records in the exam room and patient satisfaction: a systematic review. J Am Board Fam Med 22 553 562

38. JamalA

McKenzieK

ClarkM

2009 The impact of health information technology on the quality of medical and health care: a systematic review. HIM J 38 26 37

39. JerantAF

HillDB

2000 Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? J Fam Pract 49 349 357

40. KaushalR

ShojaniaKG

BatesDW

2003 Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review. Arch Intern Med 163 1409 1416

41. MadorRL

ShawNT

2009 The impact of a Critical Care Information System (CCIS) on time spent charting and in direct patient care by staff in the ICU: a review of the literature. Int J Med Inform 78 435 445

42. MitchellE

SullivanF

2001 A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97. BMJ 322 279 282

43. MontgomeryAA

FaheyT

1998 A systematic review of the use of computers in the management of hypertension. J Epidemiol Community Health 52 520 525

44. NiazkhaniZ

PirnejadH

BergM

AartsJ

2009 The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review. J Am Med Inform Assoc 16 539 549

45. OrenE

ShafferER

GuglielmoBJ

2003 Impact of emerging technologies on medication errors and adverse drug events. Am J Health Syst Pharm 60 1447 1458

46. PearsonSA

MoxeyA

RobertsonJ

HainsI

WilliamsonM

2009 Do computerised clinical decision support systems for prescribing change practice? A systematic review of the literature (1990-2007). BMC Health Serv Res 9 154

47. PoissantL

PereiraJ

TamblynR

KawasumiY

2005 The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 12 505 516

48. RandellR

MitchellN

DowdingD

CullumN

ThompsonC

2007 Effects of computerized decision support systems on nursing performance and patient outcomes: a systematic review. J Health Serv Res Policy 12 242 249

49. ReckmannMH

WestbrookJI

KohY

LoC

DayRO

2009 Does computerized provider order entry reduce prescribing errors for hospital inpatients? A systematic review. J Am Med Inform Assoc 16 613 623

50. RothschildJ

2004 Computerized physician order entry in the critical care and general inpatient setting: a narrative review. J Crit Care 4 271 278

51. SchedlbauerA

PrasadV

MulvaneyC

PhansalkarS

StantonW

2009 What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc 16 531 538

52. ShachakA

ReisS

2009 The impact of electronic medical records on patient-doctor communication during consultation: a narrative literature review. J Eval Clin Pract 15 641 649

53. ShamliyanTA

DuvalS

DuJ

KaneRL

2008 Just what the doctor ordered. Review of the evidence of the impact of computerized physician order entry system on medication errors. Health Serv Res 43 32 53

54. SheblNA

FranklinBD

BarberN

2007 Clinical decision support systems and antibiotic use. Pharm World Sci 29 342 349

55. ShekellePG

GoldzweigCL

2009 Costs and benefits of health technology information: an updated systematic review. Available: http://www.health.org.uk/public/cms/75/76/313/564/Costs%20and%20benefits%20of%20health%20information%20technology.pdf?realName=urByVX.pdf. Accessed 3 December 2010

56. ShekellePG

MortonSC

KeelerEB

2006 Costs and benefits of health information technology. Available: http://www.ahrq.gov/downloads/pub/evidence/pdf/hitsyscosts/hitsys.pdf. Accessed 3 December 2010

57. ShiffmanRN

LiawY

BrandtCA

CorbGJ

1999 Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc 6 104 114

58. ShojaniaKG

JenningsA

MayhewA

RamsayCR

EcclesMP

GrimshawJ

2009 The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev CD001096

59. SintchenkoV

MagrabiF

TipperS

2007 Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review. Med Inform Internet Med 32 225 240

60. SmithMY

DepueJD

RiniC

2007 Computerized decision-support systems for chronic pain management in primary care. Pain Medicine S3 S155 S166

61. TanK

DearPR

NewellSJ

2005 Clinical decision support systems for neonatal care. Cochrane Database Syst Rev CD004211

62. ThompsonD

JohnstonP

SpurrC

2009 The impact of electronic medical records on nursing efficiency. J Nurs Adm 39 444 451

63. UsluAM

StausbergJ

2008 Value of the electronic patient record: an analysis of the literature. J Biomed Inform 41 675 682

64. van RosseF

MaatB

RademakerCM

van VughtAJ

EgbertsAC

2009 The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review. Pediatrics 123 1184 1190

65. WolfstadtJI

GurwitzJH

FieldTS

LeeM

KalkarS

2008 The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review. J Gen Intern Med 23 451 458

66. WongK

YuSKh

HolbrookA

2010 A systematic review of medication safety outcomes related to drug interaction software. J Popul Ther Clin Pharmacol 17 e243 255

67. YourmanL

ConcatoJ

AgostiniJV

2008 Use of computer decision support interventions to improve medication prescribing in older adults: a systematic review. Am J Geriatr Pharmacother 6 119 129

68. AlexanderG

StaggersN

2009 A systematic review of the designs of clinical technology: findings and recommendations for future research. ANS Adv Nurs Sci 32 252 279

69. BerlinA

SoraniM

SimI

2006 A taxonomic description of computer-based clinical decision support systems. J Biomed Inform 39 656 667

70. CampionTRJr

WaitmanLR

MayAK

OzdasA

LorenziNM

2010 Social, organizational, and contextual characteristics of clinical decision support systems for intensive insulin therapy: a literature review and case study. Int J Med Inform 79 31 43

71. CarvalhoCJ

BoryckiEM

KushnirukA

2009 Ensuring the safety of health information systems: using heuristics for patient safety. Healthc Q 12 49 54

72. ChanKS

FowlesJB

WeinerJP

2010 Electronic health records and reliability and validity of quality measures: a review of the literature. Med Care Res Rev 67 503 527

73. ChuangJH

HripcsakG

JendersRA

2000 Considering clustering: a methodological review of clinical decision support system studies. Proc AMIA Symp 146 150

74. de KeizerNF

AmmenwerthE

2008 The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005? Int J Med Inform 77 41 49

75. DamianiG

PinnarelliL

ColosimoSC

AlmientoR

SicuroL

2010 The effectiveness of computerized clinical guidelines in the process of care: a systematic review. BMC Health Serv Res 10 1472 6963

76. DorrD

BonnerLM

CohenAN

ShoaiRS

PerrinR

2007 Informatics systems to promote improved care for chronic illness: a literature review. J Am Med Inform Assoc 14 156 163

77. EisensteinEL

OrtizM

AnstromKJ

CrosslinDR

LobachDF

2006 Assessing the quality of medical information technology economic evaluations: room for improvement. AMIA Annu Symp Proc 234 238

78. FitzpatrickLA

MelnikasAJ

WeathersM

KachnowskiSW

2008 Understanding communication capacity. Communication patterns and ICT usage in clinical settings. HIM J 22 34 41

79. FordEW

MenachemiN

PetersonLT

HuertaTR

2009 Resistance is futile: but it is slowing the pace of EHR adoption nonetheless. J Am Med Inform Assoc 16 274 281

80. FriedmanCP

AbbasUL

2003 Is medical informatics a mature science? A review of measurement practice in outcome studies of clinical systems. Int J Med Inform 69 261 272

81. GagnonMP

LegareF

LabrecqueM

FremontP

PluyeP

2009 Interventions for promoting information and communication technologies adoption in healthcare professionals. Cochrane Database Syst Rev CD006093

82. GreenhalghT

PottsHW

WongG

BarkP

SwinglehurstD

2009 Tensions and paradoxes in electronic patient record research: a systematic literature review using the meta-narrative method. Milbank Q 87 729 788

83. GruberD

CummingsGG

LeBlancL

SmithDL

2009 Factors influencing outcomes of clinical information systems implementation: a systematic review. Comput Inform Nurs 27 151 163

84. GursesAP

XiaoY

2006 A systematic review of the literature on multidisciplinary rounds to design information technology. J Am Med Inform Assoc 13 267 276

85. HandlerSM

AltmanRL

PereraS

HanlonJT

StudenskiSA

2007 A systematic review of the performance characteristics of clinical event monitor signals used to detect adverse drug events in the hospital setting. J Am Med Inform Assoc 14 451 458

86. HarrisAD

McGregorJC

PerencevichEN

FurunoJP

ZhuJ

2006 The use and interpretation of quasi-experimental studies in medical informatics. J Am Med Inform Assoc 13 16 23

87. HartMD

2008 Informatics competency and development within the US nursing population workforce: a systematic literature review. Comput Inform Nurs 26 320 329

88. HayrinenK

SarantoK

NykanenP

2008 Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 77 291 304

89. HoerbstA

AmmenwerthE

2010 Electronic health records. A systematic review of quality requirements. Methods Inf Med 49 320 336

90. HoganWR

WagnerMM

1997 Accuracy of data in computer-based patient records. J Am Med Inform Assoc 4 342 355

91. HoldenRJ

KarshBT

2010 The technology acceptance model: its past and its future in health care. J Biomed Inform 43 159 172

92. HurykLA

2010 Factors influencing nurses' attitudes towards healthcare information technology. J Nurs Manag 18 606 612

93. JohnsonKB

2001 Barriers that impede the adoption of pediatric information technology. Arch Pediatr Adolesc Med 155 1374 1379

94. JordanK

PorcheretM

CroftP

2004 Quality of morbidity coding in general practice computerized medical records: a systematic review. Fam Pract 21 396 412

95. KaplanB

2001 Evaluating informatics applications–clinical decision support systems literature review. Int J Med Inform 64 15 37

96. KawamotoK

HoulihanCA

BalasEA

LobachDF

2005 Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 330 765

97. KawamotoK

LobachDF

2003 Clinical decision support provided within physician order entry systems: a systematic review of features effective for changing clinician behavior. AMIA Annu Symp Proc 361 365

98. KeshavjeeK

BosomworthJ

CopenJ

LaiJ

KucukyaziciB

2006 Best practices in EMR implementation: a systematic review. AMIA Annu Symp Proc 982

99. KhajoueiR

JaspersMW

2008 CPOE system design aspects and their qualitative effect on usability. Stud Health Technol Inform 136 309 314

100. KukafkaR

JohnsonSB

LinfanteA

AllegranteJP

2003 Grounding a new information technology implementation framework in behavioral science: a systematic analysis of the literature on IT use. J Biomed Inform 36 218 227

101. LudwickDA

DoucetteJ

2009 Adopting electronic medical records in primary care: lessons learned from health information systems implementation experience in seven countries. Int J Med Inform 78 22 31

102. MachanC

AmmenwerthE

BodnerT

2006 Publication bias in medical informatics evaluation research: is it an issue or not? Stud Health Technol Inform 124 957 962

103. MairFS

MayC

FinchT

MurrayE

AndersonG

2007 Understanding the implementation and integration of e-health services. J Telemed Telecare 13 36 37

104. MollonB

ChongJJr

HolbrookAM

SungM

ThabaneL

2009 Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials. BMC Med Inform Decis Mak 9 11

105. MoxeyA

RobertsonJ

NewbyD

HainsI

WilliamsonM

2010 Computerized clinical decision support for prescribing: provision does not guarantee uptake. J Am Med Inform Assoc 17 25 33

106. NiesJ

ColombetI

DegouletP

DurieuxP

2006 Determinants of success for computerized clinical decision support systems integrated in CPOE Systems: a systematic review. AMIA Annu Symp Proc 594 598

107. PirnejadH

NiazkhaniZ

BergM

BalR

2008 Intra-organizational communication in healthcare–considerations for standardization and ICT application. Methods Inf Med 47 336 345

108. PoeSS

2010 Building nursing intellectual capital for safe use of information technology: a systematic review. J Nurs Care Qual. In press

109. RahimiB

VimarlundV

2007 Methods to evaluate health information systems in healthcare settings: a literature review. J Med Syst 31 397 432

110. RahimiB

VimarlundV

TimpkaT

2009 Health information system implementation: a qualitative meta-analysis. J Med Syst 33 359 368

111. RothCP

LimYW

PevnickJM

AschSM

McGlynnEA

2009 The challenge of measuring quality of care from the electronic health record. Am J Med Qual 24 385 394

112. SaboorS

AmmenwerthE

2009 Categorizing communication errors in integrated hospital information systems. Methods Inf Med 48 203 210

113. ThiruK

HasseyA

SullivanF

2003 Systematic review of scope and quality of electronic patient record data in primary care. BMJ 326 1070

114. van de WeteringR

BatenburgR

2009 A PACS maturity model: a systematic meta-analytic review on maturation and evolvability of PACS in the hospital enterprise. Int J Med Inform 78 127 140

115. van der MeijdenMJ

TangeHJ

TroostJ

HasmanA

2003 Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc 10 235 243

116. van der SijsH

AartsJ

VultoA

BergM

2006 Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 13 138 147

117. WardR

StevensC

BrentnallP

BriddonJ

2008 The attitudes of health care staff to information technology: a comprehensive review of the research literature. Health Info Libr J 25 81 97

118. WeirCR

StaggersN

PhansalkarS

2009 The state of the evidence for computerized provider order entry: a systematic review and analysis of the quality of the literature. Int J Med Inform 78 365 374

119. WenHC

HoYS

JianWS

LiHC

HsuYH

2007 Scientific production of electronic health record research, 1991-2005. Comput Methods Programs Biomed 86 191 196

120. WollersheimD

SariA

RahayuW

2009 Archetype-based electronic health records: a literature review and evaluation of their applicability to health data interoperability and access. HIM J 38 7 17

121. YarbroughAK

SmithTB

2007 Technology acceptance among physicians: a new take on TAM. Med Care Res Rev 64 650 672

122. YusofMM

StergioulasL

ZugicJ

2007 Health information systems adoption: findings from a systematic review. Medinfo 12 262 266

123. ShepperdS

LewinS

StrausS

ClarkeM

EcclesMP

2009 Can we systematically review studies that evaluate complex interventions? PLoS Med 6 e1000086 doi:10.1371/journal.pmed.1000086

124. US Institute of Standards & Technology. Available: http://www.itl.nist.gov/div897/docs/EHR.htm Accessed 3 December 2010

125. eHealth Initiative 2004 Electronic prescribing: towards maximum value and rapid adoption. Available: http://www.ehealthinitiative.org/uploads/file/eRx%202004%20Report%20Exec%20Summary.pdf Accessed 3 December 2010

126. Dixon-WoodsM

AgarwalS

JonesD

YoungB

SuttonA

2005 Synthesising qualitative and quantitative evidence: a review of possible methods. J Health Serv Res Policy 10 45 53

127. McKinstryB

PinnockH

SheikhA

2009 Telemedicine for management of patients with COPD? Lancet 374 672 673

128. McKinstryB

HammersleyV

BurtonC

PinnockH

EltonR

2010 The quality, safety and content of telephone and face-to-face consultations: a comparative study. Qual Saf Health Care 19 298 303

129. PinnockH

HanleyJ

LewisS

MacNeeW

PagliariC

2009 The impact of a telemetric chronic obstructive pulmonary disease monitoring service: randomised controlled trial with economic evaluation and nested qualitative study. Prim Care Respir J 18 233 235

130. McKinstryB

WatsonP

PinnockH

HeaneyD

SheikhA

2009 Telephone consulting in primary care: a triangulated qualitative study of patients and providers. Br J Gen Pract 59 e209 218

131. McKinstryB

WatsonP

PinnockH

HeaneyD

SheikhA

2009 Confidentiality and the telephone in family practice: a qualitative study of the views of patients, clinicians and administrative staff. Fam Pract 26 344 350

132. BlackA

CarJ

MajeedA

SheikhA

2008 Strategic considerations for improving the quality of eHealth research: we need to improve the quality and capacity of academia to undertake informatics research. Inform Prim Care 16 175 177

133. DixonBE

ZafarA

McGowanJJ

2007 Development of a taxonomy for health information technology. Stud Health Technol Inform 129 616 620

134. PagliariC

SloanD

GregorP

SullivanF

DetmerD

2005 What is eHealth (4): a scoping exercise to map the field. J Med Internet Res 7 e1

135. OhH

RizoC

EnkinM

JadadA

2005 What is eHealth (3): a systematic review of published definitions. J Med Internet Res 7 e1

136. AhernDK

2007 Challenges and opportunities of eHealth research. Am J Prev Med 32 S75 S82

137. BowlingMJ

RimerBK

LyonsEJ

GolinCE

FrydmanG

2006 Methodologic challenges of e-health research. Eval Program Plann 29 390 396

138. HeathfieldH

PittyD

HankaR

1998 Evaluating information technology in health care: barriers and challenges. BMJ 316 1959 1961

139. FriedmanCF

HaugP

2003 Report on conference track 5: evaluation metrics and outcome. Int J Med Inf 69 307 309

140. AmmenwerthE

de KeizerN

2007 A viewpoint on evidence-based health informatics, based on a pilot survey on evaluation studies in health care informatics. J Am Med Inform Assoc 14 368 371

141. AmmenwerthE

Schnell-InderstP

SiebertU

2010 Vision and challenges of Evidence-Based Health Informatics: a case study of a CPOE meta-analysis. Int J Med Inform 79 e83 e88

142. AmmenwerthE

GraberS

HerrmannG

BurkleT

KonigJ

2003 Evaluation of health information systems-problems and challenges. Int J Med Inform 71 125 135

143. FriedmanCP

WyattJC

2001 Publication bias in medical informatics. J Am Med Inform Assoc 8 189 191

144. ClampS

KeenJ

2007 Electronic health records: is the evidence base any use? Med Inform Internet Med 32 5 10

145. KaplanB

2001 Evaluating informatics applications–some alternative approaches: theory, social interactionism, and call for methodological pluralism. Int J Med Inform 64 39 56

146. AmmenwerthE

TalmonJ

AshJS

BatesDW

Beuscart-ZephirMC

2006 Impact of CPOE on mortality rates–contradictory findings, important messages. Methods Inf Med 45 586 593

147. PollockN

WilliamsR

ProcterR

2003 Fitting standard software packages to non-standard organizations: the “biography' of an enterprise-wide system. Tech Anal Strat Manag 15 317

148. RobertsonAR

CresswellK

TakianA

PetrakakiD

2010 Prospective evaluation of the implementation and adoption of NHS Connecting for Health's electronic health record in secondary care in England: interim results from a national evaluation. BMJ 341 c4564

149. AmmenwerthE

BrenderJ

NykanenP

ProkoschHU

RigbyM

2004 Visions and strategies to improve evaluation of health information systems. Reflections and lessons based on the HIS-EVAL workshop in Innsbruck. Int J Med Inform 73 479 491

150. AmmenwerthE

AartsJ

BergholdA

Beuscart-ZephirMC

BrenderJ

2006 Declaration of Innsbruck. Results from the European Science Foundation Sponsored Workshop on Systematic Evaluation of Health Information Systems (HIS-EVAL). Methods Inf Med 45 Suppl 1 121 123

151. TierneyWM

OverhageJM

McDonaldCJ

1994 A plea for controlled trials in medical informatics. J Am Med Inform Assoc 1 353 355

152. HolbrookAM

ThabaneL

ShcherbatykhIY

O'ReillyD

2006 E-health interventions as complex interventions: improving the quality of methods of assessment. AMIA Annu Symp Proc 952

153. ShcherbatykhI

HolbrookA

ThabaneL

DolovichL

2008 Methodologic issues in health informatics trials: the complexities of complex interventions. J Am Med Inform Assoc 15 575 580

154. ChuangJH

HripcsakG

HeitjanDF

2002 Design and analysis of controlled trials in naturally clustered environments: implications for medical informatics. J Am Med Inform Assoc 9 230 238

155. TalmonJ

AmmenwerthE

BrenderJ

de KeizerN

NykänenP

2010 Statement on reporting of evaluation studies in health informatics STARE-HI. Available: http://www.imia.org/endorsed/Stare-HI_as_published.pdf Accessed 3 December 2010

156. AmmenwerthE

BrenderJ

NykanenP

TalmonJ

WesselC

2010 Guidelines for best evaluation practices in health informatics GEP-HI. Available: http://iig.umit.at/efmi/Accessed 3 December 2010

157. WyattJ

2010 Assessing and improving evidence based health informatics research. Stud Health Technol Inform 151 435 445

158. PagliariC

2007 Design and evaluation in eHealth: challenges and implications for an interdisciplinary field. J Med Internet Res 9 e15

159. BrenderJ

2006 Overview of assessment methods. Handbook of evaluation methods for health informatics (first edition). Burlington Academic Press 61 72

160. GreenhalghT

RobertG

BateP

KyriakidouO

MacfarlaneF

2004 How to spread good ideas: a systematic review of the literature on diffusion, dissemination and sustainability of innovations in health service delivery and organisation. Available: http://www.sdo.nihr.ac.uk/files/project/38-final-report.pdf. Accessed 3 December 2010

161. GoldzweigCL

TowfighA

MaglioneM

ShekellePG

2009 Costs and benefits of health information technology: new trends from the literature. Health Aff 28 w282 w293

162. CollinS

ReevesBC

HendyJ

FulopN

HutchingsA

2008 Implementation of computerised physician order entry (CPOE) and picture archiving and communication systems (PACS) in the NHS: quantitative before and after study. BMJ 337 a939

163. GreenhalghT

StramerK

BratanT

ByrneE

MohammadY

2008 Introduction of shared electronic records: multi-site case study using diffusion of innovation theory. BMJ 337 a1786

164. EminovicN

WyattJC

TarpeyAM

MurrayG

IngramsGJ

2004 First evaluation of the NHS direct online clinical enquiry service: a nurse-led web chat triage service for the public. J Med Internet Res 6 e17