Assessing the impact of healthcare research: A systematic review of methodological frameworks

1. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis J, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. doi: 10.1016/S0140-6736(13)62329-6 24411643

2. Buxton M, Hanney S. How can payback from health services research be assessed? J Health Serv Res Policy. 1996;1(1):35–43. 10180843.

3. HEFCE. REF 2014: Assessment framework and guidance on submissions 2011 [cited 2016 15 Feb]. Available from: http://www.ref.ac.uk/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf.

4. Penfield T, Baker MJ, Scoble R, Wykes MC. Assessment, evaluations, and definitions of research impact: A review. Res Eval. 2014;23(1):21–32. doi: 10.1093/reseval/rvt021

5. Buykx P, Humphreys J, Wakerman J, Perkins D, Lyle D, McGrail M, et al. Making evidence count: A framework to monitor the impact of health services research. Aust J Rural Health. 2012;20(2):51–8. doi: 10.1111/j.1440-1584.2012.01256.x 22435764

6. Martin BR. The Research Excellence Framework and the 'impact agenda': are we creating a Frankenstein monster? Res Eval. 2011;20(3):247–54. doi: 10.3152/095820211x13118583635693

7. Kok MO, Schuit AJ. Contribution mapping: a method for mapping the contribution of research to enhance its impact. Health Res Policy Syst. 2012;10. doi: 10.1186/1478-4505-10-21 22748169

8. Canadian Institutes of Health Research. Developing a CIHR framework to measure the impact of health research 2005 [cited 2016 26 Feb]. Available from: http://publications.gc.ca/collections/Collection/MR21-65-2005E.pdf.

9. HEFCE. HEFCE allocates £3.97 billion to universities and colleges in England for 2015–1 2015. Available from: http://www.hefce.ac.uk/news/newsarchive/2015/Name,103785,en.html.

10. Stern N. Building on Success and Learning from Experience—An Independent Review of the Research Excellence Framework 2016 [cited 2016 05 Aug]. Available from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf.

11. Matthews D. REF sceptic to lead review into research assessment: Times Higher Education; 2015 [cited 2016 21 Apr]. Available from: https://www.timeshighereducation.com/news/ref-sceptic-lead-review-research-assessment.

12. HEFCE. The Metric Tide—Report of the Independent Review of the Role of Metrics in Research Assessment and Management 2015 [cited 2016 11 Aug]. Available from: http://www.hefce.ac.uk/media/HEFCE,2014/Content/Pubs/Independentresearch/2015/The,Metric,Tide/2015_metric_tide.pdf.

13. Bainbridge R, Tsey K, McCalman J, Kinchin I, Saunders V, Lui FW, et al. No one's discussing the elephant in the room: contemplating questions of research impact and benefit in Aboriginal and Torres Strait Islander Australian health research. BMC Public Health. 2015;15. doi: 10.1186/s12889-015-2052-3 26202429

14. LSE Public Policy Group. Maximizing the impacts of your research: A handbook for social scientists. http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf. London: LSE; 2011.

15. HEFCE. Consultation on the second Research Excellence Framework. 2016.

16. Ovseiko PV, Oancea A, Buchan AM. Assessing research impact in academic clinical medicine: a study using Research Excellence Framework pilot impact indicators. BMC Health Serv Res. 2012;12. doi: 10.1186/1472-6963-12-478 23259467

17. Kuruvilla S, Mays N, Walt G. Describing the impact of health services and policy research. J Health Serv Res Policy. 2007;12. doi: 10.1258/135581907780318374 17411504

18. Merriam-Webster Dictionary 2017. Available from: https://www.merriam-webster.com/dictionary/methodology.

19. Oxford Dictionaries—pathway 2016 [cited 2016 19 June]. Available from: http://www.oxforddictionaries.com/definition/english/pathway.

20. Oxford Dictionaries—metric 2016 [cited 2016 15 Sep]. Available from: https://en.oxforddictionaries.com/definition/metric.

21. WHO. WHO Ethical and Safety Guidelines for Interviewing Trafficked Women 2003 [cited 2016 29 July]. Available from: http://www.who.int/mip/2003/other_documents/en/Ethical_Safety-GWH.pdf.

22. Greenhalgh T, Raftery J, Hanney S, Glover M. Research impact: a narrative review. BMC Med. 2016;14(1):1–16. doi: 10.1186/s12916-016-0620-8 27211576

23. Banzi R, Moja L, Pistotti V, Facchini A, Liberati A. Conceptual frameworks and empirical approaches used to assess the impact of health research: an overview of reviews. Health Res Policy Syst. 2011;9. doi: 10.1186/1478-4505-9-26 21702930

24. Yazdizadeh B, Majdzadeh R, Salmasian H. Systematic review of methods for evaluating healthcare research economic impact. Health Res Policy Syst. 2010;8. doi: 10.1186/1478-4505-8-6 20196839

25. Milat AJ, Bauman AE, Redman S. A narrative review of research impact assessment models and methods. Health Res Policy Syst. 2015;13. doi: 10.1186/s12961-015-0003-1 25884944

26. Bornmann L. What is societal impact of research and how can it be assessed? a literature survey. American Society for Information Science and Technology. 2013;64(2):217–33. doi: 10.1002/asi.22803

27. Patel VM, Ashrafian H, Ahmed K, Arora S, Jiwan S, Nicholson JK, et al. How has healthcare research performance been assessed? A systematic review. J R Soc Med. 2011;104(6):251–61. doi: 10.1258/jrsm.2011.110005 21659400

28. Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med. 2009;151(4):264–9. 19622511

29. Kuruvilla S, Mays N, Pleasant A, Walt G. Describing the impact of health research: a Research Impact Framework. BMC Health Serv Res. 2006;6:134. 17049092; PubMed Central PMCID: PMCPMC1635046. doi: 10.1186/1472-6963-6-134

30. Weiss AP. Measuring the impact of medical research: moving from outputs to outcomes. Am J Psychiatry. 2007;164(2):206–14. 17267781. doi: 10.1176/ajp.2007.164.2.206

31. Kalucy L, et al. Primary Health Care Research Impact Project: Final Report Stage 1 Adelaide: Primary Health Care Research & Information Service; 2007 [cited 2016 26 Feb]. Available from: http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3338.pdf.

32. Guinea J, Sela E, Gómez-Núñez AJ, Mangwende T, Ambali A, Ngum N, et al. Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects. Res Eval. 2015;24(2):131–45. doi: 10.1093/reseval/rvu034

33. Canadian Academy of Health Sciences. Making an impact—A preferred framework and indicators to measure returns on investment in health research 2009 [cited 2016 26 Feb]. Available from: http://www.cahs-acss.ca/wp-content/uploads/2011/09/ROI_FullReport.pdf.

34. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: Development and implementation of the Alberta Innovates–Health Solutions impact framework. Res Eval. 2012;21(5):354–67. doi: 10.1093/reseval/rvs027

35. Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2014;13(1):3–1p. doi: 10.1186/1478-4505-13-3 109774007. Language: English. Entry Date: 20150923. Revision Date: 20150923. Publication Type: Journal Article.

36. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Libr Assoc. 2010;98(1):17–23. doi: 10.3163/1536-5050.98.1.008. 20098647

37. Brueton VC, Vale CL, Choodari-Oskooei B, Jinks R, Tierney JF. Measuring the impact of methodological research: a framework and methods to identify evidence of impact. Trials. 2014;15(1):464–1p. doi: 10.1186/1745-6215-15-464 109768210. Language: English. Entry Date: 20150313. Revision Date: 20150923. Publication Type: Journal Article.

38. Donovan C. The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded research. New Directions for Evaluation. 2008;2008(118):47–60. doi: 10.1002/ev.260

39. HEFCE. RAE 2008—Guidance in submissions 2005 [cited 2016 15 Feb]. Available from: http://www.rae.ac.uk/pubs/2005/03/rae0305.pdf.

40. Spaapen J, van Drooge L. Introducing ‘productive interactions’ in social impact assessment. Res Eval. 2011;20(3):211–8. doi: 10.3152/095820211x12941371876742

41. Royal Netherlands Academy of Arts and Sciences. The societal impact of applied health research—Towards a quality assessment system 2002 [cited 2016 29 Feb]. Available from: https://www.knaw.nl/en/news/publications/the-societal-impact-of-applied-health-research/@@download/pdf_file/20021098.pdf.

42. Morton S. Progressing research impact assessment: A 'contributions' approach. Res Eval. 2015;24(4):405–19. doi: 10.1093/reseval/rvv016

43. Meagher L, Lyall C, Nutley S. Flows of knowledge, expertise and influence: a method for assessing policy and practice impacts from social science research. Res Eval. 2008;17(3):163–73. doi: 10.3152/095820208x331720

44. Lavis J, Ross S, McLeod C, Gildiner A. Measuring the impact of health research. J Health Serv Res Policy. 2003;8(3):165–70. doi: 10.1258/135581903322029520 12869343.

45. Canavan J, Gillen A, Shaw A. Measuring research impact: developing practical and cost-effective approaches. Evidence & Policy. 2009;5(2):167–77.

46. Landry R, Amara N, Lamari M. Climbing the ladder of research utilization—Evidence from social science research. Science Communication. 2001;22(4):396–422. doi: 10.1177/1075547001022004003

47. Jacob R, McGregor M. Assessing the impact of health technology assessment. Int J Technol Assess Health Care. 1997;13(1):68–80. 9119625

48. Weiss CH. Using social research in public policy making: Lexington Books; 1977.

49. Weiss CH. The Many Meanings of Research Utilization. Public Adm Rev. 1979;39(5):426–31. doi: 10.2307/3109916

50. Kogan M, Henkel M. Government and research: the Rothschild experiment in a government department: Heinemann Educational Books; 1983.

51. Thomas P. The Aims and Outcomes of Social Policy Research. Croom Helm; 1985.

52. Bulmer M. Social Science Research and Government: Comparative Essays on Britain and the United States: Cambridge University Press; 2010.

53. Booth T. Developing Policy Research. Aldershot, Gower1988.

54. Yin R, Moore G. Lessons on the utilization of research from nine case experiences in the natural hazards field. Knowledge in Society. 1988;1.

55. Kalucy L, et al Exploring the impact of primary health care research Stage 2 Primary Health Care Research Impact Project Adelaide: Primary Health Care Research & Information Service (PHCRIS); 2009 [cited 2016 26 Feb]. Available from: http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_8108.pdf.

56. CHSRF. Canadian Health Services Research Foundation 2000. Health Services Research and Evidence-based Decision Making [cited 2016 February]. Available from: http://www.cfhi-fcass.ca/migrated/pdf/mythbusters/EBDM_e.pdf.

57. Knott J, Wildavsky A. If dissemination is the answer, what is the problem? Knowledge: Creation, Diffusion, Utilization. 1980;1.

58. W.K. Kellogg Foundation. Logic Model Development Guide 2004 [cited 2016 19 July]. Available from: http://www.smartgivers.org/uploads/logicmodelguidepdf.pdf.

59. United Way of America. Measuring Program Outcomes: A Practical Approach 1996 [cited 2016 19 July]. Available from: https://www.bttop.org/sites/default/files/public/W.K.%20Kellogg%20LogicModel.pdf.

60. Nutley S, Percy-Smith J and Solesbury W. Models of research impact: a cross sector review of literature and practice. London: Learning and Skills Research Centre 2003.

61. Spaapen J, van Drooge L. SIAMPI final report [cited 2017 Jan]. Available from: http://www.siampi.eu/Content/SIAMPI_Final%20report.pdf.

62. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):1–28. doi: 10.1186/1478-4505-1-2

63. LSHTM. The Health Risks and Consequences of Trafficking in Women and Adolescents—Findings from a European Study 2003 [cited 2016 29 July]. Available from: http://www.oas.org/atip/global%20reports/zimmerman%20tip%20health.pdf.

64. Drummond M, Cooke J, Walley T. Economic evaluation under managed competition: evidence from the U.K. Soc Sci Med. 1997;45.

65. Manandhar DS, Osrin D, Shrestha BP, Mesko N, Morrison J, Tumbahangphe KM, et al. Effect of a participatory intervention with women's groups on birth outcomes in Nepal: cluster-randomised controlled trial. Lancet. 364(9438):970–9. doi: 10.1016/S0140-6736(04)17021-9 15364188

66. Wawrzyniak AJ, Ownby RL, McCoy K, Waldrop-Valverde D. Health Literacy: Impact on the Health of HIV-Infected Individuals. Curr HIV/AIDS Rep. 2013;10(4):295–304. doi: 10.1007/s11904-013-0178-4. 24222474

67. Graham KER, Chorzempa HL, Valentine PA, Magnan J. Evaluating health research impact: Development and implementation of the Alberta Innovates–Health Solutions impact framework. Res Eval. 2012;21. doi: 10.1093/reseval/rvs027

68. van Driel ML, Maier M, De Maeseneer J. Measuring the impact of family medicine research: scientific citations or societal impact? Fam Pract. 2007;24(5):401–2. 17998264. doi: 10.1093/fampra/cmm061

69. Hansen J, Muscat N, Keskimaki L. Measuring and improving the societal impact of health care research. Eurohealth. 2013;19(3):32–5.

70. Russell G. Response to second HEFCE consultation on the Research Excellence Framework 2009 [cited 2016 04 Apr]. Available from: http://russellgroup.ac.uk/media/5262/ref-consultation-response-final-dec09.pdf.

71. Cohen G, Schroeder J, Newson R, King L, Rychetnik L, Milat AJ, et al. Does health intervention research have real world policy and practice impacts: testing a new impact assessment tool. Health Res Policy Syst. 2015;13. doi: 10.1186/1478-4505-13-3 25552272

72. Bornmann L. Measuring the societal impact of research. Research is less and less assessed on scientific impact alone-we should aim to quantify the increasingly important contributions of science to society. EMBO Reports. 2012;13(8):673–6. 22777497. doi: 10.1038/embor.2012.99

73. Erno-Kjolhede E, Hansson F. Measuring research performance during a changing relationship between science and society. Res Eval. 2011;20(2):131–43. doi: 10.3152/095820211x12941371876544