Risk of Bias in Systematic Reviews of Non-Randomized Studies of Adverse Cardiovascular Effects of Thiazolidinediones and Cyclooxygenase-2 Inhibitors: Application of a New Cochrane Risk of Bias Tool

1. Van Spall HC, Toren A, Kiss A, Fowler RA. Eligibility criteria of randomized controlled trials published in high-impact general medical journals: a systematic sampling review. JAMA. 2007;297:1233–1240. doi: 10.1001/jama.297.11.1233 17374817

2. Rothwell PM. External validity of randomised controlled trials: “To whom do the results of this trial apply?”. Lancet. 2005;365:82–93. doi: 10.1016/S0140-6736(04)17670-8 15639683

3. Black N. Why we need observational studies to evaluate the effectiveness of health care. BMJ. 1996;312:1215–1218. 8634569

4. Schneeweiss S, Avorn J. A review of uses of health care utilization databases for epidemiologic research on therapeutics. J Clin Epidemiol. 58:323–337. doi: 10.1016/j.jclinepi.2004.10.012 15862718

5. Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behav Res. 2011;46:399–424. doi: 10.1080/00273171.2011.568786 21818162

6. Golder S, Loke YK, Bland M. Meta-analyses of adverse effects data derived from randomised controlled trials as compared to observational studies: methodological overview. PLoS Med. 2011;8:e1001026. doi: 10.1371/journal.pmed.1001026 21559325

7. Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med. 2009;151:203–205. 19567618

8. Egger M, Schneider M, Davey SG. Spurious precision? Meta-analysis of observational studies. BMJ. 1998;316:140–144. 9462324

9. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928. doi: 10.1136/bmj.d5928 22008217

10. Deeks J, Dinnes J, D’Amico R, Sowden A, Sakarovitch C. Evaluating non-randomised intervention studies. Health Technol Assess. 2003;7:186. doi: 10.3310/hta7270

11. Sanderson S, Tatt ID, Higgins JP. Tools for assessing quality and susceptibility to bias in observational studies in epidemiology: a systematic review and annotated bibliography. Int J Epidemiol. 2007;36:666–676. doi: 10.1093/ije/dym018 17470488

12. Wells GA, Shea B, O’Connell D, Peterson J, Welch V, Losos M, et al. The Newcastle–Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. 2014 [cited 26 Aug 2015]. Ottawa: Ottawa Hospital Research Institute. Available: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

13. Downs SH, Black N. The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions. J Epidemiol Community Health. 1998;52:377–384. doi: 10.1136/jech.52.6.377 9764259

14. Scottish Intercollegiate Guidelines Network. Critical appraisal: notes and checklists. 2015 [cited 10 Oct 2015]. Available: http://www.sign.ac.uk/methodology/checklists.html.

15. Jüni P, Witschi A, Bloch R, Egger M. The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–1060. doi: 10.1001/jama.282.11.1054 10493204

16. Sterne J, Higgins J, Reeves B, editors. A Cochrane risk of bias assessment tool: for non-randomized studies of interventions (ACROBAT-NRSI). Version 1.0.0, 24 September 2014. Available: http://www.riskofbias.info.

17. Loke YK, Kwok CS, Singh S. Comparative cardiovascular effects of thiazolidinediones: systematic review and meta-analysis of observational studies. BMJ. 2011;342:d1309. doi: 10.1136/bmj.d1309 21415101

18. McGettigan P, Henry D. Cardiovascular risk and inhibition of cyclooxygenase: a systematic review of the observational studies of selective and nonselective inhibitors of cyclooxygenase 2. JAMA. 2006;296:1633–1644. doi: 10.1001/jama.296.13.jrv60011 16968831

19. McGettigan P, Henry D. Cardiovascular risk with non-steroidal anti-inflammatory drugs: systematic review of population-based controlled observational studies. PLoS Med. 2011;8:e1001098. doi: 10.1371/journal.pmed.1001098 21980265

20. Graham DJ, Ouellet-Hellstrom R, MaCurdy TE, Ali F, Sholley C, Worrall C, et al. Risk of acute myocardial infarction, stroke, heart failure, and death in elderly medicare patients treated with rosiglitazone or pioglitazone. JAMA. 2010;304:411–418. doi: 10.1001/jama.2010.920 20584880

21. Singh G, Mithal A, Triadafilopoulos G. Both selective COX-2 inhibitors and non-selective NSAIDs increase the risk of acute myocardial infarction in patients with arthritis: selectivity is with the patient, not the drug class. Ann Rheum Dis. 2005;64(Suppl III):85.

22. Sturkenboom MC, Dieleman J, Verhamme K, Straus S, Vander Hoeven-Borgman M, van der Lei J. Cardiovascular events during use of COX-2 selective and non-selective NSAIDs. Pharmacoepidemiol Drug Saf. 2005;14:S57.

23. Cicchetti DV, Allison T. A new procedure for assessing reliability of scoring EEG sleep recordings. Am J EEG Technol. 1971;11:101–109.

24. Bilik D, McEwen LN, Brown MB, Selby JV, Karter AJ, Marrero DG, et al. Thiazolidinediones, cardiovascular disease and cardiovascular mortality: Translating Research Into Action For Diabetes (TRIAD). Pharmacoepidemiol Drug Saf. 2010;19:715–721. doi: 10.1002/pds.1954 20583206

25. Brownstein JS, Murphy SN, Goldfine AB, Grant RW, Sordo M, Gainer V, et al. Rapid identification of myocardial infarction risk associated with diabetes medications using electronic medical records. Diabetes Care. 2010;33:526–531. doi: 10.2337/dc09-1506 20009093

26. Dormuth CR, Maclure M, Carney G, Schneeweiss S, Bassett K, Wright JM. Rosiglitazone and myocardial infarction in patients previously prescribed metformin. PLoS ONE. 2009;4:e6080. doi: 10.1371/journal.pone.0006080 19562036

27. Hsiao F-Y, Huang W-F, Wen Y-W, Chen P-F, Kuo K, Tsai Y-W. Thiazolidinediones and cardiovascular events in patients with type 2 diabetes mellitus: a retrospective cohort study of over 473,000 patients using the National Health Insurance database in Taiwan. Drug Saf. 2009;32:675–690. doi: 10.2165/00002018-200932080-00006 19591532

28. Juurlink DN, Gomes T, Lipscombe LL, Austin PC, Hux JE, Mamdani MM. Adverse cardiovascular events during treatment with pioglitazone and rosiglitazone: population based cohort study. BMJ. 2009;339:b2942. doi: 10.1136/bmj.b2942 19690342

29. Koro CE, Fu Q, Stender M. An assessment of the effect of thiazolidinedione exposure on the risk of myocardial infarction in type 2 diabetic patients. Pharmacoepidemiol Drug Saf. 2008;17:989–996. doi: 10.1002/pds.1640 18759378

30. Lipscombe LL, Gomes T, Lévesque LE, Hux JE, Juurlink DN, Alter DA. Thiazolidinediones and cardiovascular outcomes in older patients with diabetes. JAMA. 2007;298:2634–2643. doi: 10.1001/jama.298.22.2634 18073359

31. Margolis DJ, Hofstad O, Strom BL. Association between serious ischemic cardiac outcomes and medications used to treat diabetes. Pharmacoepidemiol Drug Saf. 2008;17:753–759. doi: 10.1002/pds.1630 18613215

32. Pantalone K, Kattan M, Yu C, Wells B, Arrigain S, Jain A, et al. The risk of developing coronary artery disease or congestive heart failure, and overall mortality, in type 2 diabetic patients receiving rosiglitazone, pioglitazone, metformin, or sulfonylureas: a retrospective analysis. Acta Diabetol. 2009;46:145–154. doi: 10.1007/s00592-008-0090-3 19194648

33. Stockl KM, Le L, Zhang S, Harada ASM. Risk of acute myocardial infarction in patients treated with thiazolidinediones or other antidiabetic medications. Pharmacoepidemiol Drug Saf. 2009;18:166–174. doi: 10.1002/pds.1700 19109802

34. Tzoulaki I, Molokhia M, Curcin V, Little MP, Millett CJ, Ng A, et al. Risk of cardiovascular disease and all cause mortality among patients with type 2 diabetes prescribed oral antidiabetes drugs: retrospective cohort study using UK general practice research database. BMJ. 2009;339:b4731. doi: 10.1136/bmj.b4731 19959591

35. Walker AM, Koro CE, Landon J. Coronary heart disease outcomes in patients receiving antidiabetic agents in the PharMetrics database 2000–2007. Pharmacoepidemiol Drug Saf. 2008;17:760–768. doi: 10.1002/pds.1598 18383443

36. Wertz DA, Chang C-L, Sarawate CA, Willey VJ, Cziraky MJ, Bohn RL. Risk of cardiovascular events and all-cause mortality in patients treated with thiazolidinediones in a managed-care population. Circ Cardiovasc Qual Outcomes. 2010;3:538–545. doi: 10.1161/circoutcomes.109.911461 20736441

37. Winkelmayer WC, Setoguchi S, Levin R, Solomon DH. Comparison of cardiovascular outcomes in elderly patients with diabetes who initiated rosiglitazone vs pioglitazone therapy. Arch Intern Med. 2008;168:2368–2375. doi: 10.1001/archinte.168.21.2368 19029503

38. Ziyadeh N, McAfee AT, Koro C, Landon J, Arnold Chan K. The thiazolidinediones rosiglitazone and pioglitazone and the risk of coronary heart disease: a retrospective cohort study using a US health insurance database. Clin Ther. 2009;31:2665–2677. doi: 10.1016/j.clinthera.2009.11.003 20110009

39. Bak S, Andersen M, Tsiropoulos I, García Rodríguez LA, Hallas J, Christensen K, et al. Risk of stroke associated with nonsteroidal anti-inflammatory drugs: a nested case-control study. Stroke. 2003;34:379–386. doi: 10.1161/01.str.0000053029.45352.a0

40. Curtis JP, Wang Y, Portnay EL, Masoudi FA, Havranek EP, Krumholz HM. Aspirin, ibuprofen, and mortality after myocardial infarction: retrospective cohort study. BMJ. 2003;327:1322–1323. 14656840

41. Fischer LM, Schlienger RG, Matter CM, Jick H, Meier CR. Current use of nonsteroidal antiinflammatory drugs and the risk of acute myocardial infarction. Pharmacotherapy. 2005;25:503–510. doi: 10.1592/phco.25.4.503.61021 15977911

42. García Rodríguez LA, Varas C, Patrono C. Differential effects of aspirin and non-aspirin nonsteroidal antiinflammatory drugs in the primary prevention of myocardial infarction in postmenopausal women. Epidemiology. 2000;11:382–387. 10874543

43. García Rodríguez LA, Varas-Lorenzo C, Maguire A, González-Pérez A. Nonsteroidal antiinflammatory drugs and the risk of myocardial infarction in the general population. Circulation. 2004;109:3000–3006. doi: 10.1161/01.cir.0000132491.96623.04 15197149

44. Gislason GH, Jacobsen S, Rasmussen JN, Rasmussen S, Buch P, Friberg J, et al. Risk of death or reinfarction associated with the use of selective cyclooxygenase-2 inhibitors and nonselective nonsteroidal antiinflammatory drugs after acute myocardial infarction. Circulation. 2006;113:2906–2913. doi: 10.1161/circulationaha.106.616219 16785336

45. Graham DJ, Campen D, Hui R, Spence M, Cheetham C, Levy G, et al. Risk of acute myocardial infarction and sudden cardiac death in patients treated with cyclo-oxygenase 2 selective and non-selective non-steroidal anti-inflammatory drugs: nested case-control study. Lancet. 2005;365:475–481. doi: 10.1016/S0140-6736(05)70270-1 15705456

46. Hippisley-Cox J, Coupland C. Risk of myocardial infarction in patients taking cyclo-oxygenase-2 inhibitors or conventional non-steroidal anti-inflammatory drugs: population based nested case-control analysis. BMJ. 2005;330:1366. doi: 10.1136/bmj.330.7504.1366 15947398

47. Johnsen SP, Larsson H, Tarone RE, McLaughlin JK, Nørgård B, Friis S, et al. Risk of hospitalization for myocardial infarction among users of rofecoxib, celecoxib, and other nsaids: a population-based case-control study. Arch Intern Med. 2005;165:978–984. doi: 10.1001/archinte.165.9.978 15883235

48. Kimmel SE, Berlin JA, Reilly M, Jaskowiak J, Kishel L, Chittams J, et al. The effects of nonselective non-aspirin non-steroidal anti-inflammatory medications on the risk of nonfatal myocardial infarction and their interaction with aspirin. J Am Coll Cardiol. 2004;43:985–990. doi: 10.1016/j.jacc.2003.08.064 15028354

49. Kimmel SE, Berlin JA, Reilly M, Jaskowiak J, Kishel L, Chittams J, et al. Patients exposed to rofecoxib and celecoxib have different odds of nonfatal myocardial infarction. Ann Intern Med. 2005;142:157–164. doi: 10.7326/0003-4819-142-3-200502010-00005 15684203

50. Lévesque LE, Brophy JM, Zhang B. The risk for myocardial infarction with cyclooxygenase-2 inhibitors: a population study of elderly adults. Ann Intern Med. 2005;142:481–489. doi: 10.7326/0003-4819-142-7-200504050-00113 15809459

51. MacDonald TM, Wei L. Effect of ibuprofen on cardioprotective effect of aspirin. Lancet. 2003;361:573–574. doi: 10.1016/S0140-6736(03)12509-3 12598144

52. Mamdani M, Rochon P, Juurlink DN, Anderson GM, Kopp A, Naglie G, et al. Effect of selective cyclooxygenase 2 inhibitors and naproxen on short-term risk of acute myocardial infarction in the elderly. Arch Intern Med. 2003;163:481–486. doi: 10.1001/archinte.163.4.481 12588209

53. McGettigan P, Han P, Henry D. Cyclooxygenase-2 inhibitors and coronary occlusion—exploring dose–response relationships. Br J Clin Pharmacol. 2006;62:358–365. doi: 10.1111/j.1365-2125.2006.02660.x 16934052

54. Ray WA, Stein CM, Hall K, Daugherty JR, Griffin MR. Non-steroidal anti-inflammatory drugs and risk of serious coronary heart disease: an observational cohort study. Lancet. 2002;359:118–123. doi: 10.1016/S0140-6736(02)07370-1 11809254

55. Ray WA, Stein CM, Daugherty JR, Hall K, Arbogast PG, Griffin MR. COX-2 selective non-steroidal anti-inflammatory drugs and risk of serious coronary heart disease. Lancet. 2002;360:1071–1073. doi: 10.1016/S0140-6736(02)11131-7 12383990

56. Schlienger RG, Jick H, Meier CR. Use of nonsteroidal anti-inflammatory drugs and the risk of first-time acute myocardial infarction. Br J Clin Pharmacol. 2002;54:327–332. doi: 10.1046/j.1365-2125.2002.01637.x 12236854

57. Solomon DH, Glynn RJ, Levin R, Avorn J. Nonsteroidal anti-inflammatory drug use and acute myocardial infarction. Arch Intern Med. 2002;162:1099–1104. doi: 10.1001/archinte.162.10.1099 12020178

58. Solomon DH, Schneeweiss S, Glynn RJ, Kiyota Y, Levin R, Mogun H, et al. Relationship between selective cyclooxygenase-2 inhibitors and acute myocardial infarction in older adults. Circulation. 2004;109:2068–2073. doi: 10.1161/01.cir.0000127578.21885.3e 15096449

59. Watson DJ, Rhodes T, Cai B, Guess HA. Lower risk of thromboembolic cardiovascular events with naproxen among patients with rheumatoid arthritis. Arch Intern Med. 2002;162:1105–1110. doi: 10.1001/archinte.162.10.1105 12020179

60. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. doi: 10.2307/2529310 843571

61. Nissen SE, Wolski K. Rosiglitazone revisited: an updated meta-analysis of risk for myocardial infarction and cardiovascular mortality. Arch Intern Med. 2010;170:1191–1201. doi: 10.1001/archinternmed.2010.207 20656674

62. Katikireddi SV, Egan M, Petticrew M. How do systematic reviews incorporate risk of bias assessments into the synthesis of evidence? A methodological study. J Epidemiol Community Health. 2015;69:189–195. doi: 10.1136/jech-2014-204711 25481532

63. Bernatsky S, Slim Z, Yuwan M. ACPA and future onset of rheumatoid arthritis among individuals with undifferentiated arthritis and arthritis free individuals: a systematic review of cohort studies. J Autoimmune Dis Rheumatol. 2015;3:30–40.

64. Kalkhoran S, Glantz S. E-cigarettes and smoking cessation in real-world and clinical settings: a systematic review and meta-analysis. Lancet Respir Med. 2016;4:116–128. doi: 10.1016/s2213-2600(15)00521-4 26776875

65. Emmett C, Close H, Yiannakou Y, Mason J. Trans-anal irrigation therapy to treat adult chronic functional constipation: systematic review and meta-analysis. BMC Gastroenterol. 2015;15:139. doi: 10.1186/s12876-015-0354-7 26474758

66. Staton C, Vissoci J, Gong E, Toomey N, Wafula R, Abdelgadir J, et al. Road traffic injury prevention initiatives: a systematic review and metasummary of effectiveness in low and middle income countries. PLoS ONE. 2016;11:e0144971. doi: 10.1371/journal.pone.0144971 26735918

67. Thomson H, Campbell M, Craig P, Hilton Boon M, Katikireddi V, editors. ACROBAT-NRSI for public health: reporting on feasibility and utility of applying ACROBAT to studies of housing improvement. Cochrane Colloquium 2015; 3–7 Oct 2015; Vienna, Austria.

68. Morgan R, Thayer K, Guyatt G, Blain R, Eftim S, Ross P, et al. Assessing the usability of ACROBAT-NRSI for studies of exposure and intervention in environmental health research. Cochrane Colloquium 2015; 3–7 Oct 2015; Vienna, Austria.

69. Couto E, Pike E, Torkilseng E, Klemp M. Inter-rater reliability of the Risk of Bias Assessment Tool: for Non-Randomized Studies of Interventions (ACROBAT-NRSI). Cochrane Colloquium 2015; 3–7 Oct 2015; Vienna, Austria.

70. Whiting P, Savović J, Higgins J, Caldwell D, Reeves B, Shea B, et al. ROBIS: a new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol. 2016;69:225–234. doi: 10.1016/j.jclinepi.2015.06.005 26092286

71. Shea B, AMSTAR Development Group. AMSTAR: helping decision makers distinguish high and low quality systematic reviews that include non-randomized studies. Cochrane Colloquium 2015; 3–7 Oct 2015; Vienna, Austria.