Agreement between the Cochrane risk of bias tool and Physiotherapy Evidence Database (PEDro) scale: A meta-epidemiological study of randomized controlled trials of physical therapy interventions

English version

Autoři: Anne M. Moseley ^aff001; Prinon Rahman ^aff002; George A. Wells ^aff003; Joshua R. Zadro ^aff001; Catherine Sherrington ^aff001; Karine Toupin-April ^aff004; Lucie Brosseau ^aff002
Působiště autorů: Institute for Musculoskeletal Health, Sydney School of Public Health, Faculty of Medicine and Health, The University of Sydney, Sydney, New South Wales, Australia ^aff001; Physiotherapy Program, School of Rehabilitation Sciences, Faculty of Health Sciences, University of Ottawa, Ottawa, Ontario, Canada ^aff002; School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada ^aff003; Children’s Hospital of Eastern Ontario Research Institute, Ottawa, Ontario, Canada ^aff004; Department of Pediatrics, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada ^aff005; School of Rehabilitation Sciences, Faculty of Health Sciences, University of Ottawa, Ottawa, Ontario, Canada ^aff006
Vyšlo v časopise: PLoS ONE 14(9)
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pone.0222770

Souhrn

Background

The Cochrane risk of bias (CROB) tool and Physiotherapy Evidence Database (PEDro) scale are used to evaluate risk of bias of randomized controlled trials. We assessed the level of agreement between the instruments.

Methods

We searched the Cochrane Library to identify trials included in systematic reviews evaluating physical therapy interventions. For trials that met our inclusion criteria (primary reference in Cochrane review, review used CROB (2008 version), indexed in PEDro), CROB items were extracted from the reviews and PEDro items and total score were downloaded from PEDro. Kappa statistics were used to determine the agreement between CROB and PEDro scale items that evaluate similar constructs (e.g., randomization). The total PEDro score was compared to the CROB summary score (% of items met) using an Intraclass Correlation Coefficient. Sensitivity analyses explored the impact of the CROB “unclear” category and variants of CROB blinding items. Kappa statistics were used to determine agreement between different thresholds for “acceptable” risk of bias between CROB and PEDro scale summary scores.

Results

We included 1442 trials from 108 Cochrane reviews. Agreement was “moderate” for three of the six CROB and PEDro scale items that evaluate similar constructs (allocation concealment, participant blinding, assessor blinding; Kappa = 0.479–0.582). Agreement between the summary scores was “poor” (Intraclass Correlation Coefficient = 0.285). Agreement was highest when the CROB “unclear” category was collapsed with “high” and when participant, personnel and assessor blinding were evaluated separately in CROB. Agreement for different thresholds for “acceptable” risk of bias between CROB and PEDro summary scores was, at best, “fair”.

Conclusion

There was moderate agreement for half of the PEDro and CROB items that evaluate similar constructs. Interpretation of the CROB “unclear” category and variants of the CROB blinding items substantially influenced agreement. Either instrument can be used to quantify risk of bias, but they can’t be used interchangeably.

Klíčová slova:

Research and analysis methods – Database and informatics methods – Bioinformatics – Sequence analysis – Medicine and health sciences – Health care – Pharmacology – Clinical medicine – Public and occupational health – Clinical trials – Drug research and development – Research assessment – Systematic reviews – Sequence databases – Biological databases – Complementary and alternative medicine – Randomized controlled trials – Physiotherapy – Exercise therapy

Zdroje

1. Sackett D. Evidence-based medicine. Lancet. 1995; 346(8983):1171. 7475646.

2. Gray JAM. Evidence-based healthcare and public health: how to make decisions about health services and public health. 3rd ed ed. Edinburgh: Churchill Livingstone/Elsevier; 2009. doi: 10.1186/1471-2458-9-419

3. Higgins JPT, Green S. Cochrane handbook for systematic reviews of interventions version 5.1.0: Cochrane Collaboration; 2011. Available from: http://handbook-5-1.cochrane.org/.

4. Maher CG, Sherrington C, Herbert RD, Moseley AM, Elkins M. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys Ther. 2003; 83(8):713–721. 12882612.

5. Hartling L, Ospina M, Liang Y, Dryden DM, Hooton N, Krebs Seida J, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ. 2009; 339:b4012. doi: 10.1136/bmj.b4012 19841007.

6. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013; 66(9):973–981. doi: 10.1016/j.jclinepi.2012.07.005 22981249.

7. Armijo-Olivo S, Ospina M, da Costa BR, Egger M, Saltaji H, Fuentes J, et al. Poor reliability between Cochrane reviewers and blinded external reviewers when applying the Cochrane risk of bias tool in physical therapy trials. PLoS One. 2014; 9(5):e96920. doi: 10.1371/journal.pone.0096920 24824199.

8. da Costa BR, Beckett B, Diaz A, Resta NM, Johnston BC, Egger M, et al. Effect of standardized training on the reliability of the Cochrane risk of bias assessment tool: a prospective study. Syst Rev. 2017; 6(44):Epub. doi: 10.1186/s13643-017-0441-7 28253938.

9. Elkins MR, Moseley AM, Sherrington C, Herbert RD, Maher CG. Growth in the Physiotherapy Evidence Database (PEDro) and use of the PEDro scale. Br J Sports Med. 2013; 47(4):188–189. doi: 10.1136/bjsports-2012-091804 23134761.

10. Macedo LG, Elkins MR, Maher CG, Moseley AM, Herbert RD, Sherrington C. There was evidence of convergent and construct validity of Physiotherapy Evidence Database quality scale for physiotherapy trials. J Clin Epidemiol. 2010; 63(8):920–925. doi: 10.1016/j.jclinepi.2009.10.005 20171839.

11. Tooth L, Bennett S, McCluskey A, Hoffmann T, McKenna K, Lovarini M. Appraising the quality of randomized controlled trials: inter-rater reliability for the OTseeker evidence database. J Eval Clin Pract. 2005; 11(6):547–555. doi: 10.1111/j.1365-2753.2005.00574.x 16364108.

12. Foley NC, Bhogal SK, Teasell RW, Bureau Y, Speechley MR. Estimates of quality and reliability with the physiotherapy evidence-based database scale to assess the methodology of randomized controlled trials of pharmacological and nonpharmacological interventions. Phys Ther. 2006; 86(6):817–824. 16737407.

13. Murray E, Power E, Togher L, McCabe P, Munro N, Smith K. The reliability of methodological ratings for speechBITE using the PEDro-P scale. Int J Lang Commun Disord. 2013; 48(3):297–306. doi: 10.1111/1460-6984.12007 23650886.

14. van Peppen RP, Kwakkel G, Wood-Dauphinee S, Hendriks HJ, van der Wees PJ, Dekker J. The impact of physical therapy on functional outcomes after stroke: what's the evidence? Clin Rehabil. 2004; 18(8):833–862. doi: 10.1191/0269215504cr843oa 15609840.

15. Brandt C, Sole G, Krause MW, Nel M. An evidence-based review on the validity of the Kaltenborn rule as applied to the glenohumeral joint. Man Ther. 2007; 12(1):3–11. doi: 10.1016/j.math.2006.02.011 16777466.

16. de Morton NA. The PEDro scale is a valid measure of the methodological quality of clinical trials: a demographic study. Aust J Physiother. 2009; 55(2):129–133. 19463084.

17. Yamato TP, Maher C, Koes B, Moseley A. The PEDro scale had acceptably high convergent validity, construct validity, and interrater reliability in evaluating methodological quality of pharmaceutical trials. J Clin Epidemiol. 2017; 86:176–181. doi: 10.1016/j.jclinepi.2017.03.002 28288916.

18. Armijo-Olivo S, da Costa BR, Cummings GG, Ha C, Fuentes J, Saltaji H, et al. PEDro or Cochrane to assess the quality of clinical trials? A meta-epidemiological study. PLoS One. 2015; 10(7):e0132634. doi: 10.1371/journal.pone.0132634 26161653.

19. Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychol Bull. 1955; 52(4):281–302. 13245896.

20. Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull. 1959; 56(2):81–105. 13634291.

21. Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evid Based Med. 2017; 22(4):139–142. doi: 10.1136/ebmed-2017-110713 28701372.

22. National Institute for Health Research. PROSPERO: international prospective register of systematic reviews 2019 [23 July 2019]. Available from: https://www.crd.york.ac.uk/prospero/.

23. Chmura Kraemer H, Periyakoil VS, Noda A. Kappa coefficients in medical research. Stat Med. 2002; 21(14):2109–2129. doi: 10.1002/sim.1180 12111890.

24. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33(1):159–174. 843571.

25. Fleiss JL. The design and analysis of clinical experiments. New York, NY: John Wiley & Sons Inc; 1986.

26. Juni P, Altman DG, Egger M. Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ. 2001; 323(7303):42–46. doi: 10.1136/bmj.323.7303.42 11440947.

27. Armijo Olivo SA, Macedo LG, Gadotti IC, Fuentes J, Stanton T, Magee DJ. Scales to assess the quality of randomized controlled trials: a systematic review. Phys Ther. 2008; 88(2):156–175. doi: 10.2522/ptj.20070147 18073267.

28. Armijo-Olivo S, Fuentes J, Ospina M, Saltaji H, Hartling L. Inconsistency in the items included in tools used in general health research and physical therapy to evaluate the methodological quality of randomized controlled trials: a descriptive analysis. BMC Med Res Methodol. 2013; 13(116):Epub. doi: 10.1186/1471-2288-13-116 24044807.

29. Fleiss JL, Levin B, Paik MC. The measurement of interrater agreement (chapter 18). Statistical methods for rates and proportions. Third edition ed: John Wiley & Sons Inc; 2003. p. 598–626.

30. Pandis N. Randomization. Part 1: sequence generation. Am J Orthod Dentofacial Orthop. 2011; 140(5):747–748. doi: 10.1016/j.ajodo.2011.06.020 22051497.

31. Pandis N. Randomization. Part 3: allocation concealment and randomization implementation. Am J Orthod Dentofacial Orthop. 2012; 141(1):126–128. doi: 10.1016/j.ajodo.2011.09.003 22196195.

32. Page SJ, Persch AC. Recruitment, retention, and blinding in clinical trials. Am J Occup Ther. 2013; 67(2):154–161. doi: 10.5014/ajot.2013.006197 23433269.

33. Jordan VMB, Lensen SF, Farquhar CM. There were large discrepancies in risk of bias tool judgments when a randomized controlled trial appeared in more than one systematic review. J Clin Epidemiol. 2017; 81:72–76. doi: 10.1016/j.jclinepi.2016.08.012 27622779.

34. Higgins JPT, Savović J, Page MJ, Sterne JAC, on behalf of the ROB2 Development Group. Revised Cochrane risk-of-bias tool for randomized trials (RoB 2) 2019 [23 July 2019]. Available from: https://methods.cochrane.org/risk-bias-20-tool.