Background and purpose: Systematic reviews and meta-analyses often include an evaluation of the methodological quality of the individual studies that have been included, and are usually conducted by at least 2 individuals. The objective of this study was to assess the methodological quality and reliability of a series of randomized controlled trials (RCTs) of both pharmacological and nonpharmacological interventions by use of the 10-item Physiotherapy Evidence-Based Database (PEDro) Scale.
Methods: Two abstractors independently reviewed 81 RCTs assessing a variety of interventions. The Cohen kappa statistic and the intraclass correlation coefficient (ICC) were used to assess agreement between abstractors.
Results: The average total PEDro scores were 5.94 (SD=1.43) for all studies combined, 6.88 (SD=1.2) for pharmacological studies, and 5.29 (SD=1.26) for nonpharmacological studies. The median score for pharmacological studies was significantly higher than that for nonpharmacological studies (7 versus 5). Pair-wise kappa scores ranged from a low of .452 for concealed allocation among drug trials to perfect agreement (1.00) for randomization and reporting of results from between-group comparisons. The ICCs associated with the cumulative PEDro score were .91 (95% confidence interval [CI]=.83-.94) for all studies, .89 (95% CI=.78-.95) for pharmacological studies, and .91 (95% CI=.84-.952) for nonpharmacological studies.
Discussion and conclusion: The methodological quality for pharmacological interventions was significantly higher than that for nonpharmacological interventions. There was good agreement between raters at an individual item level and in total PEDro scores. A lack of reporting clarity, poor organization of the report, or the failure to include salient details contributed to less-than-perfect agreement between raters.