Context: Colleague surveys are important sources of information on a doctor's professional performance in UK revalidation plans. Colleague surveys are analysed by deriving quantitative measures from rating scales. As free-text comments are also recorded, we explored the utility of a mixed-methods approach to their analysis.
Methods: A volunteer sample of practising UK doctors (from acute, primary and other care settings) undertook a General Medical Council (GMC) colleague survey. Up to 20 colleagues per doctor completed an online Colleague Questionnaire (CQ), which included 18 performance evaluation items and an optional comment box. The polarity of each comment was noted and a qualitative content analysis undertaken. Emerging themes were mapped onto existing items to identify areas not previously captured. We then quantitatively analysed the associations between the polarity of comments (positive/adverse) and their related item scale scores.
Results: A total of 1636 of 4269 (38.3%) colleagues recorded free-text comments (median = 14 per doctor) and most were unequivocally positive; only 127 of 1636 (7.8%) recorded negative statements and these were clustered on a subset comprising 80 of 302 (26.5%) doctors. Doctors' overall mean CQ performance scores were significantly correlated with the numbers of colleagues recording positive (r = 0.35; P < 0.0001) and adverse (r = - 0.40; P = 0.0003) comments. In total, 1224 of 1636 (74.8%) comments included statements that mapped on CQ items, and statistically significant associations (P < 0.05) were observed for 14 of 15 items. Five global themes (innovativeness, interpersonal skills, popularity, professionalism, respect) were identified in 904 of 1636 (73.9%) comments.
Conclusions: There is an inevitable trade-off between the capturing of indicators of problematic performance (i.e. adverse statements which contradict a positive scale rating) and the ease with which such statements can be identified. Our data suggest there is little benefit in routinely analysing narrative comments for the purposes of revalidation.