NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Michie S, Wood CE, Johnston M, et al. Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data). Southampton (UK): NIHR Journals Library; 2015 Nov. (Health Technology Assessment, No. 19.99.)

Cover of Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data)

Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data).

Show details

Chapter 4Training to code intervention descriptions using Behaviour Change Technique Taxonomy version 1 (study 3)

Abstract

Objectives: To evaluate two programmes of user training in improving reliable, valid and confident application of BCTTv1 to code BCTs in descriptions of BCIs.

Methods: 161 trainees (109 in face-to-face workshops and 52 in distance group tutorials) were trained to code frequently occurring BCTs. Training was evaluated by comparing three measures before and after training: (1) intercoder agreement, (2) trainee agreement with expert consensus and (3) confidence ratings. Coding was assessed for 12 BCTs in workshops and for 17 BCTs in tutorials. Trainees also completed a course evaluation.

Results: Workshop and tutorial training improved trainee agreement with expert consensus [workshops: mean prevalence- and bias-adjusted kappa (PABAK) before training = 0.39, after training = 0.50; tutorials: mean PABAK score = 0.57, after training = 0.72; both p < 0.05] but not intercoder agreement (p = 0.08 and p = 0.57, respectively) and increased confidence ratings for BCTs assessed in workshops (mean number of assessed BCTs, identified with high confidence before training = 8.38, after training = 9.56; p < 0.001). Training was evaluated positively by trainees; all components of both types of training were highly rated in terms of usefulness.

Conclusions: Both methods of training improved trainee agreement with expert consensus and confidence for BCTs assessed, but neither method improved intercoder agreement. This varied according to BCT.

Introduction

In order to enhance understanding and use of BCTs in BCIs, we have improved the quality of their labels and definitions, and developed a hierarchical structure of BCTTv1.73 However, it is not only the content and form of the taxonomy that drives reliable and valid application of BCT labels/definitions but also the extent to which the user has been trained to use the taxonomy. Identifying (coding) BCTs involves a deductive process of categorising qualitative information (e.g. descriptions of interventions) using an established coding framework and instructions. The process of coding BCTs is a highly skilled task requiring familiarity with the BCT labels and definitions and one which requires coders to make a series of complex interpretative judgements.87,88 Achievement of good intercoder reliability (i.e. the extent to which coders agree on the presence/absence of BCTs identified in intervention descriptions using the taxonomy as a coding framework) is therefore not only a function of the clarity of the taxonomy and its coding guidelines, but also of the competence of its coders.

Intercoder reliability has been demonstrated in using BCT taxonomies among coders with varying amounts of training and experience.20,23,2830,48 Training in taxonomies, when reported, has generally involved manual-based coding instructions, prompting of practice and one-to-one feedback from those familiar with using the taxonomy. Delivery and intensity of training has varied, with some coders receiving one-to-one feedback from experienced BCT coders whereas others have trained themselves using coding manuals. Systematic documentation of the training process and evaluation, involving the comparison of coding competence to apply BCT taxonomies with reliability and validity before and after training, will establish whether or not training can enhance coding competence.

One of the goals of training in the use of the taxonomy is to teach coders to recognise BCTs as defined by the taxonomy rather than relying on their own subjective judgements that are triggered by the BCT label.89,90 Another goal of training is to enhance the ‘validity’ of coder judgements, that is the extent to which coders agree with expert BCT coders as to the presence or absence of BCTs. Expert coding, assessed here as a consensus between expert coders, is the nearest we have to an objective standard of ‘validity’. An effective training programme is therefore one that not only enhances intercoder agreement between trainees but also enhances agreement with expert consensus about BCTs identified.

The research literature suggests that collaborative or co-operative training strategies (i.e. working together in small groups towards a common goal) and active learning techniques such as discussion are more effective than traditional, lecture-style training for acquiring new knowledge, building skills and increasing motivation to continue to improve skills.91 An effective training programme is built on four basic principles: (1) setting training goals involving provision of information or concepts to be learned, (2) demonstration of knowledge and skills to be learned, (3) practice or rehearsal of skills learned and (4) provision of feedback to trainees during and after practice.92,93 Skills are more likely to be retained and improve future practice if trainees feel challenged, receive positive feedback and find the learning process interesting and enjoyable.93,94 This evidence and these principles clearly reflect the relevance and use of BCTs such as feedback on behaviour, goal-setting, social support and behavioural practice/rehearsal directed at changing skilled behaviours. Coder training incorporating these principles and BCTs has previously been evaluated in applying the taxonomy to specify BCT content in written intervention descriptions of behavioural support in smoking cessation, and training in this context was a 3-hour workshop delivered to a mixture of research psychologists and non-psychologist practitioners.95 This study found that training improved trainees’ coding competence (i.e. their agreement with expert consensus about which BCTs are present).

Developing user training for applying Behaviour Change Technique Taxonomy version 1

The increasing use of the BCT approach, in particular for specifying interventions in evidence synthesis, has prompted high demand for training in the reliable and valid application of BCTTv1. In response to this, training programmes have been developed building on previous BCT coder training conducted by the study team and integrating the principles drawn from the literature.9295 These training programmes are evaluated in this chapter. They were designed to train coders to identify the most frequently occurring BCTs from BCTTv1, that is those that were found to occur most frequently in intervention descriptions. The decision was made to train coders in the most frequently occurring BCTs as training in 93 BCTs was not feasible within the proposed training intensity and duration, and because frequently occurring BCTs would be more accessible to trainees as well as more useful for them to learn. Two methods of training were developed: workshops, which involved face-to-face group training for 1 day, and distance tutorials, which were usually delivered by telephone to groups of two to four people in four, 1-hour sessions held over 1–2 months. The latter allowed training of coders across the world. Two substudies evaluated these training programmes and addressed the following research questions (RQs):

Does face-to-face training (1-day workshops) and distance training (group tutorials) improve the specification of BCIs by BCT as assessed by increased:

  1. Intercoder agreement about BCTs identified?
  2. Agreement with BCTs identified by expert consensus?
  3. Confidence ratings for BCTs identified as present?
  4. Do trainees evaluate BCTTv1 training as a useful experience?
  5. What proportion of trainees reach an acceptable standard of competence following training?

One-day workshops

Method

This study is also published as Wood et al.96 Some of the text in this section is reproduced from Wood et al.96 © The Author(s) 2014. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution and reproduction in any medium, provided the original work is properly cited.

Design

Coding competence was assessed before and after each workshop by assessment tasks and evaluation of usefulness was assessed at the end of the training in a confidential questionnaire. To overcome potential practice effects, the assessments were administered in a counterbalanced design, so that a random 50% of trainees completed assessment task A at the beginning and B at the end, and the other half completed B followed by A.

Participants

Workshops were offered to those interested in investigating, reviewing, designing or delivering behavioural interventions, and no previous knowledge or experience was required. They were advertised via scientific and professional organisations and the BCT Taxonomy Project website.97 Coders who had been involved in the development of BCTTv1 or in the sort tasks to determine hierarchical structure were not eligible for this training. Five workshops were conducted with groups of between 9 and 29 trainees (n = 109). A total of 64.9% were from the UK, 21.6% from other European countries, 7.2% from the USA, 4.5% from Asia, 0.9% from Australia and 0.9% from South Africa. A total of 35% of trainees had obtained a research or clinical doctorate and 11% identified themselves as active practitioners in their field. Trainees ranged in age from 21 years to 59 years (mean = 32.31 years, SD = 9.27 years) and 95% of trainees were female (Table 5).

TABLE 5

TABLE 5

Demographic information for one-day workshop and distance group tutorial trainees

Materials

Coding manuals of previous taxonomies20,95 were used to inform the development of the workshop training programme. Training involved trainees watching three short Microsoft PowerPoint® 1997–2003 (Microsoft Corporation, Redmond, WA, USA) presentations and participating in a series of interactive coding tasks as a group, individually and in pairs. Workshop tasks were delivered according to a number of different formats, for example via a ‘ready, steady, point!’ task in which trainees were shown a short excerpt on the presentation screen and, when prompted, were asked to point to the left if BCT X was present, to the right if BCT Y was present, or to the ceiling if they were unsure (Table 6). Content was structured around a series of learning objectives (e.g. ‘to learn the need for precise labels and definitions’ and ‘to learn appropriate levels of inference and discrimination’). The training was designed using BCTs associated with key learning principles and with changing skilled behaviour (e.g. graded tasks, behavioural practice/rehearsal, instruction on how to perform the behaviour, feedback on behaviour; for a full list of BCTs used, see Table 6).

TABLE 6

TABLE 6

Summary of 1-day workshop content and learning objectives

Trainees were taught 24 of the frequently identified BCTs from BCTTv1. Furthermore, the early part of the workshop focused on simple coding tasks working up to more difficult tasks in an attempt to approach ‘errorless learning’ as this has been shown to be effective in learning new terms.98 Each workshop was delivered by two experienced BCT coders (BCTTv1 project team members).

Measures

Prior experience

Trainees were asked if they had previously (1) designed or reported BCIs that specifically identified BCTs, (2) been involved in writing manuals or protocols of interventions, and (3) undertaken a narrative or systematic review of behaviour change literature. They also rated their expertise (i.e. knowledge, skills and familiarity) in the areas of designing, writing, reporting and systematic reviewing of BCIs using response options from 1 (‘no experience’) to 5 (‘a great deal of experience’).

Evaluating training effectiveness in increasing coding competence: coding competence for 12 frequently occurring BCTs (see Table 8) was assessed before and after training. Trainees were asked to identify the presence/absence of BCTs in descriptions of two BCIs that targeted physical activity and safe needle cleaning behaviour, respectively. They were asked to rate how confident they were that the BCT was present using +/++; whereby ‘+’ represented ‘BCT present in all probability but evidence not clear’ and ‘++’, ‘BCT present beyond all reasonable doubt; clear evidence available’. The descriptions were written (by CA and MR; see Acknowledgements) to exemplify particular BCTs, to highlight the learning principles and to ensure inclusion of the frequent BCTs targeted for training.

TABLE 8

TABLE 8

One-day workshops: before and after training agreement between coders, coder agreement with expert consensus and confidence in BCT identification

To assess trainees’ agreement with expert consensus, six experienced BCTTv1 coders who had been involved in developing BCTTv1 (study team members: MJ, SM, JF, WH, CA, MR), working in pairs, independently coded the descriptions using BCTTv1. Expert consensus was developed by discussion of any discrepancies within each of the pairs. If a resolution was not obvious, SM and the study researcher (MR) reviewed the remaining discrepancies and proposed a coding. The list of BCT codes resulting from this process was circulated to the whole study team who agreed the final codes. We used this consensus about the presence of BCTs in the descriptions as a criterion against which trained coder codings were judged and validity was assessed. The coders reached consensus about the presence of 12 BCTs in the descriptions: self-monitoring of behaviour, feedback on behaviour, behavioural practice/rehearsal, non-specific reward, goal-setting (outcome), material reward (behaviour), credible source, problem-solving, demonstration of the behaviour, information about health consequences, goal-setting (behaviour) and social support (unspecified).

Training effectiveness was evaluated by changes in intercoder agreement, in trainee agreement with expert consensus, in the proportion of high (i.e. ++) confidence ratings for the 12 BCTs assessed and in the proportion of trainees reaches an acceptable standard of competence. Effectiveness was assessed for 12 frequently observed BCTs. This task was considered an appropriate ‘challenge’ for trainees and one that was in line with the length and intensity of training undertaken. Agreement was assessed using PABAK99 (see Analysis). An acceptable standard of competence was defined in terms of trainee agreement with expert consensus (see Analysis for rationale).

Evaluating trainee experience of training: trainees rated the usefulness of each of the presentations, individual and group tasks in helping them to build skill and knowledge, using response options from 1, ‘not useful’ to 5 ‘useful’. All trainees were also asked to respond to four open-ended questions: (1) what part(s) of the training did you find the most useful?; (2) what part(s) or aspect(s) of the training, if any, did you find least useful?; (3) would you like future training? If so, do you have a specific proposal?; and (4) please provide any other feedback about using BCT taxonomies.

Procedure

Prior to attending the workshop, all trainees were sent two articles as preparatory reading.17,100 At the start of the workshop, they were asked to complete a questionnaire with demographic information (age, sex, nationality, professional background and highest qualification). All trainees completed an assessment of their coding competence before and after training. They also completed a training evaluation questionnaire. They received a BCTTv1 training certificate at the end of the workshop and individual feedback on their coding competence via e-mail.

Distance group tutorials

Method

Design

Training was conducted over four, 1-hour sessions. Training was held over an average period of 6 weeks with a minimum of 1 week between sessions. Each group was led by an experienced BCT tutor and for 9 of the 10 groups took place via teleconference call. The 10 tutors included members of the BCTTv1 study team who had been involved in development of BCTTv1 (SM, MJ, JF, WH and MR) and five experts in behaviour change recruited via the BCTTv1 Project website.97 The five experts had all been involved on at least one occasion as part of their own research and practice, in designing and reporting BCIs specifically identifying BCTs, in writing manuals or protocols of interventions, or had undertaken narrative or systematic reviews of published behaviour change studies. Trainees’ coding competence was assessed before and after training by assessment tasks and evaluation of usefulness was assessed after training in a confidential questionnaire.

Participants

Tutorial training was advertised as for workshops. BCTTv1 study team members were also asked to identify potential trainees from their own teams and networks. Those interested in taking part were asked to complete a self-evaluation form. Trainees (n = 52) were invited to join if they indicated that they had some previous experience in investigating, designing and/or delivering BCIs and were available over the training period. Those who had been involved in study 1 and/or 2 were not eligible for this training. Trainees were contacted via e-mail and offered an honorarium of £560 on completion of a coding task for research purposes (see Chapter 5) (estimated to take 2 days) following the group tutorial training programme. A total of 71% were from the UK, 15% from other European countries, 8% from the USA and 6% from Australia. A total of 81% had obtained a research or clinical doctorate and 13% identified themselves as active practitioners in their field. Trainees ranged in age from 24 years to 60 years (mean = 37.04 years, SD = 7.82 years) and 67% of the sample were female (see Table 5).

Previous experience: 24% (n = 26) of trainees had previously used a BCT taxonomy to code BCIs and 32% (n = 35) to describe BCIs. Trainees reported a low-to-moderate level of expertise associated with BCIs (mean = 2.62, SD = 0.86), with significant variation between workshops.

Materials

Training was structured around the same learning objectives as workshops (Table 7) with the manual adapted for the format and covering a more extensive range of 44 BCTs from BCTTv1. Training was piloted in a face-to-face format with one group of trainees and these data were included in the analyses as few changes were made. All other tutorial groups were conducted via teleconference call.

TABLE 7

TABLE 7

Summary of learning principles and objectives for group tutorials

Tutorials were held over an average period of 6 weeks with a minimum of 1 week between each session. Each tutorial group had four trainees who were paired into two sets of ‘buddies’. The purpose of the buddy system was to provide trainees with both practical and emotional social support throughout the training and to foster independent problem-solving. Before each tutorial session, trainees independently completed and submitted a preparatory coding task (comprising one, two or three short exercises) before discussing it with their ‘buddy’. Each pair was asked to discuss their homework in advance, identifying key issues to discuss at the tutorial session in their ‘learning log’ to maximise the usefulness of the session and increase time for discussion and reflection. Discrepancies in coding between the trainees and BCTs agreed on as present by expert consensus were identified by the expert tutor and discussed during the session.

Measures

Evaluating training effectiveness in increasing competence: trainees were asked to identify the presence or absence of BCTs in intervention descriptions before and after training. Two descriptions were used (one for the before-training assessment and one for the after-training assessment), with the target behaviour for both descriptions being increasing physical activity. As for the workshops, descriptions were written by CA and MR to exemplify specific BCTs to highlight the learning principles and to ensure inclusion of the frequent BCTs targeted in training. Expert consensus was reached about the presence of 17 BCTs in each of the two descriptions (see Table 9): feedback on behaviour, credible source, information about health consequences, information about social and environmental consequences, social support (unspecified), monitoring of outcome(s) of behaviour without feedback, non-specific reward, demonstration of the behaviour, adding objects to the environment, goal-setting (behaviour), problem-solving, self-monitoring of outcome of the behaviour, goal-setting (outcome), behavioural practice/rehearsal, self-monitoring of behaviour, instruction on how to perform the behaviour and material reward (behaviour).

TABLE 9

TABLE 9

Distance group tutorials: before and after training intercoder agreement, trainee agreement with expert consensus and confidence ratings

Trainees were provided with a training taxonomy (a shortened version of BCTTv1 comprising 44 BCTs) and coded the presence/absence of BCTs in the descriptions, rating their confidence in each BCT identification using the same +/++ ratings as for workshops.

Training effectiveness was evaluated by changes in intercoder agreement, in agreement with expert consensus and in the proportion of high (i.e. ++) confidence ratings for the 17 BCTs assessed. As additional evaluators, tutorial trainees completed measures of perceived confidence and reported intentions to use a taxonomy to code reports and describe BCIs, using response options ranging from 1 (‘strongly disagree’) to 5 (‘strongly agree’), before and after training. An acceptable standard of competence was defined in terms of trainee agreement with expert consensus (see Analysis).

Evaluating trainee experience of training: using the same response options as for 1-day workshops, trainees rated perceived usefulness of the reading materials provided prior to the first tutorial session, the materials provided for sessions 1 to 4, the content and the structure of the sessions and the preparatory coding tasks completed prior to each session, the buddy system and the learning log. They completed the same open-ended evaluation items as for workshops.

Procedure

Prior to their first tutorial session, trainees were sent the same preparatory reading and completed the same measures as for workshops. Tutorial trainees were also asked to read a short, introductory PowerPoint presentation on the advantages and challenges associated with the use of the BCT approach in specifying the content of BCIs. All trainees completed the coding competence assessment (see Measures) before training. After the final session, trainees completed the after-training coding competence assessment and a training evaluation questionnaire. Trainees received individual feedback on coding competence, generic feedback on the common coding discrepancies encountered within their coding groups across the four sessions and a BCTTv1 training certificate via e-mail.

Analysis

For both workshops and tutorials, the following analyses were conducted.

Intercoder agreement about BCTs identified was assessed by using the PABAK; see Byrt et al.99 PABAK was used rather than Cohen’s kappa statistic101 or percentage agreement because it adjusts for (1) coders sharing bias in the use of categories and (2) high prevalence of negative agreement (i.e. when both coders agree the BCT is absent). Trainees were randomly allocated into coding pairs across tutorial groups using a random number generator. When both trainees identified the BCT as present or absent, agreement was recorded and when one trainee identified the BCT but the other did not identify the BCT, disagreement was recorded. The PABAK score was calculated for each trainee pairing and means reported across pairs and for each of the BCTs assessed.

Trainee agreement with expert consensus was assessed by calculating the number of BCTs identified by each trainee that were also identified by expert consensus and by using PABAK. Each trainee was paired with the expert consensus. The number of agreements and disagreements between trainees and expert consensus were recorded and were used to calculate the PABAK score. Means were reported across trainee–expert consensus pairings and for each of the BCTs assessed. When a trainee identified the BCT identified as present by expert consensus, agreement was recorded and when the trainee did not identify the BCT, or identified a BCT not included in the consensus, disagreement was recorded.

Confidence for BCTs identified was assessed by calculating the frequency and percentage of high confidence ratings (i.e. ‘++’: BCT present beyond all reasonable doubt and clear evidence available) for BCTs identified as present. High confidence ratings were included in the analysis so that we could easily distinguish BCTs identified with certainty.

Change following training: paired sample t-tests were used to assess change in (1) agreement between trainees about BCTs identified (intercoder PABAK), calculated across trainee pairings and across BCTs; (2) the number of BCTs identified by trainees also agreed on as present by expert consensus; (3) trainee agreement with expert consensus (trainee consensus PABAK) across trainee consensus pairings (i.e. each trainee paired with the expert consensus) and across the BCTs assessed; (4) high confidence ratings (i.e. ‘++’) for BCTs identified as present; (5) perceived confidence; and (6) reported intention to use BCT taxonomies in the future.

Previous experience: frequencies and percentages were calculated to describe the number and proportion of trainees with previous experience of BCT taxonomy use. Means and SDs were calculated to describe trainees’ expertise associated with BCIs (separate means were calculated for each of designing, delivering, reporting, reviewing experience and for use of behaviour change theories; an overall mean was calculated across these categories).

Evaluation of training: trainee ratings of the content of the training and ratings of the materials used were summarised using means and SDs. A content analysis of the written feedback was conducted to identify training components that participants reported to work well or not well. Given the novelty of this type and format of coder training, we decided that content analysis provided an appropriate first step for investigating patterns within coders’ initial feedback. CW conducted the first round of analysis, allocating coder feedback into categories based on similarity of content. KS (see Acknowledgments) then checked allocation of coder feedback to content categories. Any discrepancies were resolved through discussion between the two researchers.

Proportion of trainees reaching an acceptable standard of competence following training: an acceptable standard of competence was defined in terms of trainee agreement with expert consensus. Landis and Koch102 suggested that kappa values of 0.60–0.79 indicate ‘substantial’ reliability and those > 0.80 are ‘outstanding’. Thus, we used this as a guide and considered trainees achieving a PABAK score of at least 0.60 to have reached an acceptable standard of competence. To assess change in competence from before to after training, a one-way analysis of variance (ANOVA) was carried out. To evaluate the two training methods in increasing competence, a 2 (before vs. after training) × 2 (workshop vs. distance tutorial) ANOVA was carried out.

Results

Evaluating effect of training in increasing trainee competence

One-day workshops (Tables 5 and 8)

Intercoder agreement between trainees: there was a trend for intercoder agreement to increase across trainees, t(54) = 1.77; p = 0.08 (before training: mean PABAK = 0.39, SD = 0.34); after training: mean PABAK = 0.50, SD = 0.26). Intercoder agreement increased for 6 out of the 12 BCTs assessed: self-monitoring of behaviour, feedback on behaviour, behavioural practice/rehearsal, non-specific reward, credible source and demonstration of the behaviour (average increase in PABAK for these six BCTs = 0.26, SD = 0.18; range = 0.03–0.47). However, change across the 12 BCTs was non-significant, t(11) = 0.90; p = 0.39. The BCT ‘material reward’ achieved good reliability before training and this was also achieved after training.

Agreement with expert consensus: overall, trainee agreement with expert consensus increased, t(108) = 3.26; p < 0.0001 (before training: mean PABAK = 0.39, SD = 0.29; after training: mean PABAK = 0.50, SD = 0.28). Across the BCTs assessed, agreement with expert consensus increased for 8 out of the 12 BCTs: self-monitoring of behaviour, feedback on behaviour, behavioural practice/rehearsal, non-specific reward, goal-setting (outcome), material reward (behaviour), credible source and demonstration of the behaviour (average increase in PABAK for these eight BCTs = 0.36, SD = 0.33; range = 0.02–0.95). However, change across the 12 BCTs was non-significant, t(11) = 0.56; p = 0.59.

Confidence for BCTs identified: the number of high confidence ratings (i.e. ‘++’) that trainees assigned increased, t(108) = 4.89; p < 0.001 (before training: mean number of BCTs = 8.38, SD = 1.91; after training: mean number of BCTs = 9.56, SD = 1.93). Across BCTs assessed, the number of high confidence ratings also increased, t(11) = 2.89; p < 0.05. The number of high confidence ratings increased for 6 out of the 12 BCTs: self-monitoring of behaviour, feedback on behaviour, behavioural practice/rehearsal, material reward, credible source and demonstration of the behaviour. Before training, one of the BCTs, ‘information about health consequences’ was rated with high confidence by over 90% of trainees. After training, four of the BCTs – ‘demonstration of the behaviour’, ‘material reward (behaviour)’, ‘behavioural practice/rehearsal’ and ‘self-monitoring of behaviour’ – were rated with high confidence by 90% or more of trainees.

Distance group tutorials (Tables 5 and 9)

Previous experience: 54% (n = 28) of trainees had previously used a BCT taxonomy to code BCIs and 69% (n = 36) to describe BCIs. The majority of tutorial trainees had a moderate level of expertise in BCIs (mean = 3.50, SD = 0.71).

Intercoder agreement between trainees: intercoder agreement across trainees did not change, t(25) =0 .57; p = 0.57. Across the BCTs assessed, intercoder agreement increased for 8 out of the 17 BCTs: feedback on behaviour, credible source, demonstration of the behaviour, goal-setting (outcome), behavioural practice/rehearsal, self-monitoring of behaviour, instruction on how to perform behaviour and material reward (behaviour) (average increase in PABAK for these eight BCTs = 0.38, SD = 0.26; range = 0.08–0.69). However, change across the 17 BCTs was non-significant, t(16) = 0.28; p = 0.78. As for workshops, the BCT ‘material reward’ achieved good reliability before training and this was also achieved after training. Two further BCTs – ‘self-monitoring of behaviour’ and ‘self-monitoring of outcome(s) of behaviour’ – achieved good reliability before and after training.

Agreement with expert consensus: trainee agreement with expert consensus increased, t(51) = 6.60; p < 0.001 (before training: mean PABAK = 0.57, SD = 0.11; after training: mean PABAK = 0.72, SD = 0.14). Across the BCTs assessed, trainee agreement with expert consensus also increased, t(16) = 2.35; p < 0.05. Agreement with expert consensus increased for 11 of the 17 BCTs: credible source, information about social and environmental consequences, social support (unspecified), monitoring of outcome(s) of behaviour without feedback, demonstration of the behaviour, adding objects to the environment, goal-setting (outcome), behavioural practice/rehearsal, self-monitoring of behaviour, instruction on how to perform the behaviour and material reward (behaviour) (average increase in PABAK for these 11 BCTs = 0.62, SD = 0.50; range = 0.04–1.27).

Confidence for BCTs identified: the number of high confidence ratings (i.e. ‘++’) that trainees assigned did not change, t(51) = –0.57; p = 0.57. Across the 17 BCTs assessed, the number of high confidence ratings increased, t(16) = –3.40; p < 0.001. Ratings increased for 13 out of the 17 BCTs: credible source, information about social and environmental consequences, social support (unspecified), monitoring of outcome(s) of behaviour without feedback, non-specific reward, demonstration of the behaviour, adding objects to the environment, goal-setting (behaviour), self-monitoring of outcome(s) of the behaviour, goal-setting (outcome), behavioural practice/rehearsal, self-monitoring of behaviour and instruction on how to perform the behaviour. Before training, one of the BCTs – ‘self-monitoring of behaviour’ – was rated with high confidence by over 90% of trainees. After training, four BCTs were rated with high confidence by over 90% of trainees: self-monitoring of outcome(s) of behaviour, self-monitoring of behaviour, goal-setting (outcome) and credible source.

There was an increase in self-reported confidence in using the taxonomy from before training: mean = 3.42, SD = 1.00 to after training: mean = 4.08, SD = 0.56, t(51) = –5.27; p < 0.001. Reported intention to use BCT taxonomies in the future remained high from before training: mean = 4.23, SD = 0.74 to after training: mean = 4.16, SD = 1.02; t(51) = 0.44; p = 0.66.

Proportion of trainees reaching acceptable standard of competence

Before training, 22% of workshop trainees and 35% of tutorial trainees achieved a PABAK score (in terms of agreement with expert consensus) of ≥ 0.60 (workshops: mean PABAK = 0.39, SD = 0.29; tutorials: mean PABAK = 0.57, SD = 0.11). After training, the proportion of trainees reaching this threshold increased across both methods: 46% of workshop trainees and 78% of tutorial trainees achieved a PABAK score of ≥ 0.60 therefore meeting the acceptable standard of competence (workshops: mean PABAK = 0.50, SD = 0.28; tutorials: mean PABAK = 0.72, SD = 0.14). Change from before to after training competence was significant for workshops, F(1,216) = 9.66; p < 0.05, and for tutorials, F(1,102) = 38.38; p < 0.001. Training methods were equally effective at increasing competence, F(1,318) = 0.35; p = 0.55.

Evaluating coder experience

Training was evaluated positively by trainees with all components receiving high ratings (on the scale of 1–5) in terms of usefulness (workshops: mean = 4.62, SD = 0.68; range = 3–5; tutorials: mean = 4.30, SD = 0.67; range 4–5). Trainees rated activities that used the taxonomy to code short excerpts and longer descriptions from published reports as being useful opportunities to apply their newly learned skills. In particular, trainees valued the combination of ‘working through practical examples’ alongside the ‘opportunity to clarify and discuss’ any coding they found unclear. A few trainees commented on the ‘Ready, steady, point!’ exercises as being a useful method to increase their ability to identify BCTs at speed.

Tutorial trainees reported that the combination of practical tasks and the opportunity for structured discussion during the tutorial sessions was particularly useful. Many of the trainees commented that the tutorials provided a useful opportunity to learn ‘the consensus answers and the rationale behind coding’ for specific BCTs and then ‘discuss any reasons for discrepancies with other members of the group’. The majority commented that having access to a wide range and number of excerpts taken from published reports provided a useful opportunity to practice applying the taxonomy. Although the majority agreed that the ‘Learning Log’ and ‘Buddy system’ components were both useful in principle, feedback suggested that trainees generally wanted more guidance on how to use them. Trainees commented that they occasionally found them difficult to put into practice owing to time constraints. This was particularly an issue for trainees from different time zones.

Discussion

Training in using BCTTv1 in the form of 1-day workshops or group tutorials held over 6 weeks increased average trainee agreement with expert consensus and doubled the proportion achieving the competence threshold to 46% and 78% respectively. Training also increased workshop trainees’ confidence in identifying BCTs assessed in descriptions of BCIs, but did not improve intercoder agreement for either method. The opportunity to apply new knowledge and skills in a number of coding tasks followed by group discussion was evaluated as being a useful approach by trainees.

The difference between training methods in the proportion of those achieving an acceptable standard of coding competence may be partly due to differences in the percentage showing competence before training. Workshops required less commitment and may have attracted participants with less involvement and less competence in BCT coding. The fact that tutorial training allows consolidation of learning and practice between sessions may account for the higher proportion of competent coders following training. This is the model that has been taken forward in developing an e-learning programme of training (see Chapter 7 and BCTTv1. Online Training103).

We identified BCTs for which training was effective, that is when agreement and intercoder agreement were poor before training but good after training; for example, behavioural practice/rehearsal. There was only one frequently used BCT that achieved good reliability without training for workshops and tutorials: material reward (behaviour). Two other BCTs achieved good reliability without training for tutorial trainees: self-monitoring of behaviour and self-monitoring of outcome(s) of behaviour. Overall, training achieved good validity and reliability for between one-quarter and one-third of the BCTs.

Some BCTs consistently achieved poor reliability before and/or after training, for example goal-setting (outcome), social support (unspecified), and information about social and environmental consequences. These BCTs may require further definition and/or refinement of labels and examples for trained coders to be able to identify them with high reliability and validity. For example, drawing on feedback from trainees, refinements could include further clarification of the distinction between BCTs with similar labels that also appear in the same taxonomy grouping [e.g. social support (unspecified) and social support (practical)] or addition of more examples to help coders distinguish between different ‘types’ of BCTs [i.e. ‘behaviour’ vs. ‘outcome’ BCTs such as goal-setting (behaviour) and goal-setting (outcome)]. It is therefore likely that the number of BCTs that can be effectively trained will increase as BCTTv1 develops.

Training increased confidence in identifying BCTs assessed for both training methods and significantly so among workshop trainees. Tutorial trainees also reported increased overall confidence in using the taxonomy. This may be due to the learning environment provided by tutorials which includes a support network built over the period of multiple and regular sessions, and more time for reflection and practice between sessions. In the current study, we did not compare differences in the efficacy of training across the tutorial groups. While guided by a training manual, it is feasible that the engagement style and delivery of support, guidance and feedback differed across expert tutors despite having a similar level of experience to one another. Confidence in applying BCTTv1 requires learning the complexities and challenges of applying it, which may decrease or increase confidence in the short term. Therefore, future research should examine how this changes with training, further experience and across different modes of training delivery.

Although a direct comparison of the two BCTTv1 training methods was not the focus of study 3, some reflections are possible about the comparative merits, acceptability and feasibility of each approach. Workshops may be considered the more cost-effective and time-effective option given that a relatively large group of coders can be trained over the course of a day. They may also be more accessible to participants who simply wish to learn about the approach but with less commitment to becoming expert BCT coders.

Compared with workshop trainees, tutorial trainees started with a greater level of experience and expertise in using BCT taxonomies. It may be that training involves a process of trainees changing their existing interpretations of a BCT as understood from their discipline’s perspective. Greater pre-existing experience may have meant that ‘unlearning’ needed to take place before competence could be reached. This could help to explain why some BCTs achieved poor reliability and validity before and/or after training. Small group tutorial training delivered over a longer period of time provides a learning environment which includes a support network built over the period of multiple and regular sessions and more time for reflection and practice between sessions. Having access to an interactive training resource that provides continued support and feedback beyond that of time spent with the tutorial group would provide the additional space and time that this level of learning requires. Such a resource has been developed and is described in more detail in Chapter 7.

It is important to remember that effect of training was evaluated on trainee coding of just one intervention description before and after training for each of the methods. The training materials used in 1-day workshops and tutorials were written to exemplify particular BCTs, to highlight the learning principles and to ensure inclusion of the frequent BCTs targeted for training. However, the quality of published intervention descriptions is generally poor,95 with techniques described using different terminology and referred to using different labelling.104,105 As the use of BCT methodology increases, BCT content as reported in intervention descriptions is likely to become much clearer, leading to increased ease of identifying BCTs.95 It is also important to acknowledge that, despite attempts to ensure diversity of trainees, our sample were predominantly female and from the UK. As the use of BCT methodology becomes more widely used, it will be possible to recruit a broader range of trainees for training evaluation.

For users to maintain good levels of accuracy and reliability after training, efforts should be made to maintain knowledge and skills over time.106 Therefore, we recommend that coders already trained in use of BCTTv1 regularly review training materials and check their reliability and accuracy.

Copyright © Queen’s Printer and Controller of HMSO 2015. This work was produced by Michie et al. under the terms of a commissioning contract issued by the Secretary of State for Health. This issue may be freely reproduced for the purposes of private research and study and extracts (or indeed, the full report) may be included in professional journals provided that suitable acknowledgement is made and the reproduction is not associated with any form of advertising. Applications for commercial reproduction should be addressed to: NIHR Journals Library, National Institute for Health Research, Evaluation, Trials and Studies Coordinating Centre, Alpha House, University of Southampton Science Park, Southampton SO16 7NS, UK.

Included under terms of UK Non-commercial Government License.

Bookshelf ID: NBK327613

Views

  • PubReader
  • Print View
  • Cite this Page
  • PDF version of this title (1.7M)

Other titles in this collection

Recent Activity

  • Training to code intervention descriptions using Behaviour Change Technique Taxo...
    Training to code intervention descriptions using Behaviour Change Technique Taxonomy version 1 (study 3) - Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data)

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...