Evaluating helicopter emergency medical missions: a reliability study of the HEMS benefit and NACA scores

Background: The benefits of the Helicopter Emergency Medical Service (HEMS) and dispatch accuracy are continuously debated, and a widely accepted score to measure the benefits of the mission is lacking. The HEMS Benefit Score (HBS) has been used in Finnish helicopter emergency medical services, but studies are lacking. The National Advisory Committee for Aeronautics (NACA) score is widely used to measure the severity of illness or injury in the pre-hospital setting, but it has many critics due to its subjectivity. We investigated the inter-rater and rater-against-reference reliability of these scores.

Methods: Twenty-five fictional HEMS missions were created by an expert panel. A total of 22 pre-hospital physicians were recruited to participate in the study from two different HEMS bases. The participants received written instructions on the use of the scores. Intraclass correlation coefficients (ICCs) and mean differences between rater-against-reference values were calculated.

Results: A total of 17 physicians participated in the study. The ICC was 0.70 (95% CI 0.57-0.83) for the HBS and 0.65 (95% CI 0.51-0.79) for the NACA score. Mean differences between references and raters were -0.09 (SD 0.72) for the HBS and 0.28 (SD 0.61) for the NACA score, indicating that raters scored some lower NACA values than reference values formed by an expert panel.

Conclusion: The HBS and NACA score had substantial inter-rater reliability. In addition, the rater-against-reference values were acceptable, though large differences were observed between individual raters and references in some clinical cases.