The role of environmental adversity in the development of high callous-unemotional (CU) traits in children is controversial. Evidence speaks to the traits being largely independent of adversity; however, recent data shows that those with high CU traits and high adversity and/or high anxiety might differ in important ways from those with no such history. We tested this using emotion recognition (ER) skills. We tested whether maltreatment history and anxiety levels moderated the relationship between level of CU traits and ER skills in N = 364 children with behavioral problems who were 3 to 16 years old. As hypothesised, in the full sample, the relationship between CU traits and ER differed according to maltreatment history, such that CU traits were associated with poorer recognition for those with zero or negligible history of maltreatment. This moderation of the CU-ER relationship by maltreatment was inconsistent across subgroups, however, and for the cohort utilizing youth self-report of maltreatment, high CU traits were associated with poor ER in those with lower anxiety levels. Maltreatment history and/or anxiety levels can identify different emotional impairments associated with high CU traits, and the impairments might be characteristic of "primary" high CU traits defined as occurring independently of maltreatment and/or high anxiety.