Intermittent explosive disorder (IED), as described in DSM-5, is the categorical expression of pathological impulsive aggression. Previous work has identified neurobiological correlates of the disorder in patterns of frontal-limbic brain activity and dysregulation of serotonergic neurotransmission. Given the importance of short- and-long range white matter connections of the brain in social and emotional behavior, studies of white matter connectivity in impulsive aggression are warranted. Diffusion tensor imaging (DTI) studies in the related conditions of antisocial and borderline personality disorder have produced preliminary evidence of disturbed white matter connectivity in these disorders, but to date there have been no DTI studies in IED. A total of 132 male and female adults between the ages of 18 and 55 years underwent Turboprop-DTI on a 3-Tesla MRI scanner. Of these, 42 subjects had IED, 40 were normal controls, and 50 were clinical psychiatric controls with psychiatric disorders without IED. All subjects were free of alcohol, psychotropic medications, or drugs of abuse. The diffusion tensor was calculated in each voxel and maps of fractional anisotropy (FA) were generated. Tract-based spatial statistics (TBSS) were used to compare FA along the white matter skeleton among the three subject groups. IED was associated with lower FA in two clusters located in the superior longitudinal fasciculus (SLF) when compared with the psychiatric and healthy controls. Impulsive aggression and borderline personality disorder, but not psychopathy or antisocial personality disorder, was associated with lower FA in the two clusters within the SLF. In conclusion, IED was associated with lower white matter integrity in long-range connections between the frontal and temporoparietal regions.