Performance in delayed-response working memory (WM) tasks is typically associated with sustained activation in the dorsolateral prefrontal cortex (dlPFC) that spans the delay between the memoranda and the memory probe. Recent studies have demonstrated that novel distracters presented during the delay interval both affect sustained activation and impair WM performance. However, the effect of the performance-impairing distracters upon sustained dlPFC delay activity was related to the characteristics of the distracters: memoranda-confusable distracters increased delay activity, whereas memoranda-nonconfusable emotional distracters decreased delay activity. Because these different effects were observed in different studies, it is possible that different dlPFC regions were involved and the paradox is more apparent than real. To investigate this possibility, event-related fMRI data were recorded while subjects performed a WM task for faces with memoranda-confusable (novel faces) and memoranda-nonconfusable emotional (novel scenes) distracters presented during the delay interval. Consistent with previous findings, confusable face distracters increased dlPFC delay activity, while nonconfusable emotional distracters decreased dlPFC delay activity, and these opposing effects modulated activity in the same dlPFC regions. These results provide direct evidence that specific regions of the dlPFC are generally involved in mediating the effects of distraction, while showing sensitivity to the nature of distraction. These findings are relevant for understanding alterations in the neural mechanisms associated with both general impairment of cognitive control and with specific impairment in the ability to control emotional distraction, such as those observed in aging and affective disorders, respectively.