Words processed with reference to the self are generally better remembered than words processed in semantic terms. An account of this phenomenon, labeled the Self Reference Effect (SRE), is that the self promotes elaboration and organization of encoded information. Although a few neuroimaging studies associated self-referential encoding with activations of the medial prefrontal cortex, no previous study has investigated the neural correlates of remembering emotional words encoded in an SRE paradigm. The main goal of this study was to define with fMRI the neural correlates of the successful retrieval of negative and positive personality traits encoded in a self-referential mode. Functional MRI scans were acquired for 11 subjects as they recognized positive and negative emotional personality traits adjectives encoded in a self-referential condition, a semantic condition and in a phonemic condition. The correct recognition of self-encoded personality traits engaged dorso-medial prefrontal cortex and lateral prefrontal regions, premotor cortex, parietal and occipital cortex, caudate and cerebellum. The specific recognition of self-encoded negative personality traits involved greater neural activation in the right extra-striate region than the recognition of positive personality traits. Our fMRI findings suggest that specific processes may operate at both encoding and retrieval to subserve the SRE. Unlike self-encoding, the retrieval of personality traits is modulated by the valence of the stimuli with greater activation for negative words. Our results indicate that personally relevant words may signal important emotional clues and support the notion of a widely distributed set of brain regions involved in maintaining the concepts of self.