Social neuroscience has shed light on the underpinnings of understanding other minds. The current study investigated the effect of self-involvement during social interaction on attention, arousal, and facial expression. Specifically, we sought to disentangle the effect of being personally addressed from the effect of decoding the meaning of another person's facial expression. To this end, eye movements, pupil size, and facial electromyographic (EMG) activity were recorded while participants observed virtual characters gazing at them or looking at someone else. In dynamic animations, the virtual characters then displayed either socially relevant facial expressions (similar to those used in everyday life situations to establish interpersonal contact) or arbitrary facial movements. The results show that attention allocation, as assessed by eye-tracking measurements, was specifically related to self-involvement regardless of the social meaning being conveyed. Arousal, as measured by pupil size, was primarily related to perceiving the virtual character's gender. In contrast, facial EMG activity was determined by the perception of socially relevant facial expressions irrespective of whom these were directed towards.