Previous studies have shown that task-irrelevant threatening faces (e.g., fearful faces) are difficult to filter from visual working memory (VWM; Stout et al., 2013). What is not known, however, is whether non-threatening negative faces (e.g., sad faces) are also difficult to filter and whether depressive symptoms affect filtering ability. We used a color-change detection task to test whether task-irrelevant sad and fearful face distractors could be filtered by healthy participants and by depressed participants. The groups differed in their filtering ability, as indicated by the contralateral delay activity, a specific ERP index for the number of objects stored in the VWM during the maintenance phase. The healthy group stored the same amount of VWM information under the non-distractor and the sad face distractor conditions, but more information was stored under the fearful face distractor condition than under the other conditions (non-distractor condition and sad face distractor condition), suggesting that specifically threatening faces are difficult to filter from VWM in healthy individuals. By contrast, depressed participants stored the same amount of VWM information under the non-distractor condition, fearful face distractor condition, and sad face distractor condition, suggesting no extra consumption of VWM resources for both fearful and sad face distractors. That is, a greater number of depressive symptoms seems to enhance the filtering ability of irrelevant sad and fearful face distractors from VWM. Our results for healthy participants confirm the previous findings of a threat-related filtering difficulty in average individuals, and additional findings from the sad faces suggest that non-threatening negative faces do not unnecessarily load the VWM. The unexpected finding of efficient filtering of fearful faces in participants with depressive symptoms requires further studies with clinically depressed participants.