Found the problem! Since this notification type hijacks the email stuff, you have to have "E-mail message received" under "Notification light" in the settings.