Swap, probably. All the code that handles the notification routines probably get pushed out to swap.

Here's an easy way to tell. When you get multiple SMSes within a short period of each other, is the first one slow and the rest fast?