I must be the only one that doesn't use CSSU? lol
Saying that my alarm has always gone off

I'm not saying its CSSU!!!

If you use the recording app on calls, it does slow it down!
Disable it or play with its settings.
What apps do you run/have installed?

Sounds like it could be a swap or cache issue filling.