I've experienced this annoying behaviour a couple of times in the past. There is some workaround to avoid rebooting but my memory fails me on the exact steps. I think it involves switching the phone to either silent or offline and then back again.

Strangely this bug only seems to occur if I hit the button the moment it appears and before the alarm sounds.