I can't figure out for the life of me why Nokia used such crappy glue behind that magnet.

Even if your magnet is still in place, it's probably a good idea to pull it off, clean out the rubber cement that Nokia used and replace it with some super glue.