How are you touching/holding the phone as you pull it out?

I had some issues, but found if I extract the phone from its cover by gently pulling on the phone charm, I don't touch anything and accidentally give the phone a wrong instruction. If you touch the screen or any of the hardware buttons, I think you risk telling it something. There's also the issue of it heading to silent/reject if you make it think you turned it face down.