Magnet is on the pouch indeed, but because of the way I put the phone in the pouch, the magnet is on the other side of the phone's sensor, at the center.

@Abukana I second Blaizzen's observation. There're already cases regarding the magnet in the phone pouch cause problems to the proximity sensor.

The business card you inserted probably just helped the situation a bit, but you'd like to look for some other pouch without magnet.

@MohammadAG you're like a genie that granting people wishes around. Thanks.

Thanks everyone for the help!