I had the same problem and what I did was delete contacts reboot then restore from backup with only "communication and Calendar" selected.. Now all is good.

EDIT: sorry should have been clearer .. My problem was with only one contact also.. i delete that contact and did the steps above.