Is it possible to de-associate a contact from its facebook data?

Because I associated a contact by mistake, now every time I hit retrieve, I have to manually remove the added data on that contact.

:/