There is a "Remove Your Thanks" link under messages where someone was offered their Thanks. Maybe you rubbed someone the wrong way.

Also, interesting fact that you follow your Thanks count that closely.