That's because they're only roster contacts, not "proper" contacts. That is, they'll go away when you're not online to the IM account which provides them.