That kinda explains it. It should sort itself out after a while. If it doesn't, perhaps Tracker is choking on some of your media. Of course this shouldn't happen, but that is the most likely explanation. So if it gets too bad, removing the media (and then adding it again gradually, to see if something specific triggers the issue) may be the best thing to try.