I've never had that issue. Did you install through the application installer?

To answer your second question, there is actually no icon, but if you want to remove everything, just run the following:

rm /usr/bin/ID3.py
rm /usr/bin/id3reader.py
rm /usr/share/applications/hildon/MediaCenter.desktop
rm /usr/share/dbus-1/services/MediaCenter.service
rm /usr/share/pixmaps/bg.png
rm /usr/share/pixmaps/song.png
rm /usr/share/pixmaps/test.png
rm /usr/share/pixmaps/UNKNOWN_UNKNOWN.jpg

That's all in the prerm script, but if your application install malfunctioned somehow, that may be corrupt somehow.

You might also just try to re-install a later version over it and then have a look if you can see that in the application installer.