Ideally, when uninstalling a program, any unused libs will be removed with it. The only thing that should stay are folders left in your user directory. Anything outside of your user folder should be handled by your package manager (try apt-get autoremove?)