The problem with removing the meta-package is that many packages are bound by that one, and without it become 'orphans'.
This means that next time you install some package the auto-removal function that in apt-get is executed after the installation is completed will wipe them out.