This is a feature not a bug: this way, when you decide to install again that particular application, you'll find it already configured the way it was.
There are ways to uninstall both the application and its configuration; they need the use of command line and superuser mode - apt-get purge is the command.