The virtualbox .dmg includes a 'VirtualBox Uninstall.tool' script (double-click it and it will run automatically in your terminal if you want to do it using the GUI). Guides you through a perfect uninstall...
Virtualbox IS the way to go, it doesn't look like much (in my opinion) but it works like a champ!