I wasted a lot of time with WMware too. Tried 3 images : the default one (browser console), Damn Small Linux (DSL), and the large Ubuntu 5.10 mentioned above. I could get neither to connect the 770 as a USB device (although they did see my USB keyborad, mouse, external disk and printer :-). VMware would be extremely convenient for Windows users if it worked consistently, but right now it seems very iffy and dependent on hardware and setup : for a few it works, for many it doesn't.
In the end I gave up and booted up my trusty old Knoppix (the mother of all Linux Live CDs !) and it was all over in a matter of minutes (enable RD, modify gainroot, disable RD)...