This was frustrating. Half the time it wouldn't go into USB host mode and when it did, it'd freeze at 10%. I removed the RSMMC card on a hunch and lo and behold the update worked. It's booting now.

Mark S: You need to download the windows flasher. There should be a link to it from the wiki here or on www.maemo.org (Sorry I'm pressed for time and can't find a link). Run the windows flasher and eventually it'll tell you to prepare the 770. Hold down the home button and turn your 770 on and you should see a little USB logo within 5 seconds. Pick the image you want to use (the 58MB file you downloaded (link in the first post here by aflegg)) and it'll reflash.

You can't do it with a standard USB connection because the OS is already booted... kind of like trying to reinstall windows when you're already in it.