I've only done it in a 2 step process.
First converting it into MP4 AAC in handbrake.
Then converting that movie with the media converter into a AVI format that the Nokia 770 could handle.

It takes time, but I could not get handbrake to make a file that played directly on the 770.