avi is MP4, but you'll need it have the DIVX fourcc. ffmpeg defaults seem to work well, or at least mplayers default lavcopts for vcodec=mpeg4, so long as your resolution and bit rate are low enough for the n770 to decode it.

i don't have a mac so I cant test with your tool, but this mplayer command will work:
dvd://1 is for the first track, dvd://5 would do the 5th track. there are also mplayer options to choose specific chapters and timestamps, see the mplayer docs. use scale=240:144 for widescreen and you might have to play with the bitrate. you can try a bit lower bitrate and change -ofps to 23.976 or 29.970 if it seems jumpy. keep in mind that is your fps and bitrate are too high, the 700's player will drop frames anyway and make it jumpy in a much less desirable way than 15 fps will.

and before someone jumps up and down about copying movies, this will not work on a commercial dvd without some extra effort, which i will not go into here. this command is for encoding your unecrypted dvd's for playback on the 770.