I did some testing with latest mplayer (mplayer_1.0rc1-maemo.10) with default settings (VO nokia770). And i got N800.

Very high quality 1500 kbps 704x384 XviD (audio 150 kbps mp3) seems to lag, but maybe armv6 video decoding can be optimized little bit.

High quality 4:3 512x384 1070 kbps XviD (audio 128 kbps mp3) and 711 kbps 352x288 XviD (96 kbps) seems to work pretty fine. Of course there are some serious tearing problems, but hopefully Nokia will fix that in a future firmware/patch.

Do you have any idea how much mplayer performance can be optimized? Whats the limit of flawlessy playback?