http://mplayer.garage.maemo.org/

Two parts are important:
"MPlayer currently provides the best quality of mpeg4/divx/xvid video playback for Nokia 770 / N800"
and
"Some file formats and codecs are not supported or poorly supported, mostly proprietary formats like real media or WMA/WMV"

Unfortunately H.264 and AAC fall into the poorly supported codecs category at the moment because they don't have any special ARM optimizations right now and just use a slow generic code. Adding optimizations for these codecs is possible, but they require a good share of work and time.

OS2008 is the first firmware revision to support H.264, it will need some time for mplayer to accommodate to video priorities change and catch up.