That's the problem with AAC audio decoding. MPlayer has a proper decoder for AAC which uses floating point math and also some limited fixed point implementation. Floating point math is bad for ARM devices which do not have floating point unit (including Nokia 770), so MPlayer is configured to always use fixed point decoder on ARM. This fixed point decoder is faster, but it does not support all AAC variants ("Main Profile" and "Sample-Rate Scalable").

Fortunately N800 has floating point unit and can use floating point decoder (which is still slower than fixed point decoder, but at least supports all types of AAC audio). I'll enable floating point AAC decoder for N800 in the next build of MPlayer.

Actually both Nokia 770 and N800 have DSP codec for AAC, but using it in MPlayer would require quite a lot of work.

PS. I could have easily missed your post, better ask questions in mplayer thread or even better - submit a bugreport at garage project page.