mplayer is used actually, but embeds its output in MediaBox. This does not work on 770. It's possible to have mplayer open a separate window, which would work, but toggling between MediaBox and mplayer windows would be awkward. UKMP e.g. did it like this while it still supported video.
On the other hand, Canola1 (unfortunately closed source and a dead version) embedded the video output of osso-media-server quite well on the 770. So there must be a way.