MPlayer packages can be built by running 'make deb-n770' or 'make deb-n800' for convenience. Sources can be obtained by applying patches that are available in garage mplayer downloads area to the corresponding upstream mplayer (full tarball would exceed garage file size limit). Alternatively you can checkout mplayer from garage SVN, it is usually in a good shape and very rarely broken.

As for liba52, I disabled it long ago when experimenting with Nokia 770. Don't remember exactly why (it either had problems when compiled for ARM and crashed or was using floating point math and couldn't decode anything in realtime). Anyway, feel free to experiment with AC3 support and report the results. Patches can be submitted to the mplayer garage tracker.