After having had a quick look on the Wikipedia page for Vorbis, I'd say that something like Tremor (fixed point arithmetic implementation) mixed with Lancer (platform specific SIMD instruction implementation) would be the best bet.

That'd make it, I'd guess, the easiest to get something that could use the DSP to decode the audio, moving computation away from the CPU. This would give more decoded data per cycle spent and thus giving better battery time.

There might already be such an implementation out there, since the OMAP3 is used by so many, but I haven't taken the time to look for it.

If the oggvorbis library that "Extra decoders" supplies is the reference implementation from Xiph.org, it will be all CPU bound (as far as I can tell) and it will make your CPU very busy, resulting in the drain you've observed.