What about the existing Ogg Vorbis Player
Or what about this quote from Visa Smolander with regard to Media Streamer developments in the garage section?
Apologies if this information is of little additional use and/or is un-related to your overall question.
Make sure that you use fixed point vorbis decoder implementation (tremor), also compile it with proper options (it has some arm assembly in it) and use good gcc optimization settings includion -mcpu. Unless gstreamer screws everything up and adds some unexpected overhead, it should work quite fast and provide reasonable performance.