Make sure that you use fixed point vorbis decoder implementation (tremor), also compile it with proper options (it has some arm assembly in it) and use good gcc optimization settings includion -mcpu. Unless gstreamer screws everything up and adds some unexpected overhead, it should work quite fast and provide reasonable performance.