Never happened to me, but did it happen before you installed the OGG GStreamer plugin? If not, chances are the problem (possible transcoding) is being caused by the plugin. However if transcoding were taking place I'd expect your CPU usage to go through the roof.