For the 770 your illegal instructions are the floating point instructions (which on the soft-float 770 should be compiled as function calls - this is done by default in the toolchain, just remove the "-mfpu=vfp -mfloat-abi=soft-float").

With regard to slow mp3 and ogg playback, are you using Tremor yet for ogg? Tremor (integer only) is faster than vorbis (floating point) even on the floating point-enabled N800, the soft-float 770 will certainly be very very slow having to simulate all those fp instructions. Same goes for mp3, are you using mad (integer only) or did you choose a floating point code?

Simon