I haven't delved much into Qt yet but I wonder whether the GL for Qt is full OpenGL, not GLES? Therefore, my suggestion is that the speed reduction occurs in changing from full to ES, even if the instructions/features you are using are within the ES spec, because QML includes the full instruction set [1] but the software has to adapt to the operating environment (N900).

Is there anyway you can use C/C++ & OpenGL ES for the graphics and provide a few hooks into the UI? I think there is a QtOpenGL, is there a QtOpenGL ES you could try?

I think that makes sense!!
[1] instruction set: these are only my thoughts, not definitive.

I hope you can find a work around to get smooth, shiny graphics!