Thanks to Venemo's testing we found out why desktop panning wasn't working for me. It works fine with clutter 1.4 or less but is broken with 1.5.x which I was using.
Apparently something had changed in clutter 1.5, which breakes hildon-desktop rendering. So for now we will require clutter < 1.5.

It is so good to finally see desktop scrolling smoothly under Xephyr again.