Install Arch or Debian, cross-compile the 2.6.37 kernel for ARMv7 softfp, grab the GLES drivers and use your favorite DE of choice.

And make sure to install U-boot with that kernel.

Your device won't be as useful as a phone (Text messaging kinda works, if you can find a suitable client) and battery life will suffer slightly (IIRC some PM stuff on Nemo isn't as worked on as Fremantle's), but it will be quite open (with the exception of the ImgTec SGX drivers) and as snappy, if not snappier than Maemo.

(P.S.: Fremantle Hildon-Desktop works quite crappily when built and used on its' own. Hopefully some people can find time and make it polished as it is on Fremantle.)