Technically Hildon *is* a custom UI. It's fairly customisable too, just see what Intel are doing to Hildon home on their MID devices.

Anyway, if you search here you'll find people have had KDE running on top of Maemo on an N800, as well as WindowMaker and OpenBox.

There are also entire OS replacements such as Poky, Angstom and GPE - but these have downsides where binary components are involved.