Ya, you'll have to make it in the kernel ARM scratchbox...

Also, probably better to use linux-omap 2.6.32 as a base instead of vanilla 2.6.32. Even that is still missing drivers.

I have a bit of info here:
http://wiki.maemo.org/User:Jebba

Good luck,

-Jeff