Hmm, I do not know.
But I guess Maemo's mkfs is not really the latest technology!
So it should be hard to not fulfill uboots needs? But I really do not know, so please confirm reinob's question.

Meanwhile I checked my older (all) configs and I found for nitdroid an entry of
ITEM_FSTYPE=vfat
Maybe you can try it with
ITEM_FSTYPE=ext3
or ITEM_FSTYPE="ext3"
and of course run u-boot-update-bootmenu.
But I do not think that solves your prob but worth a shot, right?