Module in kernel != automatic integration with Maemo (startup scripts, etc). See wiki for informations about how to change filesystem, and follow it to the letter, using ext4 as and example (and paying attention to mentions about things that are different with reiserfs and ext* filesystems).

I don't want to discourage you or anything, but I think you may be touching things without even basic understanding a) why you may even want them on your device b) how to achieve it, so, just in case, backup your data and be ready for reflash/backup recovery. This way, experimenting won't hurt.

Not to mention, that I have NFC why you want btrfs running on N900's flash storage, but that's completely different matter (and none of the above is really on-topic for kernel-power thread, too).

/Estel