0. Turn off FM Transmitter, "killall fmtxd", something like that.

1. Backup
/usr/sbin/fmtxd

2. Replace
/usr/sbin/fmtxd
with
fmtxd1.3
from
http://jacekowski.org/Maemo/FMTXD1.3

Now you have file named "/usr/sbin/fmtxd" but originating from jacekowski instead of Nokia.

3. In X Terminal (as root?)
chmod +x /usr/sbin/fmtxd
so that this file could be executed

4. Start Media Player, some song in it, and in menu choose FM Transmitter.
First, try transmitting on normal frequencies.
If it doesn't work, reboot.
If it works, try going to lower frequencies.

5. Probably, you will need to install FM Faker application after that.

EDIT:
0. Install rootsh package through standard Hildon App Manager.
1. Download fmtxd1.3 file from webpage
http://jacekowski.org/Maemo/FMTXD1.3

and place it into MyDocs folder
("Nokia N900" in File Manager,
"/home/user/MyDocs" in X Terminal,
"Nokia N900 (?" drive in Windows when Nokia N900 is attached to PC in Mass Storage Mode)

3. Disconnect the device from PC, if it was connected in Mass Storage Mode; and turn off FM Transmitter, if it was turned on

4. Open X Terminal

5. commands:

sudo gainroot

killall fmtxd

mv /usr/sbin/fmtxd /usr/sbin/fmtxd.old

mv /home/user/MyDocs/fmtxd1.3 /usr/sbin/fmtxd

chmod +x /usr/sbin/fmtxd

6. Try to use FM Transmitter.

7. If it doesn't work, reboot.

8. If it doesn't work, install FM Faker.