After looking with attention I discovered that N900 uses TWL4030 chip driver but not ISP1301. Sorry for confusion.

And driver for TWL4030 doesn't have any attempt to enforce HOST/OTG mode actually. It just prepare a software support for peripheral devices in case of OTG/HOST setup. It is a reason why writing word 'host' to /sys/.../mode file changes nothing.

EDIT: Actually, the configuration option CONFIG_USB_MUSB_HOST has no code in TWL4030. The driver is clearly designed for OTG-via-cable host-mode usage.