Once host mode is working properly, yes, USB data modems of all varieties should work fine. But madwifi would have nothing to do with this -- USB data modems show up as a tty which is talked to with a modified Hayes command set.

The use of madwifi (or other wifi drivers) would be with a USB wifi stick; among benefits over the N900's wifi are 802.11a/n 5GHz band, and packet injection for aircrack-in-a-pocket. This (aircrack stuff) was done by some people on the N8x0 after OTG arrived, search about for exact details...