I'm trying to use my Tytn as a modem and tether it to my N800 over BT but can't get the N800 to recognize the Tytn as a mobile phone. Anyone know why that is?

The Tytn likes to think of itself as a PDA, at least as far as its BT identification goes. I can see that when I scan the environment with my T3 - the Tytn phones show up as PDA.
Not sure how to handle this on the N800 side, unfortunately.

Pair your phone using the Bluetooth app rather than the phone app. Then create a phone profile using the wizard (I'm a bit hazy on where this is). Once that's all done, set your N800 to connect using 'phone' (or 'any'). The N800 will then automatically attempt to find your phone and use it. Your phone will never show up in the connections list, however.

Don't forget to enable bluetooth on your TyTN, too.