no, it isn't just the initial connection. i actually tried this on a "spare" n800 that i had (it had a busted screen, but magically fixed itself). this one i haven't touched in a while, and used to be able to ssh without a problem. but now it has a slow connection as well. which tells me it's a problem on the client side. both computers that i'm using are running ubuntu karmic. i can ssh between them without a problem...
strange. this is really annoying.