Does unison read ~/.ssh/config file? That is one thing to try.

I personally use unison inside a chroot.

I guess you can also do that using easy-debian.