I haven't noticed this (but I disabled the ssh server so I always open a login shell to start ssh before using it anyways)...