There are hundreds of reasons to want to be able to SSH into your desktop - it's a fantastically useful thing to be able to do. If you set it up with only one username allowed to log in, and only with a key, you're pretty secure from would-be bad guys.

If you can do it this way, you should - it's great.