Granted, ssh is very useful, I use it a lot. But I only have it enabled when and where I really need it. As a matter of fact you are not really safe. Just consider TLS. The foundation of almost all web security. That was considered safe until the renogatiation weakness was discovered not many months ago.