If you run no externally accessible services (or lock them down sufficiently), then console access is required. If it's in your pocket, console access is pretty strictly limited.

(And, with console access and any net connection, an attacker can install gainroot and use it... so it really doesn't matter.)