when you press mouse button down, the browser has to decide whether you hit a link or not.
If you hit the link it gets highlighted and follows that path, if not it enters scrollmode (because you hit the element behind)

if the ui keeps jumping and dancing around because it cannot decide whether you are near a link or wanting to scroll we are in bigger trouble.

On an unscrollable keyboard its less of a problem, but because the ui interaction on a webpage is already multifunction it will cause complications.

Of course this wont stop us from trying and seeing and it might actually work really well, i'm merely playing devils advocate.