This issue has to be solved in osso-xterm I believe. It should have a less sensitive way to pop up the keyboard indeed.

Perhaps when mouse reporting is enabled, terminal should not pop up the keyboard or something like this.