That sounds useful, but what Cas07 and I were getting at was that the problem is the modality - or more specifically, the time it takes to switch modes. The screen is pretty small to write on, so I personally (and probably others) end up zooming in a bit. If I have to switch modes often (more than once per line, in this case), and pan, it significantly lowers its utility for note-taking where speed is an issue (e.g. lecture notes, as I mentioned). I agree with Cas07 that this is not an easy problem to solve in a way that will make everyone happy, but from what I've seen, this is a fairly common use-case.
That's fine by me, but what's wrong my previous suggestion about using the arrow keys to scroll by line, rather than pixel? I'm sorry to bring it up again, but I don't think you answered it very clearly. As far as I can see, it wouldn't adversely affect anyone wanting to pan for other reasons, but it would be a quick fix for the problem I'm having. You mentioned that Ctrl+arrow will pan by a larger amount, but it's still essentially an arbitrary amount, in relation to a unit like page lines. If this isn't practical for some reason, I'll apologize for belabouring it, but it seems like it would be a simple and useful solution.

Good idea, but again - the time delay - whether I have to click a button on the other side of the screen, or wait a few hundred milliseconds for a timeout, it doesn't make much of a difference. That said, I like this idea, and it would be useful - just not as a solution to what I'm thinking of.

Sorry to keep bugging you about this. Hopefully we can find something that works for everyone.