The resolution is a variable of the dosbox.conf file. It's one of the very first line items, I think. Try tinkering with different values and see if you can get the displayed image cleared up enough that your pointing device works correctly.

Remember not to scale to the full 800 pixel wide length of the N900's screen since it's a wide screen display and most dos games were designed for a 4:3 square display. If you tell DosBox to display too wide it will either cut off part of the image, or distort your image. Both can throw off the calibration of your pointing device in game.

Also remember that performance will be slow. While not as slow as straight up emulation (DosBox is NOT an emulator per say) DosBox has not been optimized for the N900's GPU, so the performance still leaves something to be desired when running software that's newer then 1993....