Same here on my notebook so I use the NUMLK key. It's a pain so I also tried a numeric kb attached via USB but the stupid thing simply mirrors the notebooks keys, ie, I still have to use NUMLK :-( Saves one single key press.

PS: Remember x11vnc needs restarting between sessions. Don't know if this behaviour is configurable but it has caught me out a couple of times.