Could you try running x11vnc with the -dk option and see what it prints when you press shift/fn + other keys? (You may want to use a paste bin for the output)