You need to add a line 'subcp=cp1251' (without quotes) to /home/user/.mplayer/config file on your device

The information that was supposed to be enough for changing subtitles charset is available at the bottom of maemo mplayer home page. If you had some troubles understanding it, please explain what exactly was hard for you, we may try to improve that part on the website.

Surely, it would be better to have subtitles charset (and other options) configurable from GUI, but we still did not find a volunteer who would like to do that.