I have converted many rmvb files without problems. Are you using the mplayer/mencoder included, or you have replaced this files with a old version?