From what I've read in researching the error message, the mencoder devs think min rate should always be 0 for vbr and be equal to max rate for cbr. They've just recently begun enforcing it.