I am quite certain HandBrake *will* work. You just need to get the right settings.

I would try something much smaller for testing until you find the settings that work.. and then rip the 3 hour movie.