I have. mkv (h264/ac3) It can play mkv files no issues as long as it isn't too cpu intensive and is not HD resolution. (Maemo5 has a video card driver limitation where it doesn't allow videos with higher resolution than the 864X648)

I have also tried playing 720p mkv file after booting into Nitdroid on my N900 and it plays it at around 5-8fps while overclocked to 1GHz. (this is all CPU decoding using Rockplayer) If we can somehow utilize the DSP, it will most likely be watchable. But seems we may never see this happening anytime soon.