MKV is a contianer format https://en.wikipedia.org/wiki/Matroska
I have the FreemanGordon 720p upgrades installed and I can get 720p contianing MKV's to play, others not. Sometimes it just takes renaming the file to AVI or MP4 though it is a long time since I have needed that trick and never in my memory on the N900.
The file that I tried now and just failed was a MKV containing h264 720x480 video and mpeg-4 AAC audio.
BTW where did you find VLC, chroot Debian?
(edit)
found it from videolan repo
http://nokia-n900.com/vlc-on-the-n900/
(edit 2)
so I can play another with H.264 / AVC 720 x 404 and MPEG-4 AAC it plays even in the native player
I installed VLC and I also was only able to get audio for the h264 720x480 video and mpeg-4 AAC audio file which fails in the native player.