I have exactly the same problem.

I've used Mediatomb in Ubuntu. What's funny, I cannot play those files using N900 media player's UPNP functionality. I can play videos of other Nokia devices. I'm using video/mp4 as mime type.

But what is perhaps more important, I cannot play those video files even if I plug in N900 directly on PS3. Files are simply 'unsupported data' if I use 'view all' functionality.

Any solution?