Yes indeed! I believe this is the same hack that uses a Debian environment and VNC to draw the display. While it works (well, I assume), I'm sure it's a little slow for some tasks, and not terribly battery friendly (though I could be wrong about this -- however it seems like a lot of overhead for rendering).

http://hackaday.com/2009/02/22/x11-on-android/

I wonder if it would be possible to forego the display drivers, and write an app that emulates X written using the Android stack? The benefit of this setup being that it can be rolled out to any device without custom drivers being required. Perhaps a single generic 'android display driver' that hooks directly into the X emulator for low latency display...