The driver idea isn't sounding so far-fetched. As it turns out, not all drivers talk to video hardware:
http://www.x.org/wiki/vmware

So it's possible to create drivers that put up information (shared memory?) and receive input from a standard Android application. The benefits to this setup are:

1) No unique display drivers need be written so it can be distributed to any working android system in APK form
2) Very easy access to apps directly within Android -- can be launched with a launcher icon like any other app for seamless integration. Switch between apps as one normally would.
3) Can easily take advantage of advanced features of Android (eg. multi-touch) and feed them into x.org as these features become available. For example a large 1920x1080 display that can be navigated via two-finger scrolling, or simply two-finger swipe between apps/desktops.
4) Benefit from the (large) x.org codebase for quick deployment without needing to create from scratch nor reverse engineer the system.

The end result is a full Debian environment including X, running seamlessly in Android. Apps launch as any other app does (though will need to be explicitly closed), and at native speeds.

For many it will be a novelty. However, there are certain well-developed desktop apps that would be welcome additions to these capable devices. I personally would like a graphical sftp file-manager, a drawing program (eg. gimp & inkscape), and the full-featured (though painfully bloated) openoffice.org .

EDIT:
Another benefit of a setup like this is having access to the MeeGo/N900 repository of apps! There's nothing wrong with a little cross-platform, eh?