Xorg/x11 is a set of libraries/protocols that allow drawing to the screen. (x11 is the name of the protocol, just like http is the name of the protocol used for www)

GTK, QT, OpenGL, Window Managers (matchbox2) and all the other UI stuff you hear about here all go on top of/through X11.

Without X11 on linux, your only other options are direct framebuffer access (the older Qtopia platform was also linux based and eschewed X11 for direct framebuffer access to reduce overhead) or just a console (text command line).

X11 also provides network transparency, making it possible to run apps on one machine, and have them display on another (like running openoffice on my desktop, but having the UI display on my n900)

As a fun sidenote, you can try using the framebuffer out option in mplayer on the n900 and watch x11 and mplayer compete for drawing the screen