X11 is the protocol. There are multiple servers which implement the X11 protocol (xfree86, xorg). This is the basic graphics management server and is responsible for actually putting all the graphic primitives (windows, shapes, dialogs) on the screen. One step above that is the Window Manager; sawfish, twm and fluxbox are examples of various WMs. GNOME, KDE, XFCE and the crew are called "desktop environments" and combine a window manager, prettier display primitives, a high(er) level API and usually some widgets that tie into that API.

One of the neat features is that the X11 protocol was designed for network transparency. That means that I can run an X application on host A, but have the actual graphics displayed on the local X server running on host B.