What are you using to obtain Xorg memory statistics? I have a few machines with 100+ days uptime running Xorg, memory usage is not even 50% of the values you have. Applications can reserve as much memory as they want, this doesn't mean they are actually using it or wont release it if the need arises.