Xorg will typically spike shortly after running something new, especially things that redraw the screen a lot (like top or Conky). It's the thing that controls the display IO. If it's staying at >20% use, or is doing so even when the display is off (and you're in via ssh) then something is doing re-draws too often. But it's typical for it to spike to 20% at times, especially when the display is active. (On mine it spikes to 20% on launch but drops to under 5% with one-second updates.)

This is clearly a process issue, which means it's probably something in user-space causing it. Please search for one of the billions of threads on how to determine what process is using your battery (I'll give you a hint, it's probably NOT Xorg) and go from there.

There's probably an app or a widget that's doing something it shouldn't do in the background. Also, avoid installing anything new other than maybe a tool you need to find out what's causing it. Installing new things (like "patches" or other "management" software) will just complicate the issue by adding more crap to the pile of things to debug.