There is a right way, and many wrong ways for doing this. There is a command line application for doing this the right way, which retains pitch characteristics, but I can't remember what it is. As for the wrong way; you can usually do it with your player, but the pitch goes way up. This makes it pretty much un-listenable, because everybody sounds like a chipmunk.

One trick I've done in the past is to load the file into audacity and pass it through a filter that removes silence. It's not as dramatic a difference as you're looking for though, and it's time consuming to do. Overall you probably loose time that way. To give you an idea of the results, I once ran a TLLTS podcast through the filter with pretty agressive settings, and it shortened a 140 minute podcast by about 12 minutes. That's an extreme example though as those guys have a lot of audio lag in their various telephony connections, which causes a lot of artificial dead air.