Not only can you just swipe it away, but the swipe provides its own transitions effect. With the button you get none, and having the button adds some extra code and therefore makes the loading of each folder potentially slightly slower. This is my understanding anyway. See earlier comments in the thread.

There is a school of thought that buttons are a bad thing in the N9 UI. Nokia has buttons here and there, but potentially could have done without any at all.