Hm.. I think I have to agree with gammer here. The cards aren't that slow. FileZ on my Palm (which isn't fast with cards) is lightning fast when walking up and down SD card directories.
I did a little test with my old 1GB SD card on my N800.
find /media/mmc1 | wc -l
shows 1840 files and directories.
Then open the card door, close it (to re-mount the card, which will flush any memory buffers that would artificially speed up the process). Then measure how long it takes to read every file entry in every directory:
time ls -lR /media/mmc1 > /dev/null
real 0m 2.71s
user 0m 0.30s
sys om 1.96s

Quite fast..
gammer, you could try the same excersize with your particular card. That would give you an idea of how fast it could be done.
Of course, the file manager does much more - if there are pictures it'll thumbnail them and so on. Which means that it'll have to read at least the beginning of every file, not just file entries, just to classify them.