I agree that the driver part may becomes a quite big issue, I see already all the threads here like "My DVB-T stick doesn't works" or "My scanner can't be detected"..., still as long as there exist drivers for other Linux based phones it shouldn't be a big problem to port them. Making a X86 driver with ARMEL compatible is quite a lot of work, and you should give the "friendly Linux geek" at least a small donation then :P

I suggest to port drivers for the most common devices and collect them here or make a repository for it.