The reason is very simple: it uses A-GPS and it uses the GSM pole positions to assist to a quick fix. I have the same with the Nokia 5800. But I tested it in an area without GSM coverage and after 30 minutes walking NO FIX with the 5800. I tested both the N810 and 5800 (the latter with A-GPS disabled) and the N810 fixed even earlier (but stll after several minutes).
So I think the N810 and phones with GPS have a poor GPS unit. That is why I use a Bluetooth GPS which fixes within a few minutes in the worst case and many times within a minute, without GSM network.