View Single Post
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#1473
Originally Posted by Kossuth View Post
I tried to download the offline routing data for Finland at

http://modrana.org/data/monav/

But the pack is only 61 bytes in size and the package seems to be empty. I also noted that the packages of other countries were also very small in size. The whole Europe pack seems to be correct size, (4.3gigs), but I'd rather download only the countries I need? Is there something funny going on with the offline packs?
I known about this and I'm working on a fix.

How did it happen ?
While updating the repository, all extracts present on downloads.geofabrik.de are are downloaded (in sequence by a single thread) and processed (by many, MANY threads ) into Monav routing data packs. The size of the input data downloaded for each repository update is about ~100 GB and each update takes about 5 hours from start to finish.

After running an update recently, I've noticed it took just about 2 hours. I've investigated and found that downloads from Geofabrik started to fail with a 503 error after a certain amount of the data needed was downloaded - I've contacted Geofabrik and indeed, the download was tripping some overuse protection they have (recently ?) added to their repository.

I've cleared the situation with them and they granted the modRana repository a usage exception.

Still, I agree that downloading about 100GB from their repository is not very efficient and I've started working on a better long term solution.

Plan of action
The repository needs regional extract that it can turn into Monav routing data packs (and hopefully in the future also other kinds of useful geodata). Geofabrik obtains those extract by splitting the master database dump periodically released by the OpenStreetMap project, the so called Planet.osm file.

So I've decided to to implement the same procedure for the modRana data repository. A the planet file is about 25GB in gzip compressed PBF format and about 45 in uncompressed PBF. Thankfully, it can be updated incrementally using the osmupdate utility, so the amount per repository update should be drastically reduced from the previous ~100GB to maximally a few hundred megabytes per update.

Splitting the Planet.osm file can be quite demanding, but can be very efficiently parallelized. Thankfully, the machine that has been so kindly provided for shared used by the Natural Language Processing laboratory of the Masaryk University in Brno (Thanks! ) the modRana routing repository has about 48 logical CPU units, so it can do the Planet-cracking very well.

All in all, local splitting should both massively spare download bandwidth and make the modRana repository independent on Geofabrik.

The work is already about 50% done and I plan to finish it in the next few days. Of course I'll let you know in this thread once the first new update run is done.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)

Last edited by MartinK; 2013-10-07 at 18:15.
 

The Following 15 Users Say Thank You to MartinK For This Useful Post: