[SailfishOS] [Announce] Native offline maps: OSM Scout Server - Page 11 - maemo.org

Active Topics

The N900 is now 15 years old! (7)
to Nokia N900 by Danct12 - 3 days, 8 hrs ago
Sailfish OS for the Motorola Moto G7 Power (XT1955-5) - (ocean) (4)
to SailfishOS by Maemish - 4 days, 4 hrs ago
F(x)tec Pro1x (QWERTY) for sale (0)
to Buy & Sell by aln00000 - 6 days, 12 hrs ago
more...

Page 11 of 84

Thread Tools

rinigus	2017-03-05 , 15:51
Posts: 1,414 \| Thanked: 7,547 times \| Joined on Aug 2016 @ Estonia	#101

Originally Posted by eson

Top menu "Map Manager" and Settings/Maps storage "Select" is both unavailable for me.
This is output when starting Settings...

I rushed with the response, I think its much easier.

To "Select" some folder, you would have to go in FileManager widget into a parent folder, make a long press on the folder that you want to choose and then the "Select" will appear as a context menu.

For example, let's say we want to choose /home/nemo/Maps:

* You navigate to /home/nemo

* Long press on "Maps"

* Context menu "Select" appears, choose that

* Accept in Settings

* Maps storage is selected, Map Manager would be available on the main window.

Let me know if it helped.

Quote & Reply |

The Following 4 Users Say Thank You to rinigus For This Useful Post:
eson, Feathers McGraw, juiceme, velox

	seiichiro0185	2017-03-05 , 16:34
	Posts: 270 \| Thanked: 610 times \| Joined on Nov 2007 @ Leipzig/Germany	#102

Just tried the new version, selecting the storage folder and subscribing to a map works fine, but the download stops with an error after about 200-300 MB (I'm trying to download the germany map with geocoder data). Error is always "Connection Closed". My internet connection is stable without any interruption during the download.

Any idea what could be the problem?

__________________
N800 -> N810 -> N900 -> N9 -> Jolla & TOHKBD -> Jolla C -> Xperia X -> XA2 Plus Dual Sim

http://www.seiichiro0185.org

Quote & Reply |

The Following 2 Users Say Thank You to seiichiro0185 For This Useful Post:
juiceme, rinigus

	eson	2017-03-05 , 16:37
	Posts: 363 \| Thanked: 1,375 times \| Joined on Nov 2015 @ Sweden	#103

Originally Posted by rinigus

I rushed with the response, I think its much easier.

To "Select" some folder...

Ahhh... got it! This is what I saw, and found very confusing, or maybe I'm just stupid.

Thanks!
.

Attached Images

Quote & Reply |

The Following 2 Users Say Thank You to eson For This Useful Post:
juiceme, rinigus

MartinK	2017-03-05 , 16:53
Posts: 1,548 \| Thanked: 7,510 times \| Joined on Apr 2010 @ Czech Republic	#104

Originally Posted by rinigus

At present, I added a small credit for CDN. I do wonder for how long would it last, probably it would cover the first TB of downloads.

Now its a great time to propose your server space for distribution of maps.

Sure! I've talked a bit more with the NLP lab people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.

So now I need to know how can I "join" the OSM Scout Server data CDN with these resources.

Are there some mirroring scripts I could add to cron or something similar ?

I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.

__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)

Quote & Reply |

The Following 4 Users Say Thank You to MartinK For This Useful Post:
Feathers McGraw, juiceme, rinigus, seiichiro0185

rinigus	2017-03-05 , 16:58
Posts: 1,414 \| Thanked: 7,547 times \| Joined on Aug 2016 @ Estonia	#105

Originally Posted by seiichiro0185

Just tried the new version, selecting the storage folder and subscribing to a map works fine, but the download stops with an error after about 200-300 MB (I'm trying to download the germany map with geocoder data). Error is always "Connection Closed". My internet connection is stable without any interruption during the download.

Any idea what could be the problem?

So far, the biggest map I have tried is Spain and that downloaded without any issues. However, in my place, it wasn't happy about mobile net. That's not surprising since we have a place/operator combination that resets connections quite frequently. Worked fine on WiFi.

One suspicion would be that maybe phone enters a deep sleep while downloading leading to "Connection closed". Namely, I have not used any keepalive and the both of my devices (N4 and OPX) worked just fine to my surprise. So, I didn't even look into keepalive locks. Maybe its not working for you.

Try to put your device on charging and download then.

On the technical part: many of the databases are split into several parts. As long as you see INFO messages on the main screen with "File downloaded" you are progressing even if after restart of the download the apparent size remains the same. So, you should be able to download by these batches... However if you are stuck on some bigger file (larger than 200-300 MB) then it will be trying to get the same file again.

Let me know if putting on charger helped. If that''s a problem, I'll add keepalive lock and we should be OK.

Quote & Reply |

The Following 4 Users Say Thank You to rinigus For This Useful Post:
Feathers McGraw, juiceme, seiichiro0185, velox

rinigus	2017-03-05 , 17:00
Posts: 1,414 \| Thanked: 7,547 times \| Joined on Aug 2016 @ Estonia	#106

Originally Posted by eson

Ahhh... got it! This is what I saw, and found very confusing, or maybe I'm just stupid.
Thanks!
.

What you were selecting is actually a parent folder (".."). That's disabled to prevent users selecting /home/nemo instead of /home/nemo/Maps and later "cleaning up" their whole home dir

[you have to authorize deleting, unless you do that will not happen]

Quote & Reply |

The Following 5 Users Say Thank You to rinigus For This Useful Post:
Amboss, eson, Feathers McGraw, juiceme, velox

rinigus	2017-03-05 , 17:02
Posts: 1,414 \| Thanked: 7,547 times \| Joined on Aug 2016 @ Estonia	#107

Originally Posted by MartinK

Sure! I've talked a bit more with the NLP lab people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.

So now I need to know how can I "join" the OSM Scout Server data CDN with these resources. Are there some mirroring scripts I could add to cron or something similar ?

I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.

Brilliant! I'll think how to do that and contact a bit later tonight

Quote & Reply |

The Following 2 Users Say Thank You to rinigus For This Useful Post:
juiceme, velox

	seiichiro0185	2017-03-05 , 18:15
	Posts: 270 \| Thanked: 610 times \| Joined on Nov 2007 @ Leipzig/Germany	#108

Originally Posted by rinigus

[....]

Let me know if putting on charger helped. If that''s a problem, I'll add keepalive lock and we should be OK.

I tried with having the device charging(and device was on wifi for all tries so far), but the download stopped with the same error again (This time at about 400MB). Unfortunately the areas.dat for germany is bigger than that, so it doesn't get further than trying to download this file with every new start. Btw downloading the map for germany with osmscout worked fine last time I tried (same wifi, device not charging).

__________________
N800 -> N810 -> N900 -> N9 -> Jolla & TOHKBD -> Jolla C -> Xperia X -> XA2 Plus Dual Sim

http://www.seiichiro0185.org

Quote & Reply |

The Following 2 Users Say Thank You to seiichiro0185 For This Useful Post:
juiceme, rinigus

rinigus	2017-03-05 , 19:05
Posts: 1,414 \| Thanked: 7,547 times \| Joined on Aug 2016 @ Estonia	#109

Originally Posted by seiichiro0185

I tried with having the device charging(and device was on wifi for all tries so far), but the download stopped with the same error again (This time at about 400MB). Unfortunately the areas.dat for germany is bigger than that, so it doesn't get further than trying to download this file with every new start. Btw downloading the map for germany with osmscout worked fine last time I tried (same wifi, device not charging).

Not that it helps you - it worked for me when I downloaded Germany. I presume you mean the whole country, 4GB+ unpacked.

I presume you do have enough space to download full Germany.

Since you are on the charger already, I don't have to look into keepalive yet - it seems that this is not of a source of error.

Frankly, I don't know what to suggest. Its most probably something on CDN provider and I don't know how to debug it. I use S3-compatible service and, as far as I have seen in the documents, they do talk about large files and chopping them to pieces to ensure better redundancy. But how to use it, I have not really looked.

Maybe your connection is too fast/too slow and then some error on CDN provider backend is triggered?

In the case of areas.dat, you are downloading about 700MB bzip2'ed file. So, you are getting closer. Would you mind to try some part of Germany and later try to get full set again a bit later?

Let me know how it goes. We can always complain and ask for advice from the provider.

PS: OSM Scout uses @Karry's own server and has also different download implementation.

Quote & Reply |

The Following 2 Users Say Thank You to rinigus For This Useful Post:
juiceme, velox

rinigus	2017-03-05 , 21:25
Posts: 1,414 \| Thanked: 7,547 times \| Joined on Aug 2016 @ Estonia	#110

Originally Posted by MartinK

Sure! I've talked a bit more with the NLP lab people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.

So now I need to know how can I "join" the OSM Scout Server data CDN with these resources. Are there some mirroring scripts I could add to cron or something similar ?

I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.

Now a proper reply: Thank you very much for an offer! It would be of great help and I hope that this service would not be abused. We have now some amount of time to figure it out while my current CDN is up.

As it is, I can process the planet reasonably fast on the system available to me. Its not as beefy, but made for serious calculations as well

. Maybe later we could move data processing as well, but there is no need now for data processing. Unless they are interested on working on address parsing and NLP approach behind libpostal. Then we can discuss it separately.

At present, I use ~40GB for all datasets. I presume that during dataset version changes (version incompatibilities) we would need double of that space for a week or two. If the other backends would be added, the data requirement may increase, but let's see about that.

Datasets are served statically. In essence, we have a tree with different backends (osmscout, postal, geocoder-nlp) and a JSON file that describes them. By changing a line at https://github.com/rinigus/osmscout-...ded.json#L7112 , I could move all download requests to a new location (when you hit "Update list" in Map Manager you download that file by OSM Scout Server from github).

I presume we would keep conditions human and update once in a month?

So, it would boil down to the way we move the data between the servers. Or if its too complicated, we would just need cron jobs running. But that we could just discuss via PM/email. I'll get in touch with you tomorrow, maybe rather late though.