Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus
Thanks :-) I downloaded from Mega.

Re: [Announce] Native offline maps: OSM Scout Server
 

this works, but...
1. I have for Street,8: Street,8-Street,18-Street,81-Street,181-Street,22/8 etc... It's OK?
2. I request Shop_A, MyCity and I have CityB-MyCity,highway...
3. I have for Shop_B: monument "Shop_B" only(not Shop_B. because names are the same).

I have good answers for alternative method. I tested only for my city, cyrillic, offline maps.

P.S. It's possible to download maps for russian regions from http://gis-lab.info/projects/osm_dump/

Re: [Announce] Native offline maps: OSM Scout Server
 

1: In this case it should search for Street* first, and 8* after that. So, I would expect the houses which start with 8, but not 18 and 181. For 22/8, it may have split it into two parts by libpostal normalization, so that could probably happen. Fortunately, I sort it according to the string length => you should get 8 as a first hit. Please verify 18 and 181 houses in your example. If they are there, would be good to get an example which I can specifically test.

2: If the search cannot find a specific reply (judged by resolved number of hierarchies), the result for a search with the latest resolved level dropped is returned. I presume the Shop_A was not found and then it just looked for anything with MyCity.

3: It looks to me that Shops A and B are missing. Let's try to fix it. I presume its due to the fact that these shops are not included in local search database in libosmscout. For fix, I would need their type in OSM.

So, in general, if you miss some POIs in the search, let me know. Find the type as follows:

a. go to http://www.openstreetmap.org
b. find that missing POI, preferably in search
c. see what's its type. For example, for http://www.openstreetmap.org/node/2558987540 its "amenity" "restaurant".
d. let me know about it (github at https://github.com/rinigus/geocoder-nlp or this thread)

I would have to include that by modifying import data type file and regenerate planet databases and push them out.

I presume we would have to do few iterations and let's hope it would stabilize at some point :)

Thank you for suggestion. Its not a problem with getting PBFs - I use geofabrik for that. Its serves whole planet chopped by countries. Problem is mainly how to distribute my databases :)

I am now regenerating a planet using a newest libosmscout importer that should be able to resolve admin hierarchies much better. As soon as its ready, I'll push it into mega.nz and would update the links.

Re: [Announce] Native offline maps: OSM Scout Server
 

Just did a quick Test with the new sqlite DB and the search is a lot faster now. Searching for the "Parkstraße" address which took 3-4 minutes before now takes about 10 seconds.

I'll test some more and report back if I find anything else ;)

Thanks for your amazing work on this!

Re: [Announce] Native offline maps: OSM Scout Server
 

Hi there is it possible, to integrate german language to Scout Server ?
so that i have german driving instructions with offline support ?

:o

Re: [Announce] Native offline maps: OSM Scout Server
 

@seiichiro0185: great to hear that it helped!

I have just uploaded the new version of the geocoder databases to mega.nz. As earlier, please use

http://tiny.cc/geocodernlpdata

since I can change the link after each upload. For tiny.cc-blocking countries, link is

https://mega.nz/#F!1RtTyQwS!r5QDqjwQ...u9Q7Q!8Zt1RaQB

Databases are under geocoder subfolder. Postal part is unchanged.

Changes:

1. Shops are listed. Reported missing by @XOleg, please check if its ok now.

2. I have changed the internal database index and it should search much faster now. That's important for countries like Germany. As a result, database increased a bit in a size, but its probably reasonable compromise.

3. Resolution of admin relationships should be much better than before. So, if there was an error in earlier version, check if its fixed. I expect it to be as good as on http://www.openstreetmap.org, with just few misses (I'll let you find them if you can).

@seiichiro0185, would be great if you could test the speed with the latest version. It should be as fast as the version you had, but the file size is smaller.

Re localization, aka in German, @monkeyisland:

Sure, its possible. Its just German users would have to translate the strings. See https://github.com/rinigus/osmscout-...r/translations for instructions and example translations. I'll be happy to merge PRs with translations.

If someone knows how to setup easier system for users, would be great if you could submit patch with its implementation. Or open an issue and try to give me detailed instructions at https://github.com/rinigus/osmscout-server . The easier it is for me, the faster it would be implemented. :)

Thank you all for using / testing it!

Re: [Announce] Native offline maps: OSM Scout Server
 

1. rinigus, excuse me for *8*,but */8 exists. I see *8* for alternative method.

2, 3. It's OK after update.

Geofabrik provides for Russia federal districts only. This requires volume... gis-lab provides most of 89 regions separately.

Re: [Announce] Native offline maps: OSM Scout Server
 

OK, I see. */8 exists due to the way libpostal normalizes the strings. Its a side effect of allowing to search for streets with the names like Dr. John Smith by using "Smith" in the search string. How to get rid of */8 in response in a general way, I don't know. Since the database is in an open form (SQLite has several tools to work with it) and SQL queries are all in geocoder source code, anyone is welcome to analyse the alternative search commands. Just note that w need to use database indexes to make it fast - otherwise anyone with a bigger country would complain.

I see. Note that I am not distributing the libosmscout maps, just the databases required for geocoding. Libpostal part would be hard to reduce even further by taking smaller part of a country without chopping languages part that I don't have data to do (and probably time either).

SQLite geocoding database would reduce, indeed, if I take these 89 regions. Full Russia is now 720MB. I guess that by dividing it into 9 federal districts should make it reasonable. Don't forget that we have to have ~700MB libpostal dataset already anyway to make it work with the supported languages.

At present, I'll try mainly to work on organizing these databases into a bit of an order first and try to make it simply downloadable. Then I could look into chopping the planet into additional subregions. Surely that would be needed for Germany, Russia, and few others.

Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus
Thanks for huge work

Re: 0.6.2
 

I have just published an updated version (0.6.2) which includes German translation - thank you @monkeyisland!

To simplify the work on translations, a project has been opened at https://www.transifex.com/rinigus/osm-scout-server/ . @monkeyisland has been of great help and we tested it on his translation.

In addition, when you ask for a route somewhere by giving the target by name, the calculated route should display the search result under the "Target" message.

Now back to developing it further...

Re: [Announce] Native offline maps: OSM Scout Server
 

I dont know if the issue comes from poor maps or osm scout server.
Now i can choose maps from osm. But when i try navigate with osm scout in poor maps osm scoit server will crash.
Offline search is also not possible.

Re: [Announce] Native offline maps: OSM Scout Server
 

If OSM Scout Server crashes - its a problem with the server, not Poor Maps.

I suspect that the crash is related to the change in the map format that was not tracked. See https://github.com/rinigus/osmscout-server/issues/61 . I have not updated libosmscout in the latest builds and the bug is still present in the current server builds.

To check if you are hit with the same bug, please reimport the maps using importers from https://github.com/rinigus/libosmsco...0.git.20170126 . Let me know if it helped.

Re: [Announce] Native offline maps: OSM Scout Server
 

Hi there i tested to import a little map i take berlin-latest from geofabrik.
OSM cant open the map it shows an error.
so i dont have an x64 systems i compiled the importer tool by myself for powerpc.
the compiling works great and the map would be converted without errors but osmscout cant open this test map...:(

Re: [Announce] Native offline maps: OSM Scout Server
 

Hi. What error you get? Can you share your generated map please?

Re: [Announce] Native offline maps: OSM Scout Server
 

The last version of OSM Scout Server was released against buggy (in terms of version control) libosmscout. Bug was found by @Karry and got fixed.

I would suggest to wait few days, since I am preparing a new release that would resolve this issue as well. Its progressing very nicely with only few issues to get resolved (milestone 0.7 at project's github page).

If you cannot wait, please use https://github.com/rinigus/libosmsco...0.git.20170126 for compilation of the importer. If you have used some other version, you would either get crashes or refusal to load the map.

cheers,

rinigus

Re: [Announce] Native offline maps: OSM Scout Server
 

Hello @rinigus
i updated all my maps with libosmscout-importer.Linux.x86_64.tar.gz > 0.0.git.20170221 it is not work with osmscoutserver MSSG:ERROR: 13:12:47 It is not possible to open the database. work ok with karry-osmscout-sailfish

Re: [Announce] Native offline maps: OSM Scout Server
 

Hi @karlos devel:

please use only importer based on 0.0.git.20170126 for 0.6.x server.

@karry's osmscout is much smarter than a server in terms of version checks and I am not surprised it works over with that application. Its also newer and supports the last version format as produced by the importer based on 0.0.git.20170221 .

I am using libosmscout release 0.0.git.20170221 for 0.7.x development, I probably made a mistake by tagging it too early before release of the new version of the server. Just wanted to be sure that I don't forget to tag it this time.

Re: 0.7.0 release
 

This is announcement for a new release: 0.7.0

This release introduces Map Manager and incorporates maps distribution via CDN. This is a major change in approach and should make the use of the server simple. In particular, use of geocoder-nlp should be easy and all dependencies are resolved automatically. I hope that you would enjoy using it!

Starting from this release, maps are supposed to be used only through download. This is to ensure that its simple to update and cleanup free space. It should also help with the incompatibilities between the versions. So, even if you had maps prepared for your devices, you would have to download new maps.

Distribution model is as follows. I have a file in the project github source that has a list of provided countries/regions/features and URL to CDN base. Location of the file will stay and you could always update the provided list via GUI/Map Manager. Then you would have to decide which countries do you want (through subscription) and whether you want to store files needed for geocoder-nlp. After that, start to download from CDN.

When you start new server, you would be asked to allocate a separate folder that OSM Scout Server would manage. Note that you are expected to let it perform all operations within that folder (or follow its lead when shown the files that it doesn't use anymore, for example). I am not allocating the folder automatically to give you more control over proceedings (~nemo, SD card, ...). If its too confusing, I can allocate .local/shared/harbour-osmscout-server for it. But that may lead to filling up of internal storage for some users.

The files downloaded by server are all tracked in a small database, so it knows the versions and timestamps for each of them. If you download files using @Karry's OSM Scout into the same folder, they would be unknown by the server. We would have to discuss with @Karry how to proceed and ensure that compatible maps can be used in the both applications, but that may take some time to figure it out.

With geocoder-nlp easily accessible, don't forget to specify which languages do you want to use for parsing addresses. Otherwise, libpostal may eat lots of RAM!

I will gather your questions and probably add also small Q&A into the first post of the thread, if needed.

I will work on making maps distribution simpler. For that, I plan to use @MartinK scripts to chop the planet into regions and import all polygons into github. It would take some time to make it all work, but should allow us to extend provided regions through github development process.

At present, I added a small credit for CDN. I do wonder for how long would it last, probably it would cover the first TB of downloads.

Now its a great time to propose your server space for distribution of maps. Or, propose a model where we could collect money for distribution. In the money collecting case, I would prefer to have it done via some organization, to stay away from all possible issues with taxes.

Its been lots of work on making it possible and I might have forgotten to mention something important.

Enjoy!

Re: [Announce] Native offline maps: OSM Scout Server
 

Top menu "Map Manager" and Settings/Maps storage "Select" is both unavailable for me.
This is output when starting Settings...

Re: [Announce] Native offline maps: OSM Scout Server
 

Map Manager should be disabled until Map storage folder is selected. So, that's OK.

What is very strange is that you cannot select Maps storage under Settings. Frankly, I cannot reproduce it. Would you mind to move away (don't delete - it maybe its needed in future for fixing this bug) OSM Scout Server settings and then try without your old settings?

As for the messages, it doesn't seem that anything is unusual, at least at the brief look.

Re: [Announce] Native offline maps: OSM Scout Server
 

I rushed with the response, I think its much easier.

To "Select" some folder, you would have to go in FileManager widget into a parent folder, make a long press on the folder that you want to choose and then the "Select" will appear as a context menu.

For example, let's say we want to choose /home/nemo/Maps:

* You navigate to /home/nemo

* Long press on "Maps"

* Context menu "Select" appears, choose that

* Accept in Settings

* Maps storage is selected, Map Manager would be available on the main window.

Let me know if it helped.

Re: [Announce] Native offline maps: OSM Scout Server
 

Just tried the new version, selecting the storage folder and subscribing to a map works fine, but the download stops with an error after about 200-300 MB (I'm trying to download the germany map with geocoder data). Error is always "Connection Closed". My internet connection is stable without any interruption during the download.

Any idea what could be the problem?

Re: [Announce] Native offline maps: OSM Scout Server
 

Ahhh... got it! This is what I saw, and found very confusing, or maybe I'm just stupid. :confused:
Thanks!
.

Re: 0.7.0 release
 

Sure! I've talked a bit more with the NLP lab people and they are indeed fine with both hosting and generating data for OSM Scout Server as long as the data is hosted as static files that are batch-updated once in a while (every few days, etc.). As that condition is satisfied it should be possible to host at least ~100 GB, without transferred bandwidth limitations.

So now I need to know how can I "join" the OSM Scout Server data CDN with these resources. :) Are there some mirroring scripts I could add to cron or something similar ?

I can also help generate the data sets if needed, on one of the NLP compute machines, a fairly beefy system (220 GB RAM & 48 CPU core) running Fedora 24 with some 200-300 GB of local storage that could be used for the data generation run.

Re: [Announce] Native offline maps: OSM Scout Server
 

So far, the biggest map I have tried is Spain and that downloaded without any issues. However, in my place, it wasn't happy about mobile net. That's not surprising since we have a place/operator combination that resets connections quite frequently. Worked fine on WiFi.

One suspicion would be that maybe phone enters a deep sleep while downloading leading to "Connection closed". Namely, I have not used any keepalive and the both of my devices (N4 and OPX) worked just fine to my surprise. So, I didn't even look into keepalive locks. Maybe its not working for you.

Try to put your device on charging and download then.

On the technical part: many of the databases are split into several parts. As long as you see INFO messages on the main screen with "File downloaded" you are progressing even if after restart of the download the apparent size remains the same. So, you should be able to download by these batches... However if you are stuck on some bigger file (larger than 200-300 MB) then it will be trying to get the same file again.

Let me know if putting on charger helped. If that''s a problem, I'll add keepalive lock and we should be OK.

Re: [Announce] Native offline maps: OSM Scout Server
 

What you were selecting is actually a parent folder (".."). That's disabled to prevent users selecting /home/nemo instead of /home/nemo/Maps and later "cleaning up" their whole home dir :) [you have to authorize deleting, unless you do that will not happen]

Re: 0.7.0 release
 

Brilliant! I'll think how to do that and contact a bit later tonight

Re: [Announce] Native offline maps: OSM Scout Server
 

I tried with having the device charging(and device was on wifi for all tries so far), but the download stopped with the same error again (This time at about 400MB). Unfortunately the areas.dat for germany is bigger than that, so it doesn't get further than trying to download this file with every new start. Btw downloading the map for germany with osmscout worked fine last time I tried (same wifi, device not charging).

Re: [Announce] Native offline maps: OSM Scout Server
 

Not that it helps you - it worked for me when I downloaded Germany. I presume you mean the whole country, 4GB+ unpacked.

I presume you do have enough space to download full Germany.

Since you are on the charger already, I don't have to look into keepalive yet - it seems that this is not of a source of error.

Frankly, I don't know what to suggest. Its most probably something on CDN provider and I don't know how to debug it. I use S3-compatible service and, as far as I have seen in the documents, they do talk about large files and chopping them to pieces to ensure better redundancy. But how to use it, I have not really looked.

Maybe your connection is too fast/too slow and then some error on CDN provider backend is triggered?

In the case of areas.dat, you are downloading about 700MB bzip2'ed file. So, you are getting closer. Would you mind to try some part of Germany and later try to get full set again a bit later?

Let me know how it goes. We can always complain and ask for advice from the provider.

PS: OSM Scout uses @Karry's own server and has also different download implementation.

Re: 0.7.0 release
 

Now a proper reply: Thank you very much for an offer! It would be of great help and I hope that this service would not be abused. We have now some amount of time to figure it out while my current CDN is up.

As it is, I can process the planet reasonably fast on the system available to me. Its not as beefy, but made for serious calculations as well :) . Maybe later we could move data processing as well, but there is no need now for data processing. Unless they are interested on working on address parsing and NLP approach behind libpostal. Then we can discuss it separately.

At present, I use ~40GB for all datasets. I presume that during dataset version changes (version incompatibilities) we would need double of that space for a week or two. If the other backends would be added, the data requirement may increase, but let's see about that.

Datasets are served statically. In essence, we have a tree with different backends (osmscout, postal, geocoder-nlp) and a JSON file that describes them. By changing a line at https://github.com/rinigus/osmscout-...ded.json#L7112 , I could move all download requests to a new location (when you hit "Update list" in Map Manager you download that file by OSM Scout Server from github).

I presume we would keep conditions human and update once in a month?

So, it would boil down to the way we move the data between the servers. Or if its too complicated, we would just need cron jobs running. But that we could just discuss via PM/email. I'll get in touch with you tomorrow, maybe rather late though.

Re: [Announce] Native offline maps: OSM Scout Server
 

Just trying out the new version, all the dialogs seem to work as they should.

Might be nice to suggest some locations for data with the ability to select somewhere arbitrary as well - most people probably won't care what the folder is called... you could suggest ~/OSMServer or sdcard/OSMServer and create the chosen folder too, which would prevent the user having to leave the app and come back.

I also got the connection closed error downloading the map data for Great Britain (on Fairphone 2, if that matters). I'll try again whilst charging and see if that helps.

Re: [Announce] Native offline maps: OSM Scout Server
 

Also, in the download counter what do the D and W in brackets stand for?

Re: [Announce] Native offline maps: OSM Scout Server
 

I seems there is some kind of incompatiblility between my Jolla C, my WIFI at home, my ISP, the CDN provider and the Download implementation in osmscout-server (maybe not all of them, but no idea what the real problem is here).

Downloading maps for parts of Germany does work for some, but not for the bigger ones. Several tries for the whole country would always fail at some point downloading areas.dat.

Today I tried the download with the WIFI at work, and it worked perfectly. No interruptions, no errors, just downloaded the whole map in one try..

I have no idea what the problem might be, since every other Download so far worked without any Issues on my home WIFI with my Jolla C.

Re: [Announce] Native offline maps: OSM Scout Server
 

Great idea, I opened an issue at github regarding it and would implement it when I get there

OK, it looks that I need to make a more robust downloader. I think I have an idea on how to do it, it may just take some time. Let me know if your next tries worked and if the charger helped.

Issues to address it have been opened as well.

When I was implementing it, I was wondering if someone would ask.

The downloads are organized as follows: you download bz2 archive (the expected remaining part is shown as (D) in MBs) and write it to the disk unpacked (shown as W).

@seiichiro0185: I am glad it worked in other place. I'll work on implementation to make it possible to download it at your home as well.

Re: [Announce] Native offline maps: OSM Scout Server
 

I retried the download on charger, managed go download the map fine but the other data failed (several times). Will have another go this evening.

Re: [Announce] Native offline maps: OSM Scout Server
 

I'll work on fixing those broken downloads as the first priority. @Karry already gave a great suggestion on an aspect of its implementation, now just have to work on it.

Re: [Announce] Native offline maps: OSM Scout Server
 

@Feathers McGraw, Would you mind to test downloading with RPM provided at

https://mega.nz/#F!cdcEDbpb!i4A1GwKM_TXeRYBaahYgAg

It should resume downloads for few times (you would see RANGE messages in terminal if you start it over there). After testing and (hopefully fully) downloading, remove this RPM and install the one as in OpenRepos. This RPM has debug messages in it and not compiled in release mode.

If its all fine, I'll hook messages into GUI, merge translations that are ready and release 0.7.1. If it fails, then would be great to know how (as much as you know what to tell)

Re: [Announce] Native offline maps: OSM Scout Server
 

I will give it a try as soon as I can, but I'm moving house tomorrow and unfortunately I won't have internet access for a week afterwards! Any other volunteers available this week?

Re: [Announce] Native offline maps: OSM Scout Server
 

Thanks for giving a timeline. I'll continue with hooking it into GUI and would release based on my testing unless some negative feedback would come in.

My tests so far have been promising, but the conditions could be different in other places.

Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus I try this version. I don't know how to make better maps for Russia. For ex., I have interest for two regions(from 85), but for this I should to download two districts(from 8)... It's difference in space.