Re: [Announce] Native offline maps: OSM Scout Server
 

I've sent them an email, but it's indeed unlikely the will fix it during the weekend.

In any case, it's pretty strange the outages happen so often and especially before/during the weekend. One would thing they have found and fixed all the causes long ago during the previous outages.

Re: [Announce] Native offline maps: OSM Scout Server
 

Good thing about it is that it gives me initiative to write new code to make the server more robust and use multiple providers. I am on the case and, hopefully, all will go smoothly and I can test against failed data.modrana.org :)

1.3.0
 

New release is out: 1.3.0

It adds support for multiple map data download servers. I managed to test it while data.modrana.org was down, but now you would have to wait till you can test it too (modrana server is up). Download servers are ordered and, on each OSM Scout Server start, the downloads start from data.modrana.org first and, on failure, move to @bomo's server. If anyone would like to join the club, please let me know. The implementation should support as many servers as we could have, I think.

Next, Valhalla's delayed start has been fixed that, when OSM Scout Server was started by systemd in response to routing request, led to an empty first reply (No route found). Now, routing and everything else should work immediately.

Translations have been pulled from Transifex and all the updated texts for Mapbox GL support should be there now. Thank you, translators.

Finally, I have added donations link as well. The donations are appreciated and are for my coding and time I spend on this and other SFOS projects. The donations platform is Liberapay which has rather transparent approach allowing us all to see the current sum of donations and their distribution in time. I think it maybe interesting for other developers and users to see such stats.

Enjoy using 1.3.0. I am planning to work through Mapbox GL import and styles, if nothing else comes up in between.

Re: [Announce] Native offline maps: OSM Scout Server
 

I have followed this development for some time. Last week I thought, I give it a try.

First I had problems with downloading all information, once with the timeout, the others with spome unzip error. I think the 4th or 5th try got me alle information including map of germany. Is there a way to maybe download less information at a time?

Now I get errors when trying to use the server with both Modrana and Poor Maps. See attached image

Re: [Announce] Native offline maps: OSM Scout Server
 

Looks like the SQLite database used by Mapnik is damaged. Germany is rather difficult case, its huge. I would suggest to get just a part of Germany you are interested in. Start small and progress from that. Note that Mapnik/Valhalla/Geocoder-NLP as well as MapboxGL backends glue all databases on fly. So, you may as well get parts of Germany.

To proceed, I suggest to delete Germany and then get new dataset for smaller part. Please let me know whether it helped.

Note that unzipping errors also suggest that something went wrong.

Finally, during this week we had malfunctioning of data.modrana.org and that led to waiting for downloads, I imagine.

Re: [Announce] Native offline maps: OSM Scout Server
 

I just started with PoorMaps and OSM Scout Server and I like it! :D

In order to analyse my POI problem, I encountered the following issue:

The GitHub Readme says:
But if I enter this URL in Firefox on my Sailfish X while OSM Scout Server is running, all I get is: "Error while listing available POI types"
In OSM Scout Server Event-Log, I see: "Database is not open, cannot list POI types"

Test-environment:

• PoorMap can search for POI in OSM Scout Server
• I have downloaded the map: "Europe / Germany / Bayern" (with the "Default" profile)
• Languages is set to: "de"
• I use OSM Scout Server 1.3.0-1 from Jolla Store

Re: [Announce] Native offline maps: OSM Scout Server
 

Its a bug. I think you are the first user trying to access it, and this is after just a bit of more than a year of development. It has been broken for anyone not using the original libosmscout backend. To get the list for the default profile, you would have to open SQLite database that is in Maps folder, under geocoder-nlp/europe-germany-bayerngeonlp-primary.sqlite and run

open database:
in sqlite3 shell:
The names of the types are searched for using substring search. Before that, your query is normalized by libpostal. So, when you search for "ATM", its first normalized to "atm" and later would hit "amenity_atm".

I am working now on Mapbox GL styles/import, when its finished I was planning to look into geocoding and update libpostal to newer version. But that may take some time - I may have to fix some yet unknown bugs of Mapbox GL widget before.

Re: [Announce] Native offline maps: OSM Scout Server
 

Thanks for the workaround.

No need to hurry. I do not urgently need this functionality. I just stumbled upon this issue while playing with the POI data.

----

Other question on the topic POI:

Do you do some kind of tag merging, when creating the OSM Scout Server database? In OsmAnd I can search for POI with tag "amenity _library" or "amenity_police". In OSM Scout Server database libraries and police stations in my hometown can be found, but only by their name. And the type is listed by PoorMaps as "Building".

In my opinion, it would be helpful, to be able to explicitly search for "libraries" or "police stations" by tag, since if I am new in town, I don't know the name of the library. What do you think about this? Should I file a github issue?

Re: [Announce] Native offline maps: OSM Scout Server
 

Yes, please file the issue. Since its geocoder component, please file it at https://github.com/rinigus/geocoder-nlp

As far as I remember, types are merged in the import and assigned as only one for POI. Hence the problem.

Keep digging into POIs and other components. When designing geocoder, I was mainly focused on getting results by name since that was really poor. Now it's time to look into search by type as in nearby search and make it work well. I suspect that the database will have to be redesigned, but let's see how.

Re: [Announce] Native offline maps: OSM Scout Server
 

Done: https://github.com/rinigus/geocoder-nlp/issues/36

Re: [Announce] Native offline maps: OSM Scout Server
 

I recently had another problem, finding "gas station" POIs in PoorMap, using OSM Scout Server as data source.

The problem is, that in OSM data "gas stations" have the tag name "amenity : fuel" so the search text "gas station" is not matching.

More details at: https://github.com/otsaloma/poor-maps/issues/59

@rinigus:
Do you think, this is a problem which could be and should be fixed by OSM Scout Server (or Geocoder-NLP)? If yes, I would create a separate Github issue.

Re: [Announce] Native offline maps: OSM Scout Server
 

Sure, its not a problem of Poor Maps, since Poor Maps just asks for data from data provider. In this case, it will run down to geocoder-nlp - please add the issue there.

Re: [Announce] Native offline maps: OSM Scout Server
 

Just a short update - I am preparing an update to the maps. At present, the last component - Mapbox GL tiles - is generated and when ready I will distribute them. This would bring the data to the level of OSM just before Christmas.

With the addition of Mapbox GL the imports do take significantly longer and I probably would have to optimize it further. Let's see after that whether once a month reimports are feasible or we have to move to once in two months schedule.

I have reworked the styles for Mapbox GL and, if there is interest, could try to look into how to make editing of them easier. That's in case if some of you would like to participate in making the styles.

As for timeline, the map import will probably take few nights, maybe a week.

Re: [Announce] Native offline maps: OSM Scout Server
 

All maps are imported and I am going to upload the maps this weekend. Since Mapbox GL tile schema has changed, I changed the version of the tiles as well. The new Mapbox GL maps will be compatible with the next version of OSM Scout Server (planned to be released this weekend as well), not the current one.

So, there could be some disturbance in the service this weekend due to non-perfect sync with software and data publishing.

Re: [Announce] Native offline maps: OSM Scout Server
 

New version of OSM Scout Server (1.4.0) and updated maps are out.

For those using Jolla Harbour, you can start using new maps only after Jolla Harbour version is updated as well.

Main work was done not on the server side itself, but on optimization of Mapbox GL tiles import, adjustments of the tile schema, and generation of map styles. On server side, I mainly updated documentation by adding sections on running the server and what does automatic activation means in terms of file generations at the home directory (asked during Jolla Harbour review). Thanks to the translators for updated strings, I included the version available this morning.

Now to give you an idea of what's going on with the implementation and import of the tiles, I describe it a bit below. While we use Mapbox GL Native as a library, lots of work done by Mapbox on styling is not available for use in external projects (when you want to use it independently from Mapbox service). Which is fine since they do have to get income stream as well, but requires for our project to make new styles and generate tiles. Fortunately, OpenMapTiles project has been working on tiles import and schema, but there are several problems with them which I have been fixing during several months:

* Very long time required to import - 20+ days (24h) of server time
* Problems with POI data (icons were frequently missing and its hard to debug)
* Several other problems as well, some of them I don't even recall now.

Now the import time was a major issue. Import works by first pushing all data from PBF (Planet.PBF) to PostGIS database and later generation of tiles via Mapnik and tilelive-copy. Pushing data into PostGIS takes about 8 hours of intensive CPU use and later ~12+ hours by some threads to reorganize the database. As such, this part is not a problem and has rather reasonable timeframe. Now, when I used OpenMapTiles scripts, there about 20 days of import were estimated.

By default, as usually used, the tilelive-copy (main import command generating tiles) doesn't use all available CPUs for several reasons. Since for OSM Scout Server we need tiles split into smaller databases covering parts of the World, it was simple to make parallel imports and write Makefiles that pushed the server to full usage.

Next, I had to optimize queries, pre-generate as many of them as possible to avoid repetition induced by the same queries run for tiles at different zoom levels, and PostGIS data storage layout.

All these optimizations resulted in dropping tile generation time (after PostGIS import) to about 4*24h, probably a bit less. I expect to get it somewhat faster after realizing that probably 1-2 days were slowed down significantly by huge ice polygons in Alaska, Canada, and Greenland. One way to reduce the polygons would be to run as many imports as possible leading to the increase in overall ambient temperature in these regions, but a faster way would be to split the polygins into the smaller ones. Splitting of the polygons will be tested the next time. My scripts are available at https://github.com/rinigus/mapbox-gl-importer .


In addition to imports, the styles have been reworked. For Mapbox GL, styles are described using a single JSON file which defines all the layers one-by-one. There are nice GUI utilities to work with the style file, but they all suffer from rather large amount of repetitive work. Namely, many layers use the same colors, similar logic, and so on. As a result, there is a lot of point-and-click or emacs/vi replace if you want to make different colorscheme (light and dark), for example.

I have setup a project for the styles used by OSM Scout Server that tries to address some of the repetitive work in describing the styles and modifying them: https://github.com/rinigus/mapbox-gl-styles . The styles are described by collection of layers (https://github.com/rinigus/mapbox-gl...les/src/layers) that are used by all styles, style-specific variables (example for https://github.com/rinigus/mapbox-gl...variables.less) allowing to use LESS color transformations, and the list of layers used by specific style. Thus, for example, style with English names is made by trivial modification of variables file (but that's easy to do with emacs/vi as well by running replace on JSON) and dark style is made by changing colors only.

In future, when Mapbox GL Native will start supporting expressions better, it will be possible to describe style layers with less repetitions than its done now, but that would take some time till Mapbox GL Native will get it implemented.

If you wish to edit styles for OSM Scout Server, let me know. On significant interest, I will try to address the simplicity of visual feedback and providing tiles. At present, I am using an editor, OSM Scout Server on Linux PC, and Mapbox GL QML widget based viewer on Linux. Its surely possible to replace the viewer with web-based one (Mapbox GL JS) and probably make it easier to get tiles for testing.

At present, I included osmbright and osmbright-en as OSM Scout Server styles, with the dark styles expected in the future releases. Plan is to get to parity with Mapnik styles.

Hmm, that turned out to be rather long description of what has been done. Sorry :)

Enjoy using the server and updated maps.

Re: [Announce] Native offline maps: OSM Scout Server
 

At first, thanks again for your continued work on osmscout server and the surrounding infrastructure.

I updated to the latest 1.4.0 and downloaded the new maps for germany on my tablet. Unfortunately I have a Problem using it together with Poor Maps GL. In higher zoom levels all text on the maps (like streetnames and house numbers) and POI icons are somehow missing. with 1.3.0 and fitting maps it was working fine.

Does anyone else see this problem or did I miss something?

Re: [Announce] Native offline maps: OSM Scout Server
 

same issues, me too. xperia x 5121

Re: [Announce] Native offline maps: OSM Scout Server
 

Just got reports regarding Spain (https://github.com/rinigus/osmscout-server/issues/233). I'll try to download Spain/Germany and see if its the same for me.

However, try to close Poor Maps GL and delete its cache via

and try again. I suspect that maybe caching could influence it. Please let me know if it helped.

Edit Just tried with Barcelona and all seems to be fine. Just keep zooming in and you should get street names. Although, most probably cache cleaning is needed for it (I was doing it routinely)

Re: [Announce] Native offline maps: OSM Scout Server
 

For Spain it just got confirmed - clearing cache helped and all works as expected.

So, please remove your application cache when Mapbox GL schema is changed. Right now it applies to Poor Maps GL and Laufhelden

Re: [Announce] Native offline maps: OSM Scout Server
 

Clearing the Poor Maps GL cache helped. It's now working as expected.

Re: [Announce] Native offline maps: OSM Scout Server
 

I have just published a point update release: 1.4.1

In addition to updated translations, this release brings support for multiple Mapbox GL icon packs and is bundled with the dark Mapbox GL style (mc). Mapbox GL styles are under styles/mapboxgl/styles of the server source tree.

I will try to keep parity between light and dark styles allowing map client applications to switch automatically between map styles as for day/night maps. For example, Laufhelden already has day/night theme switch implemented which can be extended to the map as well.

Re: [Announce] Native offline maps: OSM Scout Server
 

I am considering a modification in the import scripts to avoid getting too large databases in Mapnik. In particular, Germany, France, and maybe few more countries that

* are huge in terms of OSM data
* have sub-regions defined

will be made available only through sub-regions. All recommended backends are able to "glue" on fly the downloaded maps allowing users to download all the regions of Germany/France if there is such interest.

From my side, I will probably stop getting questions on why Germany/France maps don't show up with the logs showing broken SQLite database messages and me suspecting that its due to filesystem and/or sqlite driver limitations.

So, if there will be no protests to this plan, I might as well do it. However, if it turns out that many of you use these bigger datasets, we may continue as it is as well.

Note that, at this stage, its a proposal, not decided strategy

Re: [Announce] Native offline maps: OSM Scout Server
 

HI Ringius on Jolla1 it is working with parts of the maps, but Poormaps take ages to rendering the maps especially for routing and searching.
But when there is no other possibility for me its okay.

Re: [Announce] Native offline maps: OSM Scout Server
 

rendering, routing, and searching are actually 3 different parts. They could be all 3 slow or one of them in particular. While rendered tiles are cached between Poor Maps sessions, the search and routing is not. So, at least for rendering, it is using cache.

To test which of the components is the slowest, try to mix OSM Scout Server with online providers. For example, take Mapbox for tiles and try to search. That would tell you whether search is slow.

With the routing, try to route between current location and a location that has been found not as a search result but pointed on a map. That will tell you whether routing is slow.

Not sure I can fix it all, but maybe some things I can.

Re: [Announce] Native offline maps: OSM Scout Server
 

I currently use the combined german database (although with the MapBox GL backend, but I guess it would be affected too), but it wouldn't be a big problem to just download all the different regions, which should work the same form the clients (PoorMaps GL) point of view if I understand correctly.

Re: [Announce] Native offline maps: OSM Scout Server
 

If you use Mapbox GL, then that is split already by tiles. Same goes for Valhalla routing. The only difference would be Geocoder-NLP which then searches in separate databases. And that's what it is doing if you keep the default settings.

Re: [Announce] Native offline maps: OSM Scout Server
 

I am happy to announce a point update (1.4.3) which includes updated translations and fixed a bug in Mapbox GL style leading to visible rivers in the new version.

What was supposed to be a simple update, turned out to be nothing like it. With the deprecation of libsystemd-daemon library and integration of its functionality to libsystemd, I suddenly had to link to full-blown systemd lib. Turned out, that Harbour wasn't very happy about it. After spending few nights on trying to get around it, I decided to stop distribution of OSM Scout Server via Harbour. This allows me to explicitly request systemd dependency in the package and focus on developing software instead of running around and avoiding Harbour barriers.

I have opened corresponding issue at https://github.com/sailfishos/sdk-ha...tor/issues/102 and the fix doesn't seem to be expected in the near future.

Thus, please use OpenRepos or OBS for getting the last versions.

I have been busy lately with other projects and couldn't work much on the server (trying to help @kimmoli with Onyx SFOS port and working on Presage keyboard with @martonmiklos @ljo). As soon as we get the keyboard into a good shape, I'll continue server development and work on resolving the outstanding issues, such as search for POIs without names and others.

Re: [Announce] Native offline maps: OSM Scout Server
 

This is really sad - it's already really hard to write non trivial native applications and apparently not well though out changes like this don't make this any better.

In any case, huge thanks for doing all this, including all the ridiculous workarounds and all the extra work you have to do to get what should be basic platform functionality. I've seen also many of your emails to sailfish-devel pointing to all these issues and I'm note sure I remember even a single case of Jolla actually doing the changes or fixing the issue you pointed out - that really does not look good for them & for Sailfish OS overall...

So thanks a lot, I really mean it! :)

And on a more positive note, we now finally have something like a roadmap & new update, so let's hope it's a sign Jolla might finally start to work on all the critical things that have been ignored for much too long.

I'll hopefully should be now (after a major conference & the obligatory February-plague) able to start in earnest on MapBoxGL integration in modRana and all the other important improvements that really need to happen. :)

Re: [Announce] Native offline maps: OSM Scout Server
 

@MartinK, I don't think it is very sad, just natural evolution. Jolla's store is just one of the channels and I am planning to distribute the work through the remaining ones. Thanks for support!

PS: Actually, they did add SQLite as a possible dependency. So, there was a progress. But, in the end, each store has their policies and we should just distribute via the ones that are compatible.

Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus, do you have the data generation pipeline documented somewhere? I'm interested in particular (/only) in generating the valhalla routing datasets.
Do you do it from raw OSM pbf?

Re: [Announce] Native offline maps: OSM Scout Server
 

Sure, see scripts and README at https://github.com/rinigus/osmscout-...scripts/import . Valhalla is quite simple, you'll have to install it on Linux PC and use valhalla tools for import. See my scripts on how to do it.

As for data, you would need OSM PBF as a source. Valhalla will need full region that you want to cover. In my case, it's Planet.pbf and it does need quite some RAM (running on server with 256gb, but probability smaller should do). But smaller regions should be fine with smaller requirements.

I was planning to start new import this weekend, but it may take now longer than 48 hours to cover all backends.

Note that if you want to have integrated with OSM Scout Server , it's bit trickier since data is now obtained from online servers. So, you may have to manage local copies outside of the provided data manager .

Let me know if you run into any problems

Re: [Announce] Native offline maps: OSM Scout Server
 

Just to let you know: The updated maps are out on data.modrana.org. Enjoy!

Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus, many thanks :-)

Re: [Announce] Native offline maps: OSM Scout Server
 

I am using version 1.4.3-10.29.1.jolla of OSMScout (updated from older versions).

https://www.dropbox.com/s/5q2itncznm...1_003.png?dl=0

Re: [Announce] Native offline maps: OSM Scout Server
 

This is very strange. I have expanded the error message in a new version. Please,

* uninstall current OSM Scout Server,
* check out that there are no files in /usr/share/harbour-osmscout-server . if there are any, delete the folder
* install server from http://repo.merproject.org/obs/home:...la.armv7hl.rpm . If you need i486, install the same from i486 repo.

Please report back with the error screenshot, if it occurs.

Re: [Announce] Native offline maps: OSM Scout Server
 

ok thanks, I am re-installing packages

Re: [Announce] Native offline maps: OSM Scout Server
 

I've re-installed OSMScout and new maps, there are no errors or warnings, however the map layer on WhoGo maps is blank.

https://www.dropbox.com/s/ijhf6ksxxf...1_004.png?dl=0

https://www.dropbox.com/s/xp2gk5qqyk...1_005.png?dl=0

Re: [Announce] Native offline maps: OSM Scout Server
 

That's better. Do you have a profile with vector tiles enabled? If not, please do so and download maps. I hope this will resolve the issue

Re: [Announce] Native offline maps: OSM Scout Server
 

I thought the default profile was including vector tiles?

Re: [Announce] Native offline maps: OSM Scout Server
 

No, not yet. Please change it to the one with vector tiles