Re: [Announce] Native offline maps: OSM Scout Server
 

Holy Moly, thanks for this really detailed explanations!

Of course it would be nice if maps would render faster, but anyway its so nice to have finally native offline maps!

Re: [Announce] Native offline maps: OSM Scout Server
 

Maybe it could be also related to the old bundled urrlib3 (issue 146) ? I've updated it in the master branch, but it is not (yet!) in any released version. So users packages from OpenRepos & Jolla Store still run with the old urrlib3 and might be getting those errors.

That should hopefully not be an issue soon as I plan to release updated modRana packages today or tomorrow. :)

Map tile scaling is unfortunately currently broken in modRana - and even if it worked, it is at present only meant to upscale current 256x256 pixel tiles - mainly to make text and small map features easier to read (at the cost of making them more blurry). The map scaling code (once fixed) should be relatively easy to adapt to supporting larger tile sizes - even though there might be some issues with overlays to overcome when some layers provide bigger tiles and others not.

You can also check the modRana debug logs: Go to Options -> Debug, enable the tile related debug logging & logging to a file and then restart modRana. On next start modRana should create a log file in ~/Documents/modrana_debug_logs/

This is something I have been wondering about - is that really needed ? Couldn't OSM Scout Server check incoming tile requests against the bounding boxes/shapefiles of multiple data packs and return data for the first match ? Or, depending on cost, just run the query on one pack after another until there is a hit ?

Some heuristic can be applied for this to run all further queries after a hit on the same data pack until there is another miss. Users could also enable just some packs in the OSM Scout Server UI to reduce the number of lookups.

Also a related idea - client side pack specification. OSM Scout Server clients could specify data pack name for the lookup. They could get the available pack names either from OSM Scout Server via a special query or by other means (listing all folders the OSM Scout Server data folder or even by downloading data packs themselves and pointing OSM scout to them).

Just a couple of ideas. :)

Yeah, that's certainly not an easy problem to solve. I wonder how the commercial/proprietary navigation systems handle that ? Or do they just have larger areas (whole Europe, etc.) so they don't have to care about it ?

BTW, a naive idea how inter pack routing might be done: :)

0. assuming two adjacent packs with a relatively straight border between them & some overlap + simple start-destination routing
1. draw a straight line from start to destination
2. note where it intersects the border
3. search for suitable (way) nodes around the intersection points in both map-packs
4. find a node near the border that is in both packs
5. compute two routes: start - shared border node, shared border node - destination
6. return combined route as the result

This is of course very simplified and would probably returns some pretty inefficient routes. It also does not handle complex border geometries, routes over multiple packs, non overlapping packs or having only packs that have no shared border at all. :)

Again - just a couple of ideas. :)

Re: [Announce] Native offline maps: OSM Scout Server
 

I already forgot about this bug, thank you for reminding about it! Yes, that's probably the reason behind the slower response. It got exposed when I switched from Qt-based HTTP server to microhttpd and is fixed in modRana's master branch. As far as I remember, the symptoms were very similar to what was described.

I am pretty confident that this is a reason for significant slow-down. We just have to wait 1-2 days to test it :)


Something like this has been implemented by Lukas (@Karry) in his OSM Scout application, if I understand correctly. There are some significant hurdles to overcome for server if we want to do that.

Server renders the map in as many threads as you have CPUs. For that, it actually shares database access. If we want to render in parallel, we may end up having the same number of databases open. I think this may end up with too much RAM consumed by the server. So, we could start writing in some logic and lock/unlock threads depending on the accessed databases. Its all possible, but not trivial.

Alternative is to limit to a single thread and render one tile in a time. This may slow down it too much taking into account that the mobile devices may start lowering CPU frequency and switching off some cores when the load is about 1.

We can return to it a bit later. Maybe Lukas will implement it that way that it will be exposed via libosmscout anyway :-). Ideally, it would be solved on the library level and all applications using the library would benefit from it (server included).

We can try to formulate it as an issue and see when and how to resolve it.

I was thinking at one point to add database and the stylesheet as a part of URL. Its possible - we just get the same problems as I was pointing out above.

I would suggest to get other issues solved first: better GUI, search, rendering, and routing come up into my mind on server side. Better routing support on the client part (voice commands?).

Routing between maps is possible. Many have solved it with what seems to be separate data packages: OSMAnd, MAPS.ME for example. From Valhalla's docs and few test runs, it seems that they also partition internally all the maps. So, its possible, just have to work on it. Corresponding issue is https://github.com/Framstag/libosmscout/issues/57

Keep them coming. Let's just hope that we can implement at least a fraction of them.

Re: [Announce] Native offline maps: OSM Scout Server
 

Yep, that definitely seems like the most sensible way. :)

A significant usability issue is also IMHO the constantly changing data format - if it was more stable there could be data repositories users could use instead of having to generate their own data packs. It might be even possible to download the data directly from the GUI (from the OSM Scout Server UI or via clients).

I might be even able to arrange something with the Masaryk University NLP lab which is already kindly hosting the 100+ GB of Monav offline routing data modRana can make use of (currently mainly on the N900).

But at the moment people might even have to re-generate their data files even due to am OSM Scout Server updates due to using a newer version of libosmscout.

Does it look like there might also be a solution for this issue ? I understand designing and supporting an unchanging binary data format is no easy task for libosmscout developers.

But at the same time I'm afraid the current unstable-data format can block many users from using OSM Scout Server & client apps - especially less advanced users who could make use of pre-generated map data.

Sounds promising! :)

Yeah, that's always the biggest problem! :D

Re: [Announce] Native offline maps: OSM Scout Server
 

The last time we had to regenerate the maps was due to the bug in map importer. As far as I remember, the format stayed the same - just data was imported wrongly. This bug was exposed during testing of the routing and I suspect that there maybe more bugs to come. Eventually, it will stabilize, we just have to accept some fluidity at this time.

To offset the map format changes I worked on the importers. Hopefully, it would be of help for users.

Its great that you have such a host - it could really become handy. Note that the maps do have to be regenerated relatively frequently. I presume that once a month should be a target. And then we would have to see how to split the maps - countries, regions, ... Maybe you should get in touch with Lukas. He was working on getting maps distributed via downloads and he may be interested in expanding on that part of the project.

But the supply of the maps to the users is, indeed, a major problem that we would have to resolve.

Re: [Announce] Native offline maps: OSM Scout Server
 

I for some reason though it was much more unstable, so it's good to hear it's not the case. :)

Sounds good. :)

I've tried the Windows importer a while ago and it seem to work fine - nice trick with drag-and-drop support on the import script. :)

That should be doable - I'm using one of the machines in the NLP compute cluster to generate the Monav data. It's a rather powerful computer (48 CPU cores & ~250 GB RAM) and can run a full update (download planet.osm.pbf, split to regional files, convert to Monav data, compress the results to tar.gz archives) in less than 24 hours. The conversion itself takes about 5-6 hours, the rest is "planet splitting".

BTW, the software used to generate the repository lives in the modrana-data-repository repository and it should not be that hard to extend it to generate also libosmscout-compatible data once the format is considered stable enough. :)

For the Monav data packs I've just used the same region definitions Geofabrik uses on their OSM data download page. They even have polygon files for each area - this is what I use to generate the regional *.osm.pbf files with osmosis from a fresh planet.osm.pbf file. No need to download all their PBF files every time the repository is updated. :)

Re: [Announce] Native offline maps: OSM Scout Server
 

It would be great if you could subscribe to https://lists.sourceforge.net/lists/...ut-development and introduce your options to distribute maps to the list. I'll mention this discussion over there as well. Lukas has been working on generating the maps for distribution and maybe you together could come up with some great solution. Through direct communication between you and Lukas we would ensure that your experience is not lost in translation :)

Re: [Announce] Native offline maps: OSM Scout Server
 

Hi all. I read this thread completely and I have to say thanks to all developers for your work and to all people who test our applications!

@rinigus you should consider to submit server to Coding Competition :-) There are some interesting prices like Jolla C. Other apps are welcome too!

To map distribution, I just recap what I send to libosmscout mailing list: server part for OSM Scout is ready, scripts are available here. I host it on Rasperry PI with 1 TiB disk dedicated for maps. It is dwarf to compare to your machine @MartinK :-) I will focus on client side next days/weeks. It will be available for osmscout server too off course. When everything will be ready, we can start working on map delivery network (MDN ?). What is university status to your server Martin. It is officially supported or it is tolerated "punk activity"?

Re: [Announce] Native offline maps: OSM Scout Server
 

Sounds good! We could try to create some simple osmscout data repo spec - for example how a JSON document describing available map packs looks like. Like this repos/repo mirrors would be independent from how they are created (your scripts, the modrana-data-repository generator, manual generation, etc.) and people will still be able to use them in a unified way from applications supporting the spec. :)

My bachelors & masters thesis were led by a supervisor from the NLP lab. We are still in touch after I've finished my studies and the current Monav data repository (and the whole modrana.org website) is hosted by the lab with his & labs consent. And I do credit them for this where appropriate this. :)

I will of course first ask them if they are OK with generating & hosting another data set, but I kinda expect it should not be an issue. In comparison to some of the corpus processing they do our data sets are very very small. :)

Re: [Announce] Native offline maps: OSM Scout Server
 

Map downloader is ready for merge to upstream library: https://github.com/Framstag/libosmscout/pull/185

Now, Silica UI...