Active Topics

 



Notices
Page 5 of 84 « First 34 5 6715 Last »
Reply
Thread Tools
Fellfrosch's Avatar
Fellfrosch is offline Fellfrosch
2016-12-02 , 18:09
Posts: 1,092 | Thanked: 4,995 times | Joined on Dec 2009 @ beautiful cave
#41
Holy Moly, thanks for this really detailed explanations!

Of course it would be nice if maps would render faster, but anyway its so nice to have finally native offline maps!
 

The Following 3 Users Say Thank You to Fellfrosch For This Useful Post:
MartinK is offline MartinK
2016-12-02 , 18:57
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#42
Originally Posted by rinigus View Post
I must say that I use modRana on PC and Poor Maps on the device so far. So, I haven't tested modRana too much on device.

Assuming that you use the latest version of the server (there could be different problems with the versions earlier than 0.3)

* you could be hitting timeouts for connections in modRana. As a result, modRana might ask multiple times for the same tiles. To check if its true, look into the events log shown in the server's main screen. Check whether the same URL is loaded multiple times.
Maybe it could be also related to the old bundled urrlib3 (issue 146) ? I've updated it in the master branch, but it is not (yet!) in any released version. So users packages from OpenRepos & Jolla Store still run with the old urrlib3 and might be getting those errors.

That should hopefully not be an issue soon as I plan to release updated modRana packages today or tomorrow.

Originally Posted by rinigus View Post
* its probable that modRana and Poor Maps use different tile sizes. This is visible in URLs requested to render the tiles. Notice if there is a difference in "shift" and "scale" parameters in URL. Poor Maps uses by default 1024x1024 or 512x512 tiles that are not very well supported by modRana (or not at all yet) with modRana using 256x256 tiles (shift=0 and scale=1 by default). As a result, tile rendering can be done in bigger chunks in Poor Maps which should produce faster overall rendering. Optimization of tile size vs parallel rendering is not a simple problem and it may vary between the devices.
Map tile scaling is unfortunately currently broken in modRana - and even if it worked, it is at present only meant to upscale current 256x256 pixel tiles - mainly to make text and small map features easier to read (at the cost of making them more blurry). The map scaling code (once fixed) should be relatively easy to adapt to supporting larger tile sizes - even though there might be some issues with overlays to overcome when some layers provide bigger tiles and others not.

Originally Posted by rinigus View Post
* maybe modRana's rendering is slower. If its true you could try to follow when the server states "Idle" on its cover and see when the map is rendered in reality. For that, browse the map in Poor Maps or modRana and peak by sliding from the right on the state of the server. In Poor Maps delay is minimal if any, in modRana - I haven't checked.
You can also check the modRana debug logs: Go to Options -> Debug, enable the tile related debug logging & logging to a file and then restart modRana. On next start modRana should create a log file in ~/Documents/modrana_debug_logs/

Originally Posted by rinigus View Post
The main drawback of smaller maps is that you would have to change them in the server's settings.
This is something I have been wondering about - is that really needed ? Couldn't OSM Scout Server check incoming tile requests against the bounding boxes/shapefiles of multiple data packs and return data for the first match ? Or, depending on cost, just run the query on one pack after another until there is a hit ?

Some heuristic can be applied for this to run all further queries after a hit on the same data pack until there is another miss. Users could also enable just some packs in the OSM Scout Server UI to reduce the number of lookups.

Also a related idea - client side pack specification. OSM Scout Server clients could specify data pack name for the lookup. They could get the available pack names either from OSM Scout Server via a special query or by other means (listing all folders the OSM Scout Server data folder or even by downloading data packs themselves and pointing OSM scout to them).

Just a couple of ideas.

Originally Posted by rinigus View Post
Also, you cannot calculate the routes between maps.
Yeah, that's certainly not an easy problem to solve. I wonder how the commercial/proprietary navigation systems handle that ? Or do they just have larger areas (whole Europe, etc.) so they don't have to care about it ?

BTW, a naive idea how inter pack routing might be done:

0. assuming two adjacent packs with a relatively straight border between them & some overlap + simple start-destination routing
1. draw a straight line from start to destination
2. note where it intersects the border
3. search for suitable (way) nodes around the intersection points in both map-packs
4. find a node near the border that is in both packs
5. compute two routes: start - shared border node, shared border node - destination
6. return combined route as the result

This is of course very simplified and would probably returns some pretty inefficient routes. It also does not handle complex border geometries, routes over multiple packs, non overlapping packs or having only packs that have no shared border at all.

Again - just a couple of ideas.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
 

The Following 5 Users Say Thank You to MartinK For This Useful Post:
rinigus is offline rinigus
2016-12-02 , 21:02
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#43
Originally Posted by MartinK View Post
Maybe it could be also related to the old bundled urrlib3 (issue 146) ? I've updated it in the master branch, but it is not (yet!) in any released version. So users packages from OpenRepos & Jolla Store still run with the old urrlib3 and might be getting those errors.

That should hopefully not be an issue soon as I plan to release updated modRana packages today or tomorrow.
I already forgot about this bug, thank you for reminding about it! Yes, that's probably the reason behind the slower response. It got exposed when I switched from Qt-based HTTP server to microhttpd and is fixed in modRana's master branch. As far as I remember, the symptoms were very similar to what was described.

I am pretty confident that this is a reason for significant slow-down. We just have to wait 1-2 days to test it


Originally Posted by MartinK View Post
This is something I have been wondering about - is that really needed ? Couldn't OSM Scout Server check incoming tile requests against the bounding boxes/shapefiles of multiple data packs and return data for the first match ? Or, depending on cost, just run the query on one pack after another until there is a hit ?

Some heuristic can be applied for this to run all further queries after a hit on the same data pack until there is another miss. Users could also enable just some packs in the OSM Scout Server UI to reduce the number of lookups.
Something like this has been implemented by Lukas (@Karry) in his OSM Scout application, if I understand correctly. There are some significant hurdles to overcome for server if we want to do that.

Server renders the map in as many threads as you have CPUs. For that, it actually shares database access. If we want to render in parallel, we may end up having the same number of databases open. I think this may end up with too much RAM consumed by the server. So, we could start writing in some logic and lock/unlock threads depending on the accessed databases. Its all possible, but not trivial.

Alternative is to limit to a single thread and render one tile in a time. This may slow down it too much taking into account that the mobile devices may start lowering CPU frequency and switching off some cores when the load is about 1.

We can return to it a bit later. Maybe Lukas will implement it that way that it will be exposed via libosmscout anyway :-). Ideally, it would be solved on the library level and all applications using the library would benefit from it (server included).

We can try to formulate it as an issue and see when and how to resolve it.

Originally Posted by MartinK View Post
Also a related idea - client side pack specification. OSM Scout Server clients could specify data pack name for the lookup. They could get the available pack names either from OSM Scout Server via a special query or by other means (listing all folders the OSM Scout Server data folder or even by downloading data packs themselves and pointing OSM scout to them).
I was thinking at one point to add database and the stylesheet as a part of URL. Its possible - we just get the same problems as I was pointing out above.

I would suggest to get other issues solved first: better GUI, search, rendering, and routing come up into my mind on server side. Better routing support on the client part (voice commands?).

Originally Posted by MartinK View Post
Yeah, that's certainly not an easy problem to solve. I wonder how the commercial/proprietary navigation systems handle that ? Or do they just have larger areas (whole Europe, etc.) so they don't have to care about it ?
Routing between maps is possible. Many have solved it with what seems to be separate data packages: OSMAnd, MAPS.ME for example. From Valhalla's docs and few test runs, it seems that they also partition internally all the maps. So, its possible, just have to work on it. Corresponding issue is https://github.com/Framstag/libosmscout/issues/57

Originally Posted by MartinK View Post
Again - just a couple of ideas.
Keep them coming. Let's just hope that we can implement at least a fraction of them.
 

The Following 3 Users Say Thank You to rinigus For This Useful Post:
MartinK is offline MartinK
2016-12-02 , 21:49
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#44
Originally Posted by rinigus View Post
We can return to it a bit later. Maybe Lukas will implement it that way that it will be exposed via libosmscout anyway :-). Ideally, it would be solved on the library level and all applications using the library would benefit from it (server included).
Yep, that definitely seems like the most sensible way.

Originally Posted by rinigus View Post
I would suggest to get other issues solved first: better GUI, search, rendering, and routing come up into my mind on server side. Better routing support on the client part (voice commands?).
A significant usability issue is also IMHO the constantly changing data format - if it was more stable there could be data repositories users could use instead of having to generate their own data packs. It might be even possible to download the data directly from the GUI (from the OSM Scout Server UI or via clients).

I might be even able to arrange something with the Masaryk University NLP lab which is already kindly hosting the 100+ GB of Monav offline routing data modRana can make use of (currently mainly on the N900).

But at the moment people might even have to re-generate their data files even due to am OSM Scout Server updates due to using a newer version of libosmscout.

Does it look like there might also be a solution for this issue ? I understand designing and supporting an unchanging binary data format is no easy task for libosmscout developers.

But at the same time I'm afraid the current unstable-data format can block many users from using OSM Scout Server & client apps - especially less advanced users who could make use of pre-generated map data.

Originally Posted by rinigus View Post
Routing between maps is possible. Many have solved it with what seems to be separate data packages: OSMAnd, MAPS.ME for example. From Valhalla's docs and few test runs, it seems that they also partition internally all the maps. So, its possible, just have to work on it. Corresponding issue is https://github.com/Framstag/libosmscout/issues/57
Sounds promising!

Originally Posted by rinigus View Post
Keep them coming. Let's just hope that we can implement at least a fraction of them.
Yeah, that's always the biggest problem!
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
 

The Following 3 Users Say Thank You to MartinK For This Useful Post:
rinigus is offline rinigus
2016-12-02 , 22:06
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#45
Originally Posted by MartinK View Post
A significant usability issue is also IMHO the constantly changing data format - if it was more stable there could be data repositories users could use instead of having to generate their own data packs. It might be even possible to download the data directly from the GUI (from the OSM Scout Server UI or via clients).

I might be even able to arrange something with the Masaryk University NLP lab which is already kindly hosting the 100+ GB of Monav offline routing data modRana can make use of (currently mainly on the N900).

But at the moment people might even have to re-generate their data files even due to am OSM Scout Server updates due to using a newer version of libosmscout.

Does it look like there might also be a solution for this issue ? I understand designing and supporting an unchanging binary data format is no easy task for libosmscout developers.

But at the same time I'm afraid the current unstable-data format can block many users from using OSM Scout Server & client apps - especially less advanced users who could make use of pre-generated map data.
The last time we had to regenerate the maps was due to the bug in map importer. As far as I remember, the format stayed the same - just data was imported wrongly. This bug was exposed during testing of the routing and I suspect that there maybe more bugs to come. Eventually, it will stabilize, we just have to accept some fluidity at this time.

To offset the map format changes I worked on the importers. Hopefully, it would be of help for users.

Its great that you have such a host - it could really become handy. Note that the maps do have to be regenerated relatively frequently. I presume that once a month should be a target. And then we would have to see how to split the maps - countries, regions, ... Maybe you should get in touch with Lukas. He was working on getting maps distributed via downloads and he may be interested in expanding on that part of the project.

But the supply of the maps to the users is, indeed, a major problem that we would have to resolve.
 

The Following 2 Users Say Thank You to rinigus For This Useful Post:
MartinK is offline MartinK
2016-12-02 , 23:37
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#46
Originally Posted by rinigus View Post
The last time we had to regenerate the maps was due to the bug in map importer. As far as I remember, the format stayed the same - just data was imported wrongly.
I for some reason though it was much more unstable, so it's good to hear it's not the case.

Originally Posted by rinigus View Post
This bug was exposed during testing of the routing and I suspect that there maybe more bugs to come. Eventually, it will stabilize, we just have to accept some fluidity at this time.
Sounds good.

Originally Posted by rinigus View Post
To offset the map format changes I worked on the importers. Hopefully, it would be of help for users.
I've tried the Windows importer a while ago and it seem to work fine - nice trick with drag-and-drop support on the import script.

Originally Posted by rinigus View Post
Its great that you have such a host - it could really become handy. Note that the maps do have to be regenerated relatively frequently. I presume that once a month should be a target.
That should be doable - I'm using one of the machines in the NLP compute cluster to generate the Monav data. It's a rather powerful computer (48 CPU cores & ~250 GB RAM) and can run a full update (download planet.osm.pbf, split to regional files, convert to Monav data, compress the results to tar.gz archives) in less than 24 hours. The conversion itself takes about 5-6 hours, the rest is "planet splitting".

BTW, the software used to generate the repository lives in the modrana-data-repository repository and it should not be that hard to extend it to generate also libosmscout-compatible data once the format is considered stable enough.

Originally Posted by rinigus View Post
And then we would have to see how to split the maps - countries, regions, ...
For the Monav data packs I've just used the same region definitions Geofabrik uses on their OSM data download page. They even have polygon files for each area - this is what I use to generate the regional *.osm.pbf files with osmosis from a fresh planet.osm.pbf file. No need to download all their PBF files every time the repository is updated.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
 

The Following 4 Users Say Thank You to MartinK For This Useful Post:
rinigus is offline rinigus
2016-12-03 , 11:49
Posts: 1,414 | Thanked: 7,547 times | Joined on Aug 2016 @ Estonia
#47
Originally Posted by MartinK View Post
That should be doable - I'm using one of the machines in the NLP compute cluster to generate the Monav data. It's a rather powerful computer (48 CPU cores & ~250 GB RAM) and can run a full update (download planet.osm.pbf, split to regional files, convert to Monav data, compress the results to tar.gz archives) in less than 24 hours. The conversion itself takes about 5-6 hours, the rest is "planet splitting".

BTW, the software used to generate the repository lives in the modrana-data-repository repository and it should not be that hard to extend it to generate also libosmscout-compatible data once the format is considered stable enough.

For the Monav data packs I've just used the same region definitions Geofabrik uses on their OSM data download page. They even have polygon files for each area - this is what I use to generate the regional *.osm.pbf files with osmosis from a fresh planet.osm.pbf file. No need to download all their PBF files every time the repository is updated.
It would be great if you could subscribe to https://lists.sourceforge.net/lists/...ut-development and introduce your options to distribute maps to the list. I'll mention this discussion over there as well. Lukas has been working on generating the maps for distribution and maybe you together could come up with some great solution. Through direct communication between you and Lukas we would ensure that your experience is not lost in translation
 

The Following 4 Users Say Thank You to rinigus For This Useful Post:
karry is offline karry
2016-12-04 , 18:42
Posts: 6 | Thanked: 43 times | Joined on Nov 2016 @ Prague
#48
Hi all. I read this thread completely and I have to say thanks to all developers for your work and to all people who test our applications!

@rinigus you should consider to submit server to Coding Competition :-) There are some interesting prices like Jolla C. Other apps are welcome too!

To map distribution, I just recap what I send to libosmscout mailing list: server part for OSM Scout is ready, scripts are available here. I host it on Rasperry PI with 1 TiB disk dedicated for maps. It is dwarf to compare to your machine @MartinK :-) I will focus on client side next days/weeks. It will be available for osmscout server too off course. When everything will be ready, we can start working on map delivery network (MDN ?). What is university status to your server Martin. It is officially supported or it is tolerated "punk activity"?
 

The Following 6 Users Say Thank You to karry For This Useful Post:
MartinK is offline MartinK
2016-12-04 , 20:16
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#49
Originally Posted by karry View Post
To map distribution, I just recap what I send to libosmscout mailing list: server part for OSM Scout is ready, scripts are available here. I host it on Rasperry PI with 1 TiB disk dedicated for maps. It is dwarf to compare to your machine @MartinK :-) I will focus on client side next days/weeks. It will be available for osmscout server too off course. When everything will be ready, we can start working on map delivery network (MDN ?).
Sounds good! We could try to create some simple osmscout data repo spec - for example how a JSON document describing available map packs looks like. Like this repos/repo mirrors would be independent from how they are created (your scripts, the modrana-data-repository generator, manual generation, etc.) and people will still be able to use them in a unified way from applications supporting the spec.

Originally Posted by karry View Post
What is university status to your server Martin. It is officially supported or it is tolerated "punk activity"?
My bachelors & masters thesis were led by a supervisor from the NLP lab. We are still in touch after I've finished my studies and the current Monav data repository (and the whole modrana.org website) is hosted by the lab with his & labs consent. And I do credit them for this where appropriate this.

I will of course first ask them if they are OK with generating & hosting another data set, but I kinda expect it should not be an issue. In comparison to some of the corpus processing they do our data sets are very very small.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
 

The Following 5 Users Say Thank You to MartinK For This Useful Post:
karry is offline karry
2016-12-11 , 00:41
Posts: 6 | Thanked: 43 times | Joined on Nov 2016 @ Prague
#50
Map downloader is ready for merge to upstream library: https://github.com/Framstag/libosmscout/pull/185

Now, Silica UI...
 

The Following 4 Users Say Thank You to karry For This Useful Post:
Reply
Page 5 of 84 « First 34 5 6715 Last »

Tags
geocoder, linux, offline maps, router, sailfish os, tiles

« Previous Thread | Next Thread »

 
Forum Jump


All times are GMT. The time now is 18:19.

Contact Us - Home - Archive - Top