maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Applications (https://talk.maemo.org/forumdisplay.php?f=41)
-   -   [SFOS] [Announce] Native offline maps: OSM Scout Server (https://talk.maemo.org/showthread.php?t=97823)

rinigus 2018-09-09 13:09

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by peterleinchen (Post 1548220)
@rinigus
this is a very special beg (so please feel free to ignore)

I have an exfat formatted SD card (I know, I know ;)) and (now) around 20G of maps data.
It works quite well with exfat-fuse drivers.

But after having increased the size of datasets from only Germany to Europe I experience long slowdowns at start and also crashes.

It is always for gz files of valhalla.
As I have seen lots of CPU (more than 100%) for harbour-osm-scout-server and mount.exfat I assume timeout of boost::filesystem actions.
If it is really time out would it be possible to extend that for me?
I know boost as a C++ library and found occurences only in mapnik*.so library. So maybe it is possible to check?

And as I fiddled with terminal I got also some other warnings of 'binding loop detected). Please see the screenshot (fingerterm sometimes kind of 'freezes' and I could not copy/paste to text).

I don't have whole Europe on my SD Card, so maybe its some scaling issue. Also, I use ext4 as a FS for SDCard.

Boost is used by Valhalla which is linked into the main program now. From the error messages, I don't see any timeouts, but some interrupted call. Why was it interrupted, I don't know.

Valhalla lib is in major rewrite right now by its devs. As a part of it, they plan to get rid of many dependencies, boost::filesystem included. Maybe that will help.

As for now, I suggest you to look realistically what do you want to use in the next 2 months. I suspect its not full EU. 2 months are from the fact that I am trying to update the maps once in 2 month period.

peterleinchen 2018-09-09 17:17

Re: [Announce] Native offline maps: OSM Scout Server
 
okay.

It was just a guess as I felt also slowdowns with bigger file operations. And could not think of any other reason as else exfat is stable.

I was used (times of Nokia maps) to have whole world on device. But at that times is was around 11G. Now with OpenMaps just Germany only is a few GBs. And I like to have a few countries I need to travel to always on device.

But I will (need to learn to) live with it ;)
Anyway, a real big thank you :)

rinigus 2018-09-09 17:52

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by peterleinchen (Post 1548230)
okay.

It was just a guess as I felt also slowdowns with bigger file operations. And could not think of any other reason as else exfat is stable.

I was used (times of Nokia maps) to have whole world on device. But at that times is was around 11G. Now with OpenMaps just Germany only is a few GBs. And I like to have a few countries I need to travel to always on device.

But I will (need to learn to) live with it ;)
Anyway, a real big thank you :)

Well, you could try ext4 as a FS. I don't know how efficient is the caching on Fuse-based FS and I would expect Valhalla to use it properly. Which makes a great use of free RAM, if its available.

When having many datasets, it could also slow down search for POIs and addresses, unless you choose to restrict search to one country / territory only.

pichlo 2018-09-09 20:51

Re: [Announce] Native offline maps: OSM Scout Server
 
Peter, have you tried keeping all maps but switching off using them all at once?
OSM Scout Server -> Settings -> Geocoder-NLP -> Search all available maps
When switched off, it enables a drop-down menu at the top of main OSM Scout Server's screen to select your currently used map.
Not sure if it makes any difference, though. The option's name suggests that it only applies for searching and I am sure rinivus would have mentioned it if he believed it was worth it, but... the map selection in the main screen does not mention any ifs or buts so I would try it :)

rinigus 2018-09-10 03:43

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by pichlo (Post 1548235)
Peter, have you tried keeping all maps but switching off using them all at once?
OSM Scout Server -> Settings -> Geocoder-NLP -> Search all available maps
When switched off, it enables a drop-down menu at the top of main OSM Scout Server's screen to select your currently used map.
Not sure if it makes any difference, though. The option's name suggests that it only applies for searching and I am sure rinivus would have mentioned it if he believed it was worth it, but... the map selection in the main screen does not mention any ifs or buts so I would try it :)

It applies to the geocoder only. Other backends (Valhalla and Mapbox ) will be still serving all datasets. Although, I would expect that Valhalla is influenced more than others due to its use of large number of files

peterleinchen 2018-09-10 04:55

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1548240)
It applies to the geocoder only. Other backends (Valhalla and Mapbox ) will be still serving all datasets. Although, I would expect that Valhalla is influenced more than others due to its use of large number of files

Yep, unfortunately it seems only (or mostly) valhalla is affected.
I tried to set the search to one specific map only. But start time of scout server is a few minutes....

rinigus 2018-09-13 06:11

Re: [Announce] Native offline maps: OSM Scout Server
 
In the next few days, there will be an update in OSM Scout Server and its geocoder databases. This update will introduce some changes in the geocoder database format. As a result, it will be impossible to use new server with the old geocoder database and the old server with the new maps.

So, if you need new maps within a week or so, get them now. I will update the server at OpenRepos only after the new format has been pushed to the distribution servers. So, there maybe 12-hours gap when you cannot get new maps.

At present the changes are tested, consider it as an early warning.

rinigus 2018-09-16 05:54

Re: [Announce] Native offline maps: OSM Scout Server
 
New release is out: 1.12.0

In this release, long waited postal code support was introduced. As a result, geocoder database format has been extended and you have to update the datasets. Press "Check for updates" and proceed with the download. If datasets were up to date, only geocoder-nlp database will be updated.

Postal codes are imported from OSM and, for UK, from https://www.freemaptools.com/. Extra postal codes were reversed geocoded on import to figure out admin hierarchy of them (which part of UK they come from).

When searching with postal codes, you could either specify the code alone or together with some part of hierarchy. When postal code is specified together with address parts, its searching for records that have both - postal codes AND address - components in the record.

Postal codes should be detected by libpostal automatically. For some UK tests that I did, I could see that it was not working. This could be due to older training dataset or its incompleteness from the beginning. Unfortunately, libpostal development has been slow these days, but maybe, together with other users, we can fix few bugs and figure out what to do with the training data.

Primitive parser supports entry of postal codes using prefix "post:" . NB! IT'S CaSe SenSITIVE. So, for example:

post: AA BB CC , city

would be parsed with city as a part of admin and AA BB CC as a postal code. For primitive parser, postal codes can be entered anywhere in the string, just keep that part separated by commas. Documentation at https://rinigus.github.io/osmscout-server/en/ covers this as well.

In addition to postal codes, geocoder now returns phone number and website, if these data were at OSM. Pure Maps will be returning it in the next release, modRana has to be patched. All API is backward compatible, so it works already, just these components are not shown to the users.

I have optimized a bit further loading of libpostal for users of primitive parser only. Would be interesting to see, if @pichlo or others on J1 would get search results faster.

Translations have been updated - great work by all translators, thank you!

rinigus 2018-09-17 18:29

Re: [Announce] Native offline maps: OSM Scout Server
 
As mentioned yesterday, I published new Valhalla's datasets with public transport data. These are available now as an update of Valhalla's datasets via Map Manager of OSM Scout Server. The server does support it already, so no extra updates of the software is needed. Pure Maps supports public transport routing from the version 1.5.1. To add support for modRana, the costing option has to be added (see Pure Maps code for reference).

The data originates from https://transit.land/. Its a pain to download, took about 2 weeks of running one of Valhalla's programs - all due to slow server on their side. This project is one of the projects by Mapzen and, as far as I understand, is continued by volunteers. Hence the slow data feed, I presume.

Public transit is also not the priority of Valhalla's team at present, according to one of the issues in Valhalla's repo.

So, don't expect miracles, but you are welcome to test. We can still discuss issues over here and file them under Valhalla's repos. Its my first import with these data and its possible that something went wrong while downloading. So, some of the issues could be from the data import too.

From testing in Tallinn, it looks to be using busses only and ignoring other means of transport. But maybe its better in other places.

Enjoy!

PamNor 2018-09-17 20:35

Re: [Announce] Native offline maps: OSM Scout Server
 
Testing navigating with public transport here i Oslo. It always results i logg error: Exception in Valhalla: Cannot reach destination - too far from a transit stop. Just for you information. I don't expect you to prioritize this. It seems you work very hard too solve other issues.:)


All times are GMT. The time now is 12:13.

vBulletin® Version 3.8.8