Re: [Announce] Native offline maps: OSM Scout Server
 

+1, the work you (including every people involved - server/clients) are doing is unbelievable, you are basically almost bringing to Sailfish a native solution for the most voted TJC feature! :cool: (almost because no voice, but I personally don't need it). It can really help promoting Sailfish.

One thing I've noticed while trying it is that the search results are totally not relevant when libpostal is not enabled. Wouldn't it make sense to enable it by default and invite the users to chose their search language(s) on first start?

Re: [Announce] Native offline maps: OSM Scout Server
 

There are still few things missing on server side - search and routing between all maps, for example - but we'll get there. The voice navigation (for me its actually rerouting that's most interesting) would surely come. As far as I remember, these features are in Poor Maps and modRana issues (aka TODO) lists.

That's a good suggestion. In general, I probably would need to add a wizard of some sorts that would help to setup the server. I was reluctant to enable libpostal by default (same with Mapnik) as it requires about 700 MB extra storage. Now the new version of libpostal may reduce that requirement, but I would have to move to the new version and test it properly first.

I'll add your suggestion into the issues of the server and will take a look into it a touch later. Its a great idea and should be implemented when I polish the interface. Right now, I am trying to get all the required features covered first, if I can do it in reasonable amount of time.

Re: [Announce] Native offline maps: OSM Scout Server
 

Hello guys,

I am currently developing a sport tracker app for SFOS.
I would like to use OSM Scout Server for providing map informations but unfortunately it does not work.

This is what I tried:
The map from the server seems not to be found and instead is trying to use WIFI connection to download the map.

The server is running nicely and works with modRana and Poor Maps.

You can find the whole code here: https://github.com/jdrescher2006/Lau...edViewPage.qml

Re: [Announce] Native offline maps: OSM Scout Server
 

Nice to see it used in map-related applications and I can only hope that it would spread :) .

I presume that you have OSM Scout Server running on the background while using the tracker app, right?

From the look of it, you are using QML Map type. I don't have experience with it, unfortunately. Maybe @otsaloma can help, or reading his Poor Maps code which seems to be using the same plugin. From reading Poor Maps MapPlugin, it looks like @otsaloma is using own server that probably re-routes requests further. Notice that URL syntax seems to be different. But again, I never used it in QML myself, others are much more qualified to help you out.

On the server side, you could check out whether it is accessed. For that, in server settings, enable "Log info messages". Then, while accessing the map in your application, check whether requests arrive to the server (they should be shown in the main window under Events.

If it turns out that Map prefers some other URL scheme than the one used in the server currently, we can add some simplified URL scheme under

http://localhost:8553/v1/tile_simple/

or something similar. That's very easy.

Re: [Announce] Native offline maps: OSM Scout Server
 

SFOS still has QtLocation 5.2. That "osm.mapping.host" parameter was added in 5.6. Documentation for QtLocation 5.2 is not officially available, but maybe Ubuntu 15.04 might be close. But note that Jolla has backported individual commits and updated individual things like the HERE plugin, so it's understandably quite difficult to figure out.

What you're trying to do is (to my knowledge) simply not possible with the SFOS version of QtLocation. My advice would be to wait for QtLocation 5.6 (maybe we need a new community IRC pressure meeting?) or if you can't wait, ask Martin about his pure QML component. Poor Maps uses QtLocation's Map component, but has custom tile loading to work around stuff like this, but that's not designed to be reusable.

Re: [Announce] Native offline maps: OSM Scout Server
 

I must say that situation with QtLocation on SFOS is frustrating. Its probably a naive question but I am going to ask it anyway: now, when we have Qt5.6, is it possible to compile missing QtLocation 5.6 and add it ourself via OpenRepos or other distribution mechanism? @otsaloma, you have been following for a while, do you know whether it is possible?


To keep you in the loop and give an overview of the server development:

On the server side, I have been working on making it simple to use it with multiple maps. Rendering of the maps has been resolved already with Mapnik. Now I have added support for search using the data from all available maps. This is done for geocoder-nlp backend.

Finally, I think I know how to get routing through multiple maps supported as well. Namely, I am planning to incorporate Valhalla (routing engine behind Mapzen routing solution). After few days of porting, I managed to run the first routing tests using valhalla's program on SFOS device. Calculation of a route from Stockholm (Sweden) to Aarhus (Denmark) took just 2 seconds on Nexus 4. That's compared to ~45 seconds on the same device on the same route when using libosmscout. Not sure whether valhalla is always faster, but the start is promising. In addition to speed, valhalla routing engine allows us to "glue" the countries/territories together solving the long-standing issue of routing between countries.

There are several technical issues to be resolved with the adaptation of Valhalla, but it should be possible. Also we would have to discuss with @otsaloma and @MartinK few technical choices on how to do that when I know a bit more about Valhalla requirements.

I might make one more release before adding Valhalla to push out support for search in all maps and catch up with the advances in libosmscout.

Re: [Announce] Native offline maps: OSM Scout Server
 

I don't actually know if it's possible, but looking the commit activity, there's a lot of fixes related to e.g. geoclue positioning, and just using upstream 5.6 might not work well on all devices and trying to port those patches to 5.6 might be difficult. Additionally, there are API changes between 5.2 and 5.6, which means upgrading would break many apps written for 5.2. The API changes are small and it's easy to adapt to them, but it might be difficult for app authors to support both versions.

Also note, that this problem of using using a custom tile source, e.g. OSM Scout Server, is only partially served by the OSM plugin in QtLocation 5.6. It allows you to set the host, e.g. "http://tile.openstreetmap.org/" in "http://tile.openstreetmap.org/{z}/{x}/{y}.png", but the requested URL will always end in "{z}/{x}/{y}.png". What we'd really need is a plugin that supports a template URL string, so that you can have X, Y and Z as URL parameters instead of path components or additional parameters like day/night or API keys etc. And also, a plugin parameter for tile size so that the requested tiles can be shown correctly. A custom QtLocation plugin would be easy to distribute separately, e.g. at OpenRepos, but probably difficult to use in Harbour apps.

Re: [Announce] Native offline maps: OSM Scout Server
 

@otsaloma, thank you very much for this explanation! I don't know much about it, but would try to read up on QtLocation in future.

What's telling is that there has been no activity visible with QtLocation 5.6. From this maybe we can conclude that we should not expect situation change in a short term.

Re: [Announce] Native offline maps: OSM Scout Server
 

Yes, that's likely true. I get the impression that Jolla has been busy with Xperia and other adaptations, leaving that QtLocation upgrade for later.

Re: [Announce] Native offline maps: OSM Scout Server
 

Thanks for the infos.
I have a lot of features to implement so I can wait a little longer.

All in all for developers it's still a sad situation after so many years with SFOS. You can't even release a bluetooth app into the harbour.
I think Jolla should make it as easy as possible for developers. Instead they are doing it the other way around :mad::eek:

Re: [Announce] Native offline maps: OSM Scout Server
 

While its good to release in the Harbour, its rules should not be on our way and the releases in OpenRepos are very much appreciated as well.

In this case, as you see from the discussion above, it could be just manpower limitation on Jolla's part (in terms of not working on QtLocation). We could try to chip in as a community, if there is someone around who has knowledge and/or time to look into QtLocation 5.6 adaptation :)

Re: [Announce] Native offline maps: OSM Scout Server
 

Ok, that was a very good hint. Of course OpenRepos are fine with me. I already have an app which is released to openrepos only because of Jolla store restrictrions.
So I in my yaml file I had:
And changed it to:
But now there is an error while compiling:
Do I need to install this lib on the phone first?

Re: [Announce] Native offline maps: OSM Scout Server
 

Your requirements are correct. Unfortunately, as we discussed above, qt5-qtlocation >= 5.6.0 is not available yet on Sailfish. To my knowledge, no-one has compiled and (if needed) patched it. All we can do is either try to get it running ourselves or wait till Jolla gets developers on it.

Re: [Announce] Native offline maps: OSM Scout Server
 

I have just released a module for the OSM Scout Server via Harbour. It looks that through such modules I would be able to split the server package into parts and supply multiple executables when needed. Users would still have to run only the main executable to get full functionality of the server - OSM Scout Server.

Right now, the fonts module is released. It was done early to ensure its released before future upgrades. I am working on getting one more module out which would contain Valhalla's routing engine. At present, only these two modules are planned.

The modules will be released via Harbour and OpenRepos. Since the timing of OpenRepos is under my control, I don't have to rush them out early on that channel.

For users, that would mean installation of additional programs/apps via Harbour or OpenRepos. On server's start, the availability of the modules is checked and, if missing, installation is suggested.

I am planning to update the modules separately, as needed. I hope that this split will not create further confusion, at least not in long term.

Re: [Announce] Native offline maps: OSM Scout Server
 

I am planning to release a new version in few days. Among other developments, this version will have a libosmscout library updated to incorporate latest changes in the library. As a result, the new maps for libosmscout would be incompatible with the old versions of the server. The maps would be uploaded to the modRana.org probably tonight or tomorrow. So, if you depend on the maps and are planning a trip these days, be aware of possible disruption in the maps download service.

Re: [Announce] Native offline maps: OSM Scout Server
 

Thanks. It would be better to install the osmscout-server dependencies without icons rinigus?

Re: [Announce] Native offline maps: OSM Scout Server
 

There have been two main reasons for using a custom QML map component:
- lack of functionality & unconvincing performance of QtLocation
- QtLocation not being Jolla Store compatible and would thus be preventing modRana from being distributed via the Jolla store

Number one likely is less of an issue nowadays due to the ongoing work on QtLocation but it's quite a shame number two still stands after all those years and complicates application development if any mapping/navigation functionality is required.

In any case this is what modRana is using:
https://github.com/M4rtinK/modrana/b...l/PinchMap.qml
It's a QML/Javascript component for efficient display of tiled maps with overlay support. Feel free to use it if you want (it's GPLv3) but I have to note it's not fully standalone and you would likely have to also take the related modRana subsystems as well. For example map tile download and caching is done by a Python module, etc. I have tentative plans to make QML map component more standalone and more easy to reuse in other application by I haven't really got to that.

For that reason it might make sense to also take a look at the QML map component used by Foursail:
https://github.com/Maledictus/foursa...l/PinchMap.qml

It should be quite similar to the modRana PinchMap (both modRana and Frousail projects started with the old PoinchMap.qml from AGTL), a bit less advanced and possibly quite a bit more standalone. :)

Re: [Announce] Native offline maps: OSM Scout Server
 

I presume you are talking about modules, right?

It would be better if I could use normal linux-style dependencies, but there are limitations when we want to have apps in the Harbour. Another limitation is imposed by RPM (or more specifically, the way we install them).

1. If I want to publish via Harbour, I cannot have multiple executables in one package. I can add as many libraries as I want, as much data, but not executables. I maybe able to smuggle one as a zipped file and unpack on a start, but that becomes iffy from the security point of view.

Valhalla is provided as its own executable that is run by the server if needed (if user wants to choose it). In the end, this will allow me to keep Valhalla up-to-date with relatively small time demand. So, I needed to have a mechanism for delivering multiple executables.

2. If we forget about Harbour and go only through OpenRepos then I'll be able to install the modules as dependencies. However, while highly unlikely and extremely hypothetical, imagine, if someone wants to temporarily uninstall such a brilliant software as OSM Scout Server is. In this case, due to the lack of "apt-get autoremove" (or similar alternative in almost all other Linux package managers), users would be stuck with these dependencies on their devices. On my device, OSM Scout Server and its modules, take the top spot in /usr/share, then place #3 and #7. These are big packages and users should have an ability to simply remove them.

As for extra icons, I would suggest to put modules into some folder far away and not touch them unless you need to uninstall them for one reason or another.

Re: 0.10.0
 

I have just released a new version: 0.10.0

In short, this release is one of the major ones. This release has rather large amount of new features, so I'll try to address them one by one:

Search covering all databases

If you use Geocoder-NLP for search, its performed over all countries or territories that you have on device. Search is started from the map selected in GUI and later the data in other databases is considered. If you want to limit search to only one database, see Geocoder-NLP settings.

New backend: Valhalla

This is a major new functionality. I have ported Valhalla to SFOS. Valhalla is a routing engine behind Mapzen routing service. Now you could use it offline on your devices. Its fast, allows you to route between countries, and there are many other interesting features that are or could be easily made available to client programs. Main problem has been on how to distribute the datasets for Valhalla. For that, packaging the datasets and their download support had to be written as a part of the new release. Hence it took a while to implement it.

Since Valhalla is a full-blown server by itself, the communication between Valhalla and the client uses Valhalla's own protocol. Clients have to make adjustments to support it. Right now, before these changes are done in the clients, use libosmscout for routing.

Poor Maps and modRana do not support Valhalla backend yet. For Poor Maps, support is "simple" by making a copy of Mapzen Turn-by-Turn plugin and adjusting its URL + used geocoder. For modRana its probably a bit more complicated, unfortunately. Please wait for corresponding release and the new routing options in the clients.

Since its a new backend, use unsubscribe/subscribe for all the countries and territories that you update. As you did with Mapnik.

Libosmscout

The backend has been updated and new version of the maps has been uploaded to the data.modrana.org for distribution. This version should play better with OSM Scout by @Karry. But note that full compatibility is never guaranteed.

Storage of libosmscout datasets is also made optional. If yo don't use it, you can now opt out.

Modules

This release is the first one that uses modules. There are two modules: Fonts and Route. Fonts are packaged separately to avoid huge downloads on every update.

Fonts module contain Noto fonts by Google and should be able to display all possible languages, to my knowledge. It should be surely more than native SFOS fonts are supporting (rejoice people of Sri-Lanka).

Route module contains Valhalla routing engine that had to be distributed as a separate executable to allow separate updates of Valhalla and the server and to comply with Harbour rules.

Please install the modules with this release. If you don't have the module installed, a warning is displayed in the main view of the server.

Translations

There are two targets for translations now - OSM Scout Server and Valhalla. Valhalla's routing instructions come from Valhalla itself. So, the client programs have to request the corresponding language using Valhalla API. Since we are using Sailfish, ability of selecting en-US-x-pirate locale would be great to get authentic Sailfish navigation experience. I hope that Poor Maps and modRana could implement such selection :) .

As for translators, Valhalla's routing instructions are translated via Valhalla project. This is not done via transifex, but some other arrangement. See translations README for updated pointers (https://github.com/rinigus/osmscout-...ions/README.md)

For translators: I'll be happy to release as many point releases as needed. So, we could expect several 0.10.x releases with the updated translations.


Current state and upcoming new default

To put things into perspective: As with libpostal, to my knowledge, SFOS is the first mobile platform supporting Valhalla in offline mode. These are major libraries in the field and I do wonder, when the clients adopt Valhalla, what is the real life performance of SFOS maps vs others (Android, iPhone)...

As a part of the discussion with @otsaloma and @MartinK, it was suggested to select a sensible default backend(s) and mainly support that. I am mainly using Mapnik / Geocoder-NLP / Valhalla myself and I plan to develop and support this combination of backends. As soon as the adaptation of Valhalla by clients will be ready, its planned to make Mapnik / Geocoder-NLP / Valhalla combination as a default. Note that this combination allows you to render, search, and route between all the maps on device (if the maps are for the territories next to each other, though).

Libosmscout backend is there to stay. If you wish you could still use libosmscout, but please note that its testing would be limited. Libosmscout has a smaller datasets, which is currently its main advantage.


What's next

Before getting 1.0 out, I would like to mainly address the following.

* Helping clients to work with Valhalla backend. Poor Maps is easy on paper since it has support for the protocol already. I just don't know how to make it so that there would be minimal code duplication between OSM Scout and Mapzen routers. modRana is trickier for me - I am not familiar with its implementation of routing.

* Bugfixes. Please report bugs, I'll try to fix them as soon as I can

* Polishing. I am planning to mainly polish the experience. It should be easy for users to start using offline maps. So, if you have ideas on how to make it simple, wish to write documentation, howto guides, or just record all steps that you need to make it work on a fresh install. I am planning to make a small wizard on the first start to help with the selection of backends and settings sensible defaults.

* Night mode for Mapnik. Mainly design of the style. Default style adjustments / comments are always welcome.

* Geocoder-NLP does not support "Nearby" search requests. I am planning to implement that as well.

PS: Hopefully I haven't forgotten anything.

Re: 0.10.0
 

It's such a small change to Mapzen's online Valhalla, that I can do the Poor Maps implementation myself. And I'll probably just duplicate the code, it's fine. I'll ping you on GitHub so you can check the work. But, I'll want to do some testing too -- I'll probably have the time to do it over the weekend.

Re: [Announce] Native offline maps: OSM Scout Server
 

Translation and some testing done...

If not a Valhalla issue, there is a missing translation string. Maybe also a bug? Can't really test it.
.

Re: [Announce] Native offline maps: OSM Scout Server
 

Thank you very much!

That's Valhalla's output, unfortunately. So, we have to translate that through Valhalla.

What the server does is starting Valhalla as a separate process. Then it catches the output and forwards it to INFO stream. Unfortunately, Valhalla is a rather chatty beast and would be telling a lot of details when calculating the route. Maybe we should just ignore these messages and show on GUI only if requested by the user? Let's see what is the reaction of the users when it all gets supported by the clients.

PS: This WARN message is harmless since we don't use extracts in such form

Re: [Announce] Native offline maps: OSM Scout Server
 

Ok, thanks! I will take a look at Valhalla translation, but as Swedish translation isn't even started, it will take some time. Lots of complicated strings to translate.
Think I'll need client support to get a grip of it all.

Re: [Announce] Native offline maps: OSM Scout Server
 

Yes, and as mentioned earlier, client support is coming :) .

Before getting into translating Valhalla, its probably good to add github issue in their project (https://github.com/valhalla/valhalla). Valhalla's developers are quite fast to respond, but they do it during workdays - its their day job! As I could see in the issue tracker, Valhalla is considering moving translations into a separate platform as well. It would be good to tell your plan to translate via the issue and, I am sure, they will help you with the process.

Re: [Announce] Native offline maps: OSM Scout Server
 

Do you know a nice desktop client (linux) I can use to play with Valhalla?

Re: [Announce] Native offline maps: OSM Scout Server
 

No, not really. modRana is not supporting it yet. I played a bit with the leaflet, but its not ready for mobile release. Maybe I'll look into it soon again.

On Linux (or any PC), you could subscribe to Mapzen developer id (should be free as long as you don't exceed their limits) and make a small web-page with leaflet. See https://github.com/valhalla/valhalla...ng-to-a-map.md for tutorial. Note that you don't need any web-server, just point the browser to a corresponding HTML file.

Re: [Announce] Native offline maps: OSM Scout Server
 

For a fun of it, I benchmarked the routers on the same hardware. I used Nexus 4 and calculated a route from Stockholm to Aarhus (or nearby it). Rote itself is a bit more than 900 km.

Test results:

SFOS, Valhalla: on cold start (boot of the phone), it took 12 seconds from starting calculations in Poor Maps till getting the result. This includes geocoder search result for target and origin, communication between Poor Maps and the server (3 requests, 2x geocoder + 1 route). Out of these, Valhalla used only 5 seconds according to its logs.

SFOS, Valhalla: reroute calculation (Poor Maps navigation cleared, closed, and started again) took 5-6 seconds (2 were used by Valhalla itself). In this case, server was still running after it calculated the route. So, few things were in FS and application caches.

Android, MAPS.ME: Same route, without geocoder since it searches for origin and target separately. For route calculation, it took ... 50 seconds . Recalculation of the route by using a target nearby: 50 seconds again.

Android OsmAnd+: Again, geocoder is out of equation since GPS coordinates are used when calculating route. Here it took about 2 minutes(+-5 seconds).

So, we are getting a rather fast router. It would take some time to get it into clients, but not too long (NB: 'soon' not used). Libosmscout is quite fast as well (I leave its benchmarking for users when Valhalla can be compared against it).


@eson and other translators: Valhalla developers would love to get your help and extra languages on their server.

Re: [Announce] Native offline maps: OSM Scout Server
 

Just wanted to note I am aware of this and it's one of the items on top of my priority list. :)

I plan to do some traveling in the summer, so support for a robust and fast offline router in modRana will certainly be handy. :)

Re: 0.10.1
 

0.10.1 is out with the updated translations. Translators, thank you very much for this great job!

Re: [Announce] Native offline maps: OSM Scout Server
 

Excelent work. But I have a problem, when I activate Valhalla as route engine in OSM Scout Server, PoorMaps can't calculate any route. It always reports "No results". What is what I'm doing wrong?

If I disable Valhalla in OSM Scout Server, It works perfect, as expected.

I have SFOS 2.1.0.11 installed.

Thank you.

Re: [Announce] Native offline maps: OSM Scout Server
 

You don't do anything wrong. Poor Maps and modRana don't support Valhalla in OSM Scout Server yet. Please wait for a new release of Poor Maps and modRana to use it. See
http://talk.maemo.org/showpost.php?p...&postcount=260 http://talk.maemo.org/showpost.php?p...&postcount=268 for details.

Re: [Announce] Native offline maps: OSM Scout Server
 

Poor Maps 0.29 is out now with Valhalla support.

Re: [Announce] Native offline maps: OSM Scout Server
 

Thanks for the update, routing with Valhalla works perfect!!
Is possible to get routing instruccions in other languages?

Re: [Announce] Native offline maps: OSM Scout Server
 

Poor Maps should (hopefully) detect your OS language and request navigation instructions in that language. If Valhalla doesn't have translations for your language, see the previous two pages about where the translation is done. If you want to use a different language for navigation and rest of the UI, that's not currently possible. Should it be?

Re: [Announce] Native offline maps: OSM Scout Server
 

Valhalla is translated to Spanish language: https://github.com/valhalla/valhalla...master/locales

But I get routing instructions in English.

Re: [Announce] Native offline maps: OSM Scout Server
 

OK, looks like its a bug in Valhalla or Valhalla/Poor Maps interaction. I will look into it.

I am assuming that you are using Spanish in your GUI.

Meanwhile, would you mind checking whether Mapzen Turn-by-Turn fetches the instructions in Spanish for you?

PS: @otsaloma, thank you for the update!

Re: [Announce] Native offline maps: OSM Scout Server
 

Yes, I use Spanish language in my phone.

If I select 'Mapzen Turn-by-Turn' in Navigation, the instructions are in English too.

Re: [Announce] Native offline maps: OSM Scout Server
 

Re translations: we were using, what looks like, an outdated API. I have fixed the call and submitted corresponding pull request to Poor Maps ( https://github.com/otsaloma/poor-maps/pull/40 ). This seemed to fix routing instructions for OSM Scout Server Valhalla and Mapzen Turn-by-Turn. At least when I called Poor Maps with LANG=es_ES (or de_DE), I've got instructions in what appears to be Spanish (German).

Re separate option for specifying language: Maybe it would be great to have that option which would allow users to choose the second language. I would use it to get "Pirate routing version", but its surely not a high-priority task :) . So, would be nice to have as an option, but not essential with other aspects of navigation probably much more important.

Re: [Announce] Native offline maps: OSM Scout Server
 

I can not use osm scount server from -v 0.10.0. not load the maps ... i updated the maps in -v 0.10.0 but same. including poormaps-modrana

Re: [Announce] Native offline maps: OSM Scout Server
 

To help, I need to get more details:

* I assume that you want to view maps, right?

* Which backend do you use, Mapnik or libosmscout?

* If Mapnik, did you install fonts module?

* When viewing with Poor Maps or modRana, do you see jobs submitted to OSM Scout Server?

* What do the logs of the server say?

* Does Map Manager show that your maps are fine (nothing listed as incompatible/needs download)?