Re: [Announce] Native offline maps: OSM Scout Server
 

Thank you! Fixed in the latest commit and uploaded for translation.

Re: [Announce] Native offline maps: OSM Scout Server
 

Norwegian translation is now complete. -->Norsk oversettelse er nå komplett

Re: [Announce] Native offline maps: OSM Scout Server
 

Thank you very much! Its a big work.

Re: [Announce] Native offline maps: OSM Scout Server
 

With the map distribution server back online (thank you @MartinK!), I am preparing for a new release with an updated geocoder-nlp implementation that reduces the used storage size and should be faster as well. Unfortunately, it requires a new data storage format. While I did my best with the testing, its a major change and could lead to some breakage.

So, if you are planning a trip and depend on OSM Scout Server then I suggest to get the required maps today/tomorrow and store them in a safe place before the upgrade. If all goes as planned, I will publish new version on Wednesday morning with the World maps uploaded to the maps distribution server during a night between Tuesday and Wednesday.

0.8.0
 

I am happy to announce 0.8.0 release.

This release is mainly about geocoder-nlp polishing. Searches using geocoder-nlp should be faster and the datasets are about 2 times smaller. The libpostal part is still as large as before, but that is out of my hands.

The database format has changed and, as a result, you would have to get new version of the maps. The maps have been uploaded, so all should be ready. If you get the server from Harbour, it should be published on 29th, to sync with the new maps update.

To update the maps, open Map Manager and click Check for updates. Note that due to incompatibility of geocoder-nlp part, you cannot use 0.7.x maps on 0.8.x if you subscribed to geocoder-nlp. Update the maps as soon as you move to 0.8.x, please.

In this version, Linux CLI is supported again. Use command line options to subscribe and download the maps. Some configuration is done via INI configuration file that is populated with defaults on the first run. If you have any questions, please ask and I will help you out.

There are several translations updated. In addition, several fixes are carried over via update of libosmscout version.

I consider geocoder-nlp as ready for use. Please let me know if you experience slow response, too much RAM used, missing or not found data in search. I'll be happy to improve it as much as I can.

Unless major bug reports will start flying in, I will focus now on other aspects of the map server.

Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus I updated without problems. Maps (77MB + 81MB) have been updated via 3G in 8 minutes.

Re: 0.8.0
 

Thanks for the update!! I've installed it on my tablet, but I can't open it. Is this a bug?

Re: 0.8.0
 

Sounds like a bug. I presume you use it on i486, right? Do you have any output when you start it from command line? did the earlier versions work ?

Re: [Announce] Native offline maps: OSM Scout Server
 

Great to hear!

Re: 0.8.0
 

Yeah, I installed on i486 device. If I open the app via Terminal, it shows this message: 'Instrucción ilegal' (in English 'Illegal instruction').
0.7.5 version works fine, but I can't download maps because they aren't compatible and I must update the app.

Re: 0.8.0
 

Yes, its kind of a limbo with the change in the map versions. You could use 0.7.5 without geocoder-nlp, but let's try to fix it for 0.8.0.

I don't know what could be the cause for Illegal instruction (broken build, wrong SFOS version, bug). So, I just recompiled the server and uploaded a debug build at

http://osmscoutserver-test.e24files....8.0-1.i486.rpm

Would you mind to test if this one works? I don't have a tablet and, obviously, all works in the emulator, as far as I can tell.

Re: 0.8.0
 

Thanks again!! I've installed the new rpm file, but this bug is not solved.

If I try open the app via Terminal I get the same message:

[nemo@Sailfish Downloads]$ harbour-osmscout-server
[D] unknown:0 - QML debugging is enabled. Only use this in a safe environment.
Instrucción ilegal

Re: 0.8.0
 

If you know how to debug, you could use

gdb /usr/bin/harbour-osmscout-server

issue "run" in gdb and see what happens. It will probably give lots of messages regarding missing symbols, but maybe we can pull something out of it. When it crashes, see what "backtrace" command would give.

Since the server passed Harbour checks, I presume that this bug does not show up on clean installation. You could also try to move away configuration file (please move, don't delete, so we could examine it later!):

mv .config/harbour-osmscout-server/harbour-osmscout-server.conf ~/harbour-osmscout-server-send-for-repair.conf

then move away Maps.OSM (if the folder was as proposed by default) and try to start server again. If that doesn't help, try to reboot (maybe something is wrong with the tablet state?).

Would be good to know if there are other tablet users around and if they see the same error...

Re: 0.8.0
 

I've made a clean install, but it doesn't work, I reboot my tablet and doesn't work.
I've tried to use 'gdb' in my tablet, but I get 'command not found', If I've tried to install it I get error (file doesn't found). :(

I use SailfishOS 2.1.0.10, could it be the problem?

Well, maybe someone here who has a tablet can help us with this bug.

Thank you so much for your attention :)

Re: 0.8.0
 

@carmenluci, sorry that I don't know exactly how to help you.

I presume we have two differences already: use of i486 and 2.1.0.10. I am using only ported devices (OnePlus X and Nexus 4), with the both of them ARM and 2.0.5.6. Since these are ported devices, we'll move t 2.1.x series only after EA period is finished and there is a proper release. While there might be some builds for these devices around, or its possible to build myself, I would prefer not to jump over yet until its released.

So, would be good to know from other users of i486 (any SFOS version) or 2.1.0.10 (any CPU) how things are working for them. It would be helpful to know with the respect to this bug.

@carmenluci, when you reinstalled the server, did you move away the configuration file as well? Uninstall does not remove them, you have to move them away yourself.

Probably that wouldn't be of great help, but maybe you could run :

harbour-osmscout-server --help

It should just give you a list of command line options and stop the program. I wonder whether your crash occurs before or after the command line options processing.

Re: [Announce] Native offline maps: OSM Scout Server
 

The crash with 0.8 seems to be x86/i486 wide (or at least Tablet wide) - it also happens on my tablet. This is the output from Journal when I click on the OSM scout Server icon (not very informative):

Running the binary directly is similarly uninformative:
BTW, gdb can be installed by (as root):

Running the binary with gdb:

Let me know if there is something else I should run to get you mere information. :)

Also IIRC the Sailfish OS emulator which should be part of the SDK is also i486 & the SDK runs i486 binaries in it (citing the docs):

So it might be worth trying if the crash could be reproduced in the emulator, as that should make debugging it quite a bit easier. :)

Re: [Announce] Native offline maps: OSM Scout Server
 

Martin, thanks for analysis and feedback! @a-dekker has also opened a bug on GitHub, so we could move technical discussion over there: https://github.com/rinigus/osmscout-server/issues/107 (thank you @a-dekker!). I'll be following the both channels, but would try to reply to the questions via the same channel as the question.

SFOS emulator runs 0.8.0 fine for me, but I use stable SDK, not EA version. Since I develop for the stable SFOS, I would prefer to use that. Unfortunately, just compiling "for fun" by others is rather complicated - you have to install a large amount of support libraries that provide "brains" of the server (libosmscout, marisa, libpostal, ...)

To isolate the issue, tablet users out there, which version of SFOS do you run? Martin, do you have 2.1.x on the tablet or 2.0.x?

Do we have anyone running 2.0.x on the tablet that could test if OSM Scout Server starts?

For those who are stuck with this situation: you could use 0.7.5 and in Map Manager and Settings disable geocoder-nlp. That should allow you to use new maps (its only that component that changed), but would limit you to searches via libosmscout.

I did not work on making these transitions between map formats smoother. At present, I am trying to get the whole functionality of the served maps polished and hope that these bumps would be smaller in future.

Update: I tested OSM Scout Server 0.8.0 on 2.1.x SDK emulator - that worked fine. Note that if there is some illegal instruction for i486, it could be still fine for emulator if the instruction is propagated through VM to the host machine.

Re: [Announce] Native offline maps: OSM Scout Server
 

Re: Crash on i486 tablet

I have posted new RPM (broken functionality) for testing the crash, see https://github.com/rinigus/osmscout-server/issues/107 .

Re: [Announce] Native offline maps: OSM Scout Server
 

It looks like the issue with the crash on i486 is fixed, or at least I hope so. I have just replaced i486 0.8.0 RPM in OpenRepos with the new version. I presume that it should be OK now. Please let me know how it works. If all is fine, I'll upload a new version to the Harbour as well.

Re: [Announce] Native offline maps: OSM Scout Server
 

It works perfect!!! Thank you so much!!

Re: Mapnik
 

Just few words about the latest developments. I have been looking into improving the rendering of the maps. As a part for ensuring quality rendering, I am planning to add Mapnik backend for rendering.

The compilation of Mapnik for SFOS and its packaging is done, at first approximation, for ARM. For i486, I have been held back with what seems to be a bug in SDK and have asked for advice on how to proceed in developers list. When rendering some tiles with a test application on phone, the performance was good as well as the memory consumption. But its early days and would depend on the stylesheet / database layout.

Lots of work is ahead. I have to incorporate Mapnik into the server, design stylesheet together with the database layout. The plan is to use OSM Bright as a starting point (imposm import) and SQLite/Spatialite for data storage. If anyone around is a Mapnik guru, help would be appreciated. At present, I could render Estonia, World map outlines with SQLite database and shapefiles describing the World.

In addition, I am considering to publish OSM Scout Server with Mapnik only via OpenRepos and avoid publishing it in Harbour. That would allow me to distribute following sane Linux model with separate packages for Mapnik and the server. Otherwise I would have to start configuring Mapnik to look for input plugins / fonts in some obscure OSM Scout Server directories. In Harbour, I can publish then the version with libosmscout only, for example. What do you think about such approach?

Re: Mapnik
 

Sounds sensible to me, the harbour restrictions can be a real pain sometimes. Seems like if they wanted tight control over what can be installed for each app to avoid dependency issues etc, then they should have used a container format... but that's OT.

Can you explain what you think the benefits to the end user of using Mapnik will be? I've had a quick look at the OSM wiki but it's still unclear to me - is it speed of rendering, or quality, or both?

Re: Mapnik
 

Quality of the rendering and, in particular, position of labels should be top notch. There has been lots of work done on rendering in this library with lots of attention to details. When compared to libosmscout, Mapnik had much more developer-hours poured into it with the focus on one aspect of the function - rendering.

The stylesheet I was thinking to start with is https://github.com/mapbox/osm-bright . Maybe there are better styles out there that should be considered as a starting point. If so, let me know (ASAP).

When I manage to get stylesheet fully ported (there are several steps involved), it would be possible to engage end-users into design of the map styles using tools developed for Mapnik. Again, eco-system of Mapnik is much bigger allowing us to reap benefits from it.

As for speed of rendering, this is hard to judge right now. With a crippled style I was using it was very fast, but let's see it later.

There will probably be drawbacks too. In particular, right now, the data files (SQLite database) for feeding Mapnik is 2-3 times larger than libosmscout import. But that datafile includes all traffic lights and probably some other not so important information. For now, it was important to see that its actually possible. Let's see how well would it work out.

Re: [Announce] Native offline maps: OSM Scout Server
 

Sounds interesting! Two questions come to mind: What stylesheet language do you plan on using and are there graphical editors available for it? Does Mapnik and your stylesheet format make it easy to do different ppi tiles?

Re: [Announce] Native offline maps: OSM Scout Server
 

Its in CartoCSS which would be converted into XML that Mapnik uses to define maps and styles. Since we have multiple maps (different regions/countries) on the device, I am planning to define styles using normal mapnik approach (Carto -> XML) and then populate it by the server with the maps installed on device. That should allow to render all installed maps at once without the need to select one.

I am planning to take OSM Bright (it seems to be a rather popular style and the base for several others) and adapt it to SQLite import. I am not planning to edit style as such, there are probably better artists out there :)

As for GUI, there are probably at least 2, https://tilemill-project.github.io/tilemill/ is one of them.

Yes, there seem to be no problem with high PPI. As far as I can see, user would be able to define scaling factor that (s)he likes and its gonna be rendered on-fly with the same stylesheet but correspondingly wider/larger elements (lines, fonts, ...). So, it all fits the bill, as far as I can see.

Re: 0.8.1
 

I have just published a minor update by releasing 0.8.1.

Main job was done by translators who updated the strings, thank you. The update was published to get i486 fix submitted to Harbour as well. There is no new functionality added nor anything else refined.

I hope nothing got broken with i486 now.

Re: [Announce] Native offline maps: OSM Scout Server
 

Release: 0.9.0

I am happy to release 0.9.0 that brings Mapnik as an alternative rendering backend. This is probably the biggest release so far in terms of size - with the bundled libraries we are reaching 15MB for RPM package. And that's without mentioning huge map files (more on that a bit later). The release is not perfect, but I can probably tinker on it for months if I want to get it out very polished. So, I decided to release a bit early.

Mapnik brings an excellent rendering of maps including world coastlines and rendering of all available maps on device. But all this comes at cost - large maps and possibly slower rendering. So, its up to you to decide which backend to use. I have a preference to use Mapnik :)

With the introduction of new backend, there is a problem with the map updates. However, since I am not planning to add too many backends, I think its better to work around it on your (as a user) side than to start investing time into programming fixing the issue. So, a general remark: When a new backend is added, such as with the release of 0.9.0, please ensure that you have subscription information updated. For that, in Map Manager, click on "Check for updates", decline updates, and unsubscribe/subscribe each territory that you have subscribed. After that, proceed with the downloads. This is not needed if you don't want to use the added backend(s).

Finally, regarding this release, there are some issues with the maps distribution server. Right now, I switched to old CDN model. As soon as data.modrana.org is back online, I will stop distribution via CDN (or when the credit runs out, whatever comes first). So, before downloads, just click on "Check for updates". This should reroute you to the current distribution server. Sorry for inconvenience. If anyone has an access to the server that can be used for maps distribution and wants to contribute, please let me know, we could always use them as a backup, for example.

I will look into how to reduce Mapnik datasets. There is an idea on switching to different geometry BLOB format, but it requires some testing. I will not be able to probably get as low as libosmscout does, but would give a shot on reducing it. Right now, serving Germany/France as a single database is not that simple on mobile with Mapnik, but I haven't tried.

In addition to the reduction of the map database size, there is some work to be done on styling it as well. I have adopted a style used as a basis for many others and modified it to suit SQLite database. Style is available at https://github.com/rinigus/osm-bright-sqlite . It does require further review on which features should be visible at which zoom level. Try to use it and give the feedback on when should we should house numbers, POIs, and such. Some braver users could try to re-style it, make a style for night driving, ...

Enjoy using it!

Re: [Announce] Native offline maps: OSM Scout Server
 

712+(241+223) MB downloaded in 25 minutes.

Re: [Announce] Native offline maps: OSM Scout Server
 

Tested the Mapnik rendering on the tablet, it seemed to work fine. Rendering speed was reasonably good. The map style though is as they state a bit of a "starting point".

Re: [Announce] Native offline maps: OSM Scout Server
 

I am planning to mainly work on the performance and shrinking the databases. Some changes may require special patches for mapnik, but I'll submit them upstream. For example, I think I found the way to reduce maps significantly, about to the half of their current form. To my surprise it did not effect the performance (initial testing on onyx), so we should probably switch to that. However, that means I would need to keep two versions for the style - one for mobile with all patches applied and the second for PC and style design.

For best performance, I tune the database layout to match the style. This means that if we want to make major changes in loading sequence, we would be better off to tune the database too. Which is not a major issue, as long as we remember it. In particular, you would want SQL SELECT statements in https://github.com/rinigus/osm-brigh...spatialite.mml to match SQL database as arranged at https://github.com/rinigus/osm-brigh...reorganize.sql .

I have not looked into GUIs on style editing. Someone who wants to adjust styles, should probably look into https://github.com/kosmtik/kosmtik or https://tilemill-project.github.io/tilemill/ . TileMill seems to come in a package that installs mapnik as well. So, its not good for me (I have to use a custom one), but could be great for a designer. I don't know if they come with SQLite plugin enabled though. kosmtik I tried, but it didn't seem to work on my Linux PC immediately, probably some tinkering is required.

Anyway, to contribute to style development: we could collect the ideas either here or in issues of the style project; try to setup GUI and see if it works. I'll be happy to help if you run into problems with the setup of GUI, as much as I understand how it works.

If there is someone interested in working on a style, let us know and see how we can help.

Re: [Announce] Native offline maps: OSM Scout Server
 

I remember making a couple map styles when TileMill was a Mapbox product and connected to their APIs. It's now a bit more work set up data for TileMill, but I remember basic styling, such as changing colors and changing the zoom levels at which elements are visible, being fairly easy. Things like symbols for POIs and road number shields are more complicated, but you can get quite far without those.

I'm not quite a designer, but definitely interested in map styling. However, I'm more likely to try to get my own online tileserver running and style that with the Mapbox GL JSON style format.

Re: [Announce] Native offline maps: OSM Scout Server
 

POIs were mainly taken from official openstreetmap style. There are probably several other sets that can be used. But you are right, its relatively simple task. Just needs someone to work on it with an eye on design. Let's see if someone will step up.

Re: [Announce] Native offline maps: OSM Scout Server
 

I am preparing for the next release that would alter Mapnik SQLite database schema. As a result, the new database will be incompatible with the current OSM Scout Server. So, if you need the maps now, please download them.

I expect to finish map imports and upload them to data.modrana.org in the next 24-48 hours. Within this timeframe the corresponding builds will be made and uploaded to OpenRepos/Harbour. For Harbour users: expect the new version appearing with a delay due to review by Jolla.

This timeline will hold assuming that there will be no surprises during the roll-out of the maps nor something will come out during the final testing.

As for Mapnik style development, I have reworked the current one including POI appearance and other details. The base is now OSM Bright version TM2.

Turned out that many map styles cannot be used as a base due to the copyright, including Mapbox Streets, for example. However, if someone wants to design a new style, especially color schemes, we could make it as follows. You could use Mapbox Studio or some other GUI to design it (please read fine print regarding the style that you want to use as a base, it should allow us to use the style without Mapbox servers; acknowledgment is fine - can always be added into About). When ready, let me know and share your Mapbox GL JSON style (including license of your work). I will then try to translate it into Mapnik stylesheet. While not optimal, it allows us to use simple GUI tools. Easiest would be to adjust just colors. Fonts and appearance of the roads, POIs at different levels are possible to change, but not that simple since it may require changes in map database.

Re: [Announce] Native offline maps: OSM Scout Server
 

The new release is out - 0.9.5, followed by 0.9.6.

In addition to updated translations (thank you for all involved), this release changes were related to polishing Mapnik backend. Via a simple patch, I have added support for TWKB geometry encoding in Mapnik/SQLite. This allowed to reduce the size of the Mapnik databases by roughly 2x times. Note that it maybe at expense of CPU usage during reading, but I did not notice any slowdown due to TWKB packing.

Second part of enhancements are related to the style of Mapnik rendering. I profiled rendering and adjusted stylesheet and underlying database layout to speed it up. However, on slower hardware and using small tiles, as with modRana, there might be some problems (see https://github.com/M4rtinK/modrana/issues/201 , possible fix waiting in review).

While Mapnik databases are large, further reduction of them would require investigation of alternative formats and possibly writing an additional input plugin for Mapnik. Taking into account that the current database file size seems to be smaller than "the-next-big-thing" vector tiles, reduction of the file size would be non-trivial project.

As for the map style, I am looking for someone who would like to work on them. As mentioned earlier, I'll be happy to adapt the styles from other formats, if available.

With the respect of further development, I was planning to extend search support to all available countries next. Also would have to update underlying libosmscout library to catch up with the development.

Would be great to know what are the further expectations of the users and what should be done in future with the respect of the server. See https://github.com/rinigus/osmscout-server/issues for current TODO list.

PS: 0.9.6 fixed a typo in Mapnik style that slowed down at z<9

Re: [Announce] Native offline maps: OSM Scout Server
 

Looks like something went terribly wrong while importing the planet. At least, Latvia got roads and nodes from all over Europe. I suspect that planet.pbf was updated while I was downloading it leading to a very strange import for at least that country. Please let me know if you see errors in the territory that of interest for you. If its just few countries, I can update country-by-country faster. Latvia is uploaded now and will be probably in the data.modrana.org server in few hours.

I will redownload the planet and make a new import in the next few days.

Edit: Guatemala and Latvia are fixed and available on the maps distribution server.

Re: [Announce] Native offline maps: OSM Scout Server
 

@rinigus. Thanks. It's good for my two regions.

Re: [Announce] Native offline maps: OSM Scout Server
 

Many thank @rinigus

Re: [Announce] Native offline maps: OSM Scout Server
 

Thank you very much for this piece of gold for Sailfish offline navigation. I have no words to be grateful for your work! Thank you very much @rinigus !

Re: [Announce] Native offline maps: OSM Scout Server
 

@ferlanero: thank you for your kind words. Note that large work is made by everyone working on the client part of the solution. Without excellent GUI clients, there would be no offline solution!

The new databases for countries and territories are uploaded to data.modrana.org. So, all of you who had problems with the previous upload, please check if its resolved. For countries that I tested with, it all seemed to be fine. In addition, planet download was checked against MD5 in the scripts now, which should improve robustness of the import. For those, who did not have any problems, please wait few days to avoid majorly stressing the data.modrana.org over the weekend :) . Please let me know if there are still bugs with the import.

Now I wonder if anything has to be changed in Mapnik style? Comments/suggestions?

Re: [Announce] Native offline maps: OSM Scout Server
 

You are right @rinigus. Big thank, for all those who have made navigation possible in SFOS too @MartinK @otsaloma