maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Applications (https://talk.maemo.org/forumdisplay.php?f=41)
-   -   [SFOS] [Announce] Native offline maps: OSM Scout Server (https://talk.maemo.org/showthread.php?t=97823)

rinigus 2018-05-18 19:16

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by pichlo (Post 1544455)
I second that. I was just trying to send you a donation but you do not make it exactly easy :(

Thanks for trying! I'll look into it a bit later...

As for OSM Scout Server, I am working on updating libpostal to 1.0 version. This will bring changes in Address parser database format leading to incompatibilities of the future maps with the current server version. I'll try to warn in advance, so you could plan map downloads and server updates.

rinigus 2018-05-25 08:06

Re: [Announce] Native offline maps: OSM Scout Server
 
As mentioned above, I am working on updating libpostal to 1.0. While API was adopted fast, there were few issues that I had to hack around. The main is https://github.com/openvenues/libpostal/issues/351 , for which I seem to have workaround (at least the planet was imported).

I'll be testing it over the next few days. The tests over the last week (even before workaround) were promising. If all goes fine, I'll update the software and the maps via corresponding distribution channels. However, note that its rather large change and those who have to rely on the server in the next couple of weeks, please check for the feedback of the ones who updated first. Also, if you wish to delay the update, please get all the maps before the update will hit the servers. Otherwise, you'll be unable to do that later.

rinigus 2018-05-27 04:06

Re: [Announce] Native offline maps: OSM Scout Server
 
New version is out: 1.6.0

This is a major upgrade and, due to the upgrade of libpostal API, the address parsing part of the maps is incompatible with the earlier versions. After upgrade, go to Map Manager, click "Check for updates" and proceed as usual. Below, is the description of what's new.

This version is the first one supporting QtQuick Controls GUI. So, it opens full GUI for Linux desktops. It could also simplify porting to Nemo, Ubuntu Touch, Plasma Mobile, if any porting is needed at all.

Upgrade of libpostal brings us a large work done by libpostal author in 2017. I would like to encourage you to test performance of the search and check whether your query was parsed correctly. From disk storage requirements side, the new libpostal doesn't use language parser dataset anymore which should reduce storage requirements by ~500MB on global dataset. Country datasets have changed as well, but that could be larger or smaller than before, depending on the country.

Geocoder now sorts the results somewhat differently to promote Glasgow (city) in front of the pub with the same name. This is done by preferring results with smaller number of admin levels (so Malmö kommun is preferred to Malmö city) and, for the same admin level results, with the shorter name.

Valhalla packaging changes were already in use earlier, but now the scripts are part of the release. I am also not showing country selection when its not needed in the main screen (its used only if geocoder is limited to one country or libosmscout is in use).

The translations have been updated and Dutch (BE) added to the list by @pljmn. Thanks for all the translators!

Current libpostal SFOS adaptation uses one reversed commit. Such adaptation allowed me to import the planet, but it may have some side effects. So, please tell if the server is killed for one reason or another. The corresponding issue has been opened at libpostal repository, but its not properly resolved yet.

Finally, as a simpler version, I added bitcoin donation link. Please don't ask for paypal, I don't want to engage with it at the moment. Bitcoin, as liberapay, gives you the list of transactions. So, all donations are transparent in terms of time and amounts.

New updated libpostal and geocoder-nlp datasets are available, enjoy the release.

MartinK 2018-05-28 20:37

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1544688)
New version is out: 1.6.0
This version is the first one supporting QtQuick Controls GUI. So, it opens full GUI for Linux desktops. It could also simplify porting to Nemo, Ubuntu Touch, Plasma Mobile, if any porting is needed at all.

Nice! :) On a related note, I finally managed to have some progress in my Fedora packaging efforts:

prime_server
- PR: https://github.com/rinigus/pkg-prime_server/pull/1 (already merged, thanks! :) )
- builds fine in Fedora COPR
- opened an upstream issue to clarify licensing: https://github.com/kevinkreiser/prime_server/issues/70

valhalla
- PR: https://github.com/rinigus/pkg-valhalla/pull/1
- fails during build in COPR due to some weird protocol buffer errors, details are in the pull request

What OSM Scout Server dependencies would you recommend next ? Looking at the OSM Scout Server spec file:

already available
- marise (0.2.4) is already packaged
- mapnik (3.0.19) is already packaged
- libmicrohttpd (0.9.59) is already packaged
- tokyocabinet (1.4.48) is already packaged

missing in Fedora
- libosmscout is not yet packaged
- libpostal is not yet packaged

rinigus 2018-05-29 06:38

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by MartinK (Post 1544734)
Nice! :) On a related note, I finally managed to have some progress in my Fedora packaging efforts:

prime_server
- PR: https://github.com/rinigus/pkg-prime_server/pull/1 (already merged, thanks! :) )
- builds fine in Fedora COPR
- opened an upstream issue to clarify licensing: https://github.com/kevinkreiser/prime_server/issues/70

valhalla
- PR: https://github.com/rinigus/pkg-valhalla/pull/1
- fails during build in COPR due to some weird protocol buffer errors, details are in the pull request

What OSM Scout Server dependencies would you recommend next ? Looking at the OSM Scout Server spec file:

already available
- marise (0.2.4) is already packaged
- mapnik (3.0.19) is already packaged
- libmicrohttpd (0.9.59) is already packaged
- tokyocabinet (1.4.48) is already packaged

missing in Fedora
- libosmscout is not yet packaged
- libpostal is not yet packaged

Marisa, libmicrohttpd, tokyocabinet should be OK

Mapnik: Mapnik in 3.0.x line still doesn't include TWKB support for SQLite databases. The corresponding PR has been merged upstream (https://github.com/mapnik/mapnik/pull/3660) and I am currently applying patch while building Mapnik for SFOS (https://github.com/rinigus/pkg-mapni...nik.twkb.patch). Without it, Mapnik maps were 2x larger, if memory serves me right. Hence, all distributed maps use this format and vanilla 3.0.x Mapnik will be unable to render them. Se, we'll need either to convince Fedora packagers to add this patch or package it separately and link statically to the server.

libosmscout: I am using rather old version of it, see https://github.com/rinigus/libosmscout/tree/sailfish . Newer versions have somewhat different API and I decided not to get involved into keeping it up to date. So far, nobody volunteered to keep this part of code in sync with the latest libosmscout releases. We can also skip libosmscout for this packaging round and compile the server without it (there is a disable option in qt pro file). The release that I use is https://github.com/rinigus/libosmsco...git.20170521-1

libpostal: Current libpostal has a bug that can be triggered by searching for "Хозтовары" (issue https://github.com/openvenues/libpostal/issues/351). Please use https://github.com/rinigus/pkg-libpostal with devel branch (https://build.merproject.org/package...ice?expand=1); loads https://github.com/rinigus/libpostal...nsion_fail_fix branch. Usually, libpostal developer is fast in responding to issues. For some reason, he hasn't been that active lately. But I am sure it will get fixed and we can later switch to the upstream branch.

So, while lots of functionality is provided by packages, there is still quite some work that has to be done behind the scenes to make it all work in practice :)

Thanks for working on it, looking forward to see the server in mainstream Linux.

DrYak 2018-05-29 09:52

Re: [Announce] Native offline maps: OSM Scout Server
 
Big Kudos to you guys !
http://www.sympato.ch/smileys/megabounce.gif

MartinK 2018-06-02 11:39

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1544742)
Marisa, libmicrohttpd, tokyocabinet should be OK

Mapnik: Mapnik in 3.0.x line still doesn't include TWKB support for SQLite databases. The corresponding PR has been merged upstream (https://github.com/mapnik/mapnik/pull/3660) and I am currently applying patch while building Mapnik for SFOS (https://github.com/rinigus/pkg-mapni...nik.twkb.patch). Without it, Mapnik maps were 2x larger, if memory serves me right. Hence, all distributed maps use this format and vanilla 3.0.x Mapnik will be unable to render them. Se, we'll need either to convince Fedora packagers to add this patch or package it separately and link statically to the server.

Oh, so the patch is already upstream, just not yet in a released version of Mapnik? For now I've just built the Fedora Mapnik package with the patch on top in COPR:
https://copr.fedorainfracloud.org/co.../build/761846/

Then once we are far enough to submit OSM Scout Server for package review (so that it can go the the official Fedora repos) we can either ask the Mapnik maintainers to include the patch or possibly a new upstream version including it has been released (and the Mapnik maintainers seem to track upstream releases pretty closely).


Quote:

Originally Posted by rinigus (Post 1544742)
libosmscout: I am using rather old version of it, see https://github.com/rinigus/libosmscout/tree/sailfish . Newer versions have somewhat different API and I decided not to get involved into keeping it up to date. So far, nobody volunteered to keep this part of code in sync with the latest libosmscout releases. We can also skip libosmscout for this packaging round and compile the server without it (there is a disable option in qt pro file). The release that I use is https://github.com/rinigus/libosmsco...git.20170521-1

Is there some functionality libosmscout provides that is not covered by other libraries (Valhalla/libpostal/Mapnik/etc.) ? I think we can indeed skip libosmscout for the initial packaging round, as even if it builds fine in COPR, it would be weird to submit an old version of libosmscout for package review if we know OSM Scout Server will not work with a new version if it ever gets updated.


Quote:

Originally Posted by rinigus (Post 1544742)
libpostal: Current libpostal has a bug that can be triggered by searching for "Хозтовары" (issue https://github.com/openvenues/libpostal/issues/351). Please use https://github.com/rinigus/pkg-libpostal with devel branch (https://build.merproject.org/package...ice?expand=1); loads https://github.com/rinigus/libpostal...nsion_fail_fix branch. Usually, libpostal developer is fast in responding to issues. For some reason, he hasn't been that active lately. But I am sure it will get fixed and we can later switch to the upstream branch.

I've tried to build libpostal yesterday and while the packing & the software itself (a C library with few depndencies) looks fairly simple, the build failed with some rather weird errors:

Code:

/bin/sh ../libtool  --tag=CC  --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g  -Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/libpostal.la  -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o  -L/usr/local/lib ../src/.libs/libpostal.so -lstdc++ -lm
/usr/bin/ld: test_libpostal-test.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_expand.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_parser.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_transliterate.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_numex.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_trie.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_string_utils.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_crf_context.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:575: test_libpostal] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0/test'
make[1]: *** [Makefile:453: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0'
make: *** [Makefile:362: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.TOmV84 (%build)

Full log: https://copr-be.cloud.fedoraproject....ilder-live.log

The spec file used: https://github.com/M4rtinK/pkg-libpo...libpostal.spec

Failed build in COPR: https://copr.fedorainfracloud.org/co.../build/761920/

Googling for "Nonrepresentable section on output" I found some hints that it might be related to PIE/PIC and linking. Any ideas/pointers what to try to fix the build ? Thanks in advance! :)

Quote:

Originally Posted by rinigus (Post 1544742)
So, while lots of functionality is provided by packages, there is still quite some work that has to be done behind the scenes to make it all work in practice :)

Thanks for working on it, looking forward to see the server in mainstream Linux.

It's a important project to which I don't know about any alternatives - so it should also be as widely available as possible, even if it requires some work. :)

rinigus 2018-06-02 12:52

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by MartinK (Post 1545020)
Oh, so the patch is already upstream, just not yet in a released version of Mapnik? For now I've just built the Fedora Mapnik package with the patch on top in COPR:
https://copr.fedorainfracloud.org/co.../build/761846/

Then once we are far enough to submit OSM Scout Server for package review (so that it can go the the official Fedora repos) we can either ask the Mapnik maintainers to include the patch or possibly a new upstream version including it has been released (and the Mapnik maintainers seem to track upstream releases pretty closely).

Its in the master branch for a year, I think. However, that's for upcoming 3.1 series and they don't seem to include it into 3.0 versions.


Quote:

Originally Posted by MartinK (Post 1545020)
Is there some functionality libosmscout provides that is not covered by other libraries (Valhalla/libpostal/Mapnik/etc.) ? I think we can indeed skip libosmscout for the initial packaging round, as even if it builds fine in COPR, it would be weird to submit an old version of libosmscout for package review if we know OSM Scout Server will not work with a new version if it ever gets updated.

No, there is none. In this context, it just allows to use smaller databases. And I agree, it will be weird to submit old libosmscout version.

Quote:

Originally Posted by MartinK (Post 1545020)
I've tried to build libpostal yesterday and while the packing & the software itself (a C library with few depndencies) looks fairly simple, the build failed with some rather weird errors:

Code:

/bin/sh ../libtool  --tag=CC  --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g  -Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/libpostal.la  -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o  -L/usr/local/lib ../src/.libs/libpostal.so -lstdc++ -lm
/usr/bin/ld: test_libpostal-test.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_expand.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_parser.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_transliterate.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_numex.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_trie.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_string_utils.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_crf_context.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:575: test_libpostal] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0/test'
make[1]: *** [Makefile:453: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0'
make: *** [Makefile:362: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.TOmV84 (%build)

Full log: https://copr-be.cloud.fedoraproject....ilder-live.log

The spec file used: https://github.com/M4rtinK/pkg-libpo...libpostal.spec

Failed build in COPR: https://copr.fedorainfracloud.org/co.../build/761920/

Googling for "Nonrepresentable section on output" I found some hints that it might be related to PIE/PIC and linking. Any ideas/pointers what to try to fix the build ? Thanks in advance! :)

Try to drop https://github.com/M4rtinK/pkg-libpo...ostal.spec#L33

Code:

CFLAGS="$CFLAGS -fPIC -lstdc++"
CXXFLAGS="$CXXFLAGS -fPIC"

I had to use it for SFOS packaging, but it shouldn't be needed for you.

Quote:

Originally Posted by MartinK (Post 1545020)
It's a important project to which I don't know about any alternatives - so it should also be as widely available as possible, even if it requires some work. :)

Well, there are servers around (valhalla, tile servers, and geocoders) that are surely better in many aspects. Probably, the main advantage is that all map data is packaged and available online. It helps that you "just" need to install only one server as well [which means that the hard part of install is done by packagers].

MartinK 2018-06-02 23:15

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1545023)
Try to drop https://github.com/M4rtinK/pkg-libpo...ostal.spec#L33

Code:

CFLAGS="$CFLAGS -fPIC -lstdc++"
CXXFLAGS="$CXXFLAGS -fPIC"

I had to use it for SFOS packaging, but it shouldn't be needed for you.

Tried that, still the same errors (at least AFAICT):
Code:

/bin/sh ../libtool  --tag=CC  --mode=link gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR='"/usr/local/libpostal/data/libpostal"' -g  -Wl,-z,relro  -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -L/usr/local/lib -o test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o ../src/libpostal.la  -lm
libtool: link: gcc -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -O3 -Wfloat-equal -Wpointer-arith -std=gnu99 -DLIBPOSTAL_DATA_DIR=\"/usr/local/libpostal/data/libpostal\" -g -Wl,-z -Wl,relro -Wl,-z -Wl,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -o .libs/test_libpostal test_libpostal-test.o test_libpostal-test_expand.o test_libpostal-test_parser.o test_libpostal-test_transliterate.o test_libpostal-test_numex.o test_libpostal-test_trie.o test_libpostal-test_string_utils.o test_libpostal-test_crf_context.o  -L/usr/local/lib ../src/.libs/libpostal.so -lm
/usr/bin/ld: test_libpostal-test.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_expand.o: relocation R_X86_64_32 against `.rodata.str1.8' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_parser.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_transliterate.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_numex.o: relocation R_X86_64_32 against symbol `greatest_type_info_string' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_trie.o: relocation R_X86_64_32S against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_string_utils.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: test_libpostal-test_crf_context.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
make[2]: *** [Makefile:575: test_libpostal] Error 1
make[2]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0/test'
make[1]: *** [Makefile:453: all-recursive] Error 1
make[1]: Leaving directory '/builddir/build/BUILD/libpostal-1.0.0'
make: *** [Makefile:362: all] Error 2
error: Bad exit status from /var/tmp/rpm-tmp.NmocgJ (%build)

Full log: https://copr-be.cloud.fedoraproject....ilder-live.log

Link to the failed build: https://copr-be.cloud.fedoraproject....ilder-live.log

Spec file diff from previous attempt: http://copr-dist-git.fedorainfraclou...7c50aef7480688

Googling for "relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a PIE object; recompile with -fPIC" uncovered some discussion about cases in other programs when trying to link relocatable and non-relocatable code together, so I wonder if we are seeing something like that as well. Looking at the test subfolder where this is happening, it seems to have some custom flags defined here:
https://github.com/openvenues/libpos...st/Makefile.am
So maybe it compiles the testing code wrong or something like that ?
Also I wonder if the tests could just be disabled, but that feels like a hack.

Quote:

Originally Posted by rinigus (Post 1545023)
Well, there are servers around (valhalla, tile servers, and geocoders) that are surely better in many aspects. Probably, the main advantage is that all map data is packaged and available online. It helps that you "just" need to install only one server as well [which means that the hard part of install is done by packagers].

Yes, that's what I think OSM Scout Server is best in the world at - making all the power of the existing open geodata & libraries seamlessly available both to application developers (language independent & well documented common API) and users (super easy offline data downloading & updates + data sharing with all apps using the OSM Scout Server API).

I don't think any existing solution can come close to this - otherwise developers would have to handle every lib themselves (compilation/packaging/bundling) & library APIs are often limited (C/C++ only, API that requires a heavy server software to run, etc.), not to mention all the hassles with offline data management that would have to be solved more or less separately for every application.

From the user side, there is super easy & unified offline data management & all the nice features made possibly by the easily accessible APIs. :)

rinigus 2018-06-03 05:27

Re: [Announce] Native offline maps: OSM Scout Server
 
@MartinK, it builds static and shared versions of the library. I would suggest to add
Code:

--disable-static
to configure options and try again. Since static lib is built after the shared one, it may got confused and linked tests to the static libs.

Does it build on your PC?

MartinK 2018-06-11 01:09

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1545040)
@MartinK, it builds static and shared versions of the library. I would suggest to add
Code:

--disable-static
to configure options and try again. Since static lib is built after the shared one, it may got confused and linked tests to the static libs.

Does it build on your PC?

So looks like the tests have been somehow at fault, as they are compiled against libpostal in some ultra weird manner. So for now I've hard-disabled them (dropped their compilation from the makefile) and the build finally runs to the end. This should be good for now, later one for the actual package review something more robust (possibly including reporting the issue upstream) might be needed. I'll open a PR for the libpostal packaging soon, just have to cleanup my changes a bit.

Anyway, as far as I can tell this concludes the missing dependencies & I can finally start packaging OSM Scout Server itself. :)

rinigus 2018-06-11 03:53

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by MartinK (Post 1545310)
So looks like the tests have been somehow at fault, as they are compiled against libpostal in some ultra weird manner. So for now I've hard-disabled them (dropped their compilation from the makefile) and the build finally runs to the end. This should be good for now, later one for the actual package review something more robust (possibly including reporting the issue upstream) might be needed. I'll open a PR for the libpostal packaging soon, just have to cleanup my changes a bit.

Anyway, as far as I can tell this concludes the missing dependencies & I can finally start packaging OSM Scout Server itself. :)

Sounds great! Please note that I haven't tested any 'make install' on desktop Linux. The path for directories is coded in pro-file and I was waiting till you get here and help me to fix it. So, if you know what's the common way to specify paths then please do so and send PRs :)

rinigus 2018-06-17 07:37

Re: 1.7.1
 
New release is out: 1.7.1. With the 1.7.0, followed by 1.7.1, I focused on geocoder enhancements.

New geocoder supports aliases when you search for data using guide search (Nearby). Aliases, as suggested by @palbr (thank you!), are pulled from https://wiki.openstreetmap.org/wiki/...pecial_Phrases with the used ones listed at https://rinigus.github.io/osmscout-server/tags/ . The list is also available as an API call, via poi_types URL, allowing us later to extend WhoGo Maps and modRana to automatically provide it to the users (if the client developers will approve it). Tag aliases are also used when returning the search results to identify objects in a localized manner.

Geocoder now imports and allows you to search for POIs without names (playground, ATM, toilets, ...), Again, suggested by @palbr. This is done for guide (nearby) search. When using guide search, its also possible to separate search by name or type when looking for POI (at least one has to be specified). Released WhoGo Maps already supports it, modRana is under development. Corresponding map update has been released and available if you check for updates in map manager. Note that only geocoder-nlp maps were updated this time.

As reported by @legacychimera247, some villages were missing in the search. This is fixed now, but it looks like I have inserted these missing villages into a bit high level in the hierarchy. This will be fixed in near future.

Many more types are now imported and supported by geocoder. Let me know if something important is missing.

The latest version of translations has been also pulled into the package. Thank you to all the translators doing great job!

feedme 2018-06-17 18:26

Re: [Announce] Native offline maps: OSM Scout Server
 
Hello,
I am deeply in love to osm!
But yhis darling does not allow to use sdcard .
I created a directory to sdxard ( devel-su) gave rights and osm stopped nagging "no rights" but then there is open database error.

( there is / nemo/ Maps.OSM) I also tried to copy that but again even after givin rights to/ run/nemo I have a problem , not able to create...)

running sailfish X

peterleinchen 2018-06-17 20:51

Re: [Announce] Native offline maps: OSM Scout Server
 
I do not think it has something to do with X or Jolla C (which I am using).
Why using devel-su (root) at all?
Just create any directory as normal user (nemo) and assign this directory within settings of OSM. Done.

DrYak 2018-06-19 10:03

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by feedme (Post 1545574)
But yhis darling does not allow to use sdcard .
I created a directory to sdxard ( devel-su) gave rights and osm stopped nagging "no rights" but then there is open database error.

( there is / nemo/ Maps.OSM) I also tried to copy that but again even after givin rights to/ run/nemo I have a problem , not able to create...)

running sailfish X

My own strategy (inherited back from older setups with other mapping software) :

OSM server set to use : /home/nemo/Maps/OSM

/home/nemo/Maps is a symlink that point to /run/media/nemo/SDCard/Maps

/run/media/nemo/SDCard is a SDXC card reformatted to a linux partition (BTRFS in my case, but will work as well with EXT4 or F2FS).

/run/media/nemo/SDCard/Maps and /run/media/nemo/SDCard/Maps/OSM
are owned by nemo:nemo,
access rights 0775 (rwxrwxr-x)

works flawlessly.

rinigus 2018-07-09 10:59

Re: [Announce] Native offline maps: OSM Scout Server
 
New maps are out and available for download.

It looks like the last time I uploaded older Mapbox GL tiles. This time, all the data should be up to date, about 1-2 weeks old extract from OSM Planet.pbf.

rinigus 2018-07-12 20:18

Re: 1.8.0
 
New version is out - 1.8.0

This version is linked directly to Valhalla via its C++ API. As a result, OSM Scout Server provides Valhalla's exported functionality in its own process. Earlier, the server had to start Valhalla separately and work as a proxy between map clients and Valhalla. Now, this extra HTTP connection is skipped.

For end users, such integration means that, after upgrading to 1.8.0, OSM Scout Server Module: Route can be uninstalled. Its not used anymore.

For map client developers, there are lots of exciting features exported, in addition to routing. Some of them are:

* map matching - for just-in-time snapping position to the road/path network; for exporting recorded GPX as a set of navigation instructions.

* time-distance matrix calculation between set of locations.

* finding optimized route between set of locations (so the order and route are optimized).

* isochrone - for example, where can I get in 30 minutes by car from here? see example at https://cdn-images-1.medium.com/max/...Lqn7HoMo_8.png

Corresponding section in README: https://github.com/rinigus/osmscout-...other-services .

Note that while elevation service is exported as well, I don't distribute the data for it. I will look into it, but it sounded like it takes 1.2TB for a planet - bit excessive for mobile. If they find the way to reduce the data requirements, would be great way to get elevation-aware routing and its visualization.

For packagers (@MartinK), the server doesn't need anymore prime_server, zmq. I was made aware of this Valhalla C++ API few days ago (heard about it earlier when they started working on it in 2017, but didn't check the progress). I made a new packaging for Valhalla (https://github.com/rinigus/pkg-valhalla-lite) that is preferred for the compilation of the server. This packaging and the full Valhalla packaging (https://github.com/rinigus/pkg-valhalla) scripts were also modified to use cmake, as they switched the build system.

As always, the translators have updated the translations. I will try to remember to release point releases 1.8.x this time and get translations up-to-date. Sorry for delays.

I have added car styles for Mapbox GL maps. These are right now just increasing the priority of the gas stations, parking spaces, and such. Later we can maybe polish it further, feel free to think on how to improve them :)

I am planning to work on map matching and exploring it with the map client. Very rough POC has been done and my phone is able to map match almost in real time already with the current API (you can see the location moving as you are, but with the direction found using map matching). So, the early results are promising, but there is still much to do.

rinigus 2018-07-13 07:23

Re: [Announce] Native offline maps: OSM Scout Server
 
Quick notice: WhoGo Maps (and probably Poor Maps) check whether routing module is installed when they list available routers. So, keep the routing module on device until corresponding change is done in WhoGo Maps: https://github.com/otsaloma/whogo-maps/pull/45

rinigus 2018-07-23 08:10

Re: [Announce] Native offline maps: OSM Scout Server
 
New release is out: 1.9.0

This release brings OSM Scout Server to D-Bus and makes it very simple to use map matching functionality in QML apps.

Via D-Bus, the server allows now to establish a session with it by the external application and receive current street name, speed limit and the speed assumed by the routing engine for cars.

For QML, I made a QML type that can utilize this functionality: https://github.com/rinigus/positions...mapmatched-qml . It is expected to be a drop in replacement for PositionSource in the code of the application. When map matching mode is requested, it will establish a contact with OSM Scout Server, wake it up via systemd if users have that feature enabled, and will start providing map matched coordinates and other data. To simplify publishing, I would suggest to copy PositionSourceMapMatched.qml into the source tree of the application and include it. Example usage is shown in https://github.com/rinigus/positions...ample/main.qml

PositionSourceMapMatched also simplifies app development by providing facilities for testing in the absence of GPS signal and timing information. See description in README and example.

As with regards to timing, when measured on 1+X (onyx) during ~20 min drive, it takes on average <40ms per single call from QML to map matching service via D-Bus, for processing its response, until the position and associated data are updated. The longest call over that test was <100ms.

The current DBus API can be extended to include other functionality of the server, if there is interest.

In addition, map matching support can be extended further to include other data provided by Valhalla. One example use is to utilize map matching to follow navigation progress. Valhalla's edges all have global IDs and one can just check the progress along the edge, the current edge, and the expected edge on the basis of expected route. While making navigation instructions processing efficient, that would make processing Valhalla's specific.

MartinK 2018-07-23 13:50

Re: [Announce] Native offline maps: OSM Scout Server
 
Wow, looks really nice & useful, thanks a lot! :)

carlosgonz 2018-07-23 16:47

Re: [Announce] Native offline maps: OSM Scout Server
 
thanks, just waiting the other half. modrana - whogo

rinigus 2018-07-26 07:39

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by carlosgonz (Post 1546478)
thanks, just waiting the other half. modrana - whogo

the PR for WhoGo is submitted. For modRana, its not very clear for me how to wire it in with its multi-platform approach. [just don't know enough modRana's guts to start operating there]

carlosgonz 2018-07-26 17:29

Re: [Announce] Native offline maps: OSM Scout Server
 
moved from whogo thread
Quote:

Originally Posted by rinigus (Post 1546605)
For Valhalla-based routers, menu of modes is here https://github.com/valhalla/valhalla...costing-models

For car, its possible. Please open an issue (https://github.com/otsaloma/whogo-maps) regarding router options, then we have a chance to remember it.

seems be auto_shorter option in valhalla, nice valhalla-router-engine. what version valhalla is running in osmscoutserver?

rinigus 2018-07-26 17:34

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by carlosgonz (Post 1546608)
moved from whogo thead

seems be auto_shorter option in valhalla, nice valhalla-router-engine. what version valhalla is running in osmscoutserver?

its 2.6.2 - on recommended stable branch. and yes, km saving goes through auto_shorter. osm scout server already supports them all, we need to add gui options in whogo maps and modrana for its support

carlosgonz 2018-07-26 17:51

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1546609)
its 2.6.2 - on recommended stable branch. and yes, km saving goes through auto_shorter. osm scout server already supports them all, we need to add gui options in whogo maps and modrana for its support

ok, yes will good this feature on osmscoutserver clients, valhalla 3.0.0 almost done. Thanks rinigus for maps & offline solutions to SF

rinigus 2018-07-31 12:56

Re: [Announce] Native offline maps: OSM Scout Server
 
Updated translations and mainly bugfixes are released as a part of 1.9.1.

One of the bugs - reported by persmule, thank you! - caused crash of the server for libosmscout users.

I have also added unidirectional street indicator and ferries to Mapbox GL map styles. Ferry line names will come with the next map import.

juiceme 2018-07-31 20:05

Re: [Announce] Native offline maps: OSM Scout Server
 
Hey @rinigus, you might want to update your donation methods.
As librepay went belly-up and they just refunded all my deposited credit that I was paying weekly to you...

carlosgonz 2018-07-31 20:39

Re: [Announce] Native offline maps: OSM Scout Server
 
me too, liberapay is rejecting accept Amex card, then i am looking to diferent label to truly support mr-rinigus

rinigus 2018-07-31 20:44

Re: [Announce] Native offline maps: OSM Scout Server
 
Thank you very much for support :) . Liberapay has been kicked out for some reason (unknown to me) from their payment processing platform and they look now for an alternative. So, let's wait a bit and see how (and if) they are going to resolve it. And its great that they sent money back - its not disappeared somewhere.

juiceme 2018-07-31 20:51

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by carlosgonz (Post 1546788)
me too, liberapay is rejecting accept Amex card, then i am looking to diferent label to truly support mr-rinigus

Liberapay died, something to do with their banking provider.
The odd thing about it is that they send me an email that they cannot continue to operate any longer, however on their website there is no mention about it.

carlosgonz 2018-07-31 20:58

Re: [Announce] Native offline maps: OSM Scout Server
 
I want to talk on libosmscount, seems not updated in the server, I would like to know the reason?. Also the name of the osmscountserver is too big, is out of layout of safish names.
thanks

carlosgonz 2018-07-31 21:12

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by juiceme (Post 1546790)
Liberapay died, something to do with their banking provider.
The odd thing about it is that they send me an email that they cannot continue to operate any longer, however on their website there is no mention about it.

bad news, i hope liberapay still keep going

rinigus 2018-08-01 06:45

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by carlosgonz (Post 1546793)
bad news, i hope liberapay still keep going

They are working on it, as I can see from Github issues. Patience is all what's needed at this stage :)

rinigus 2018-08-01 07:03

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by carlosgonz (Post 1546791)
I want to talk on libosmscount, seems not updated in the server, I would like to know the reason?. Also the name of the osmscountserver is too big, is out of layout of safish names.
thanks

Re libosmscout: I decided to maintain only the backends which I think are the best and which I am using personally. I believe that, without personal usage, I cannot monitor nor fix the issues occurring with the backends.

In december 2017, I asked for help in maintaining libosmscout in the corresponding developer list and most probably notified here as well. So far, nobody has stepped up and worked on upgrading the backend interface to the newer versions of libosmscout.

I have reorganized the code and import scripts to make libosmscout part independent from others - so all should be ready for its update if someone is interested. However, note that OSM Scout by Karry is progressing nicely and maybe the users of this backend should look into that application. Don't know the current status regarding routing and search though.

Re name OSM Scout Server: yes, its long. And there are historical reasons for that. However, rebranding is significant work and there should be initiative / reason for it (github and transifex repo, pointers at openrepos, docs, aggregation lists and such would have to be changed). And this work will have to be done instead of the development.

MartinK 2018-08-01 22:17

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1546798)
Re libosmscout: I decided to maintain only the backends which I think are the best and which I am using personally. I believe that, without personal usage, I cannot monitor nor fix the issues occurring with the backends.

In december 2017, I asked for help in maintaining libosmscout in the corresponding developer list and most probably notified here as well. So far, nobody has stepped up and worked on upgrading the backend interface to the newer versions of libosmscout.

For the record, In my efforts (at the moment rather slow due to summer) to package OSM Scout Server to Fedora I plan to skip the libosmscout based backend, at least for the first packaging round.

Quote:

Originally Posted by rinigus (Post 1546798)
Re name OSM Scout Server: yes, its long. And there are historical reasons for that. However, rebranding is significant work and there should be initiative / reason for it (github and transifex repo, pointers at openrepos, docs, aggregation lists and such would have to be changed). And this work will have to be done instead of the development.

I agree, while we got into an interesting situation by simple historic development, it's not really causing any actual issues AFAIK that would warrant a complicated rebranding exercise. :)

pichlo 2018-08-02 09:11

Re: [Announce] Native offline maps: OSM Scout Server
 
What's wrong with the name? OK, it's not very obvious and it took me a while to figure out that this is something I need but now I quite like it.

rinigus 2018-08-14 15:33

Re: [Announce] Native offline maps: OSM Scout Server
 
New version is out: 1.10.1

Starting from this version, I changed the license from LGPL3 to GPL3. I think it applies better, since the server is a regular application and there is no reason to use LGPL in this case.

For users, the main new feature is search along the route. Its been my intention to implement it from the beginning, just was postponing it from 0.1 till now. So, starting from 1.10.x, you can find all upcoming fuel stations, cafes and such in a specified distance from route. Works quite well - for a route of 900 km, it finds all cafes in 7 seconds on onyx. Corresponding PR has been submitted for WhoGo Maps. For modRana, I'll be happy to help and consult when @MartinK will find the time to deal with it (its mainly limited by my knowledge of modRana source).

To implement the search, I have added POST request support by the server. This allows the clients to push the data, such as route path, using JSON object in POST body. All the details are documented in accompanying README, sections https://github.com/rinigus/osmscout-....md#url-schema and https://github.com/rinigus/osmscout-...ition-or-route .

Finally, we have Italian translation coming up by @Watchmaker - thank you! Also, all other translations have been updated to their latest version, as for yesterday.

MartinK 2018-08-21 00:51

Re: [Announce] Native offline maps: OSM Scout Server
 
Quote:

Originally Posted by rinigus (Post 1547178)
For users, the main new feature is search along the route. Its been my intention to implement it from the beginning, just was postponing it from 0.1 till now. So, starting from 1.10.x, you can find all upcoming fuel stations, cafes and such in a specified distance from route. Works quite well - for a route of 900 km, it finds all cafes in 7 seconds on onyx. Corresponding PR has been submitted for WhoGo Maps. For modRana, I'll be happy to help and consult when @MartinK will find the time to deal with it (its mainly limited by my knowledge of modRana source).

I have concluded most of my summer related vacation activities and should thus be able to concentrate on modRana development yet again (and I've actually used it quite a bit while vacationing, even spotting a bug or two). :)

We can certainly "cook something up" with this new feature & I'm standing by to provide any help needed with modRana development, including documenting unclear stuff & fixing/improving weird API. :)

The first thing that comes to my mind for "search along route" is searching along the currently active route (if any). I'm sure there are more advanced options, but this seems like possibly a good start.

This could be implemented as a search screen similar to the screens currently used for location/local/Wikipedia search, possibly with some additional options:
- search origin (route start/current position on route (if any))
- search distance (full route, next N km; pending any OSM Scout Server limitations)

It's a bit unclear where to put these settings, as there are currently none for the existing search pages. WhoGo maps handles this by a page that triggers the search and switches the the results, which still seems a bit cumbersome to me & prevents easily returning to existing result listings. So a settings page for search-around-route easily accessible via a menu seems like a better match to me, it can easily accessed when needed but does not need to be traversed every time.

The could be the GUI part at a glance, but how does modRana actually request and return search results ?

The search interface is exposed to QML via the Qt5 GUI module (modules/gui_modules/gui_qt5/gui_qt5.py) and a search id string is used to differentiate what to search for. The Search class has a hardcoded list of functions to call for various search id strings.

In general the GUI asynchrously request a search and then gets a signal once search results are available.

All the search functions are currently provided by the (not well named in retrospective) onlineServices module (modules/mod_onlineServices/mod_onlineServices.py), even offline search calls handled by OSM Scout Server (hey, it goes over localhost, that's basically a network :P).

You may notice all the functions have the Async suffix - that's because they are asynchronous & the caller is supposed to supply a callback that will be called once results for the search query are available. The signature for the asynchronous local search function looks like this:

def localSearchAsync(self, term, callback, around=None, maxResults=32, sensor='false'):

There is the term to search for, the callback to call once results are available and some additional options (where to center the search, how many results to request, etc.).

If there are more providers available for the given search category, the *Async function decides which one to use and instantiates the asynchornous online/offline provider in a thread, returning the (unique) thread name, which can be in some cases useful to wait for the query to finish via the modRana ThreadManager interface.

As I don't thing we have any other suitable provider of search-along-route, the *Async function can be very simple, just always calling the OSM Scout Server provided search-along-route offline provider.

The offline providers live in modules/mod_onlineServices/offline_providers.py and inherit from the POIProvider class. The general idea is to implement a synchronous "search()" method and the POIProvider takes care of the the rest, including running the method in a thread and passing any arguments the *Async function passes to the searchAsync() function of the provider to the search() function.

To summarize, it could look like this:

1) QML GUI page requests search along route
2) Python Qt 5 GUI module resolves search function & registers a callback
3) onlineServices function searchAlongRoadAsync() instantiates the provider & feeds it data
4) the offline provider executes the query in its search() function, feeding OSM Scout Server the needed data & waits for reply
5) callback in the Qt 5 GUI module is called with results (including possible errors)
6) the QML GUI is notified about the results via Python -> QML signal & displays the results

That should be more or less it at a glance, please let me know if anything is unclear or even if you think the interface is bad and could be improved in some way. :)

rinigus 2018-08-23 12:49

Re: [Announce] Native offline maps: OSM Scout Server
 
Took longer than expected to reply, but I became rather stretched with the recent developments. Let's see if it was such a good idea to get into the client GUI software in full. But that I will know somewhat later, when the setting up and fixes are done. Anyway, I would expect to have even less time to code for modRana specifically.

Quote:

Originally Posted by MartinK (Post 1547324)
...

This could be implemented as a search screen similar to the screens currently used for location/local/Wikipedia search, possibly with some additional options:
- search origin (route start/current position on route (if any))
- search distance (full route, next N km; pending any OSM Scout Server limitations)

It's a bit unclear where to put these settings, as there are currently none for the existing search pages. WhoGo maps handles this by a page that triggers the search and switches the the results, which still seems a bit cumbersome to me & prevents easily returning to existing result listings. So a settings page for search-around-route easily accessible via a menu seems like a better match to me, it can easily accessed when needed but does not need to be traversed every time.
...

I have implemented the search along the route for WhoGo using the current road only. So, its active only if there is a road available. As for limitations, OSM Scout Server searches the full road with the search optionally from some position. Imagine driving along road and looking for upcoming gas stations. So, some options are needed to be specified.

To me, it looks like an additional Search on modRana Search screen, next to Local and Wikipedia. As for options, you would probably want to add some options for Local search as well, to get support for specifying type and name in OSM Scout Server options. Probably also distance from current location or road. There are not that many options, so, screen can be shared with the results, but still some.


All times are GMT. The time now is 05:55.

vBulletin® Version 3.8.8