Notices
Page 80 of 207 « First 707879 80 818290 Last »
Reply
Thread Tools
beermad is offline beermad
2011-07-26 , 23:57
Posts: 21 | Thanked: 7 times | Joined on Jul 2010 @ Ipswich, England
#791
Hi Martin,

From looking through the database code and structure to produce the migration script, I think I've found a few points where the code and schema could be more efficient.
  • When you're retrieving tiles, you're pulling the epoch_timestamp from the databases. Except when you're checking if you need (or want) to download an updated tile, this is unnecessary, so just wastes CPU cycles.
  • After retrieving details from the lookup table, it seems inefficient to find the right tile in the store table using x,y,z when you've already done that on the lookup table. It would be more efficient to have a single primary index in the store table, referenced in the lookup table entry.
  • Moving on logically from the previous idea, it might be possible to make the lookup table more efficient. At the moment you need to work through three indexes (x,y,z) to find the right tile. It might be more processor efficient to have a single index on this table which is either a combination of the x,y,z parameters or a hash based on them (neither option is simple, but would probably speed things up a worthwhile amount).
 

The Following 2 Users Say Thank You to beermad For This Useful Post:
白い熊 is offline 白い熊
2011-07-30 , 16:13
Posts: 451 | Thanked: 334 times | Joined on Sep 2009
#792
Originally Posted by MartinK View Post
ModRana now overlays the two tiles only once and then caches the result & displays the result. Before the tiles are combined, only the loading tile is shown.

Like this, when tiles from one layer are missing, nothing is shown.
This can be IMO improved:
  • just showing the "good" layer
  • overlaying with the error tile to inform the user that one layer is broken
I'd vote for the 2nd option, that way you know one layer is missing...
 
白い熊 is offline 白い熊
2011-07-30 , 16:13
Posts: 451 | Thanked: 334 times | Joined on Sep 2009
#793
Originally Posted by MartinK View Post
Thanks for reporting! I'll check it out - looks like a Unicode conversion error to me.
Same problem with Japanese POIs.
 
白い熊 is offline 白い熊
2011-07-30 , 19:10
Posts: 451 | Thanked: 334 times | Joined on Sep 2009
#794
When downloading a large area, modRana segfaults. Thought it was only on the N900, but the PC version exhibits the same behavior.

When trying to download a 40km area, I get an estimated download of 90 thousand tiles, modRana segfaults after cca 10 thousand tiles with:

Code:
Batch tile dl working... (threads: 8) pending: 46496, done: 43730
Batch tile dl working... (threads: 8) pending: 46489, done: 43737
Batch tile dl working... (threads: 8) pending: 46482, done: 43744
*** glibc detected *** python: corrupted double-linked list: 0x0bbafbe8 ***
======= Backtrace: =========
/lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x6aac1)[0xb75acac1]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(+0x6c3ec)[0xb75ae3ec]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(cfree+0x6d)[0xb75b13dd]
/usr/lib/libsqlite3.so.0(+0x15c80)[0xb623fc80]
/usr/lib/libsqlite3.so.0(sqlite3_free+0x81)[0xb6230a51]
/usr/lib/libsqlite3.so.0(+0x4c1c2)[0xb62761c2]
/usr/lib/libsqlite3.so.0(sqlite3_close+0x11f)[0xb627635f]
/usr/lib/python2.6/lib-dynload/_sqlite3.so(pysqlite_connection_close+0xfd)[0xb62d315d]
python(PyEval_EvalFrameEx+0x44d0)[0x80e08c0]
python(PyEval_EvalCodeEx+0x848)[0x80e2968]
python(PyEval_EvalFrameEx+0x476f)[0x80e0b5f]
python(PyEval_EvalCodeEx+0x848)[0x80e2968]
python(PyEval_EvalFrameEx+0x476f)[0x80e0b5f]
python(PyEval_EvalCodeEx+0x848)[0x80e2968]
python[0x8174d87]
python(PyObject_Call+0x4a)[0x8066d7a]
python(PyEval_EvalFrameEx+0x3133)[0x80df523]
python(PyEval_EvalFrameEx+0x5437)[0x80e1827]
python(PyEval_EvalFrameEx+0x5437)[0x80e1827]
python(PyEval_EvalCodeEx+0x848)[0x80e2968]
python[0x8174c9c]
python(PyObject_Call+0x4a)[0x8066d7a]
python[0x806f931]
python(PyObject_Call+0x4a)[0x8066d7a]
python(PyEval_CallObjectWithKeywords+0x42)[0x80dc2f2]
python[0x8113faf]
/lib/i386-linux-gnu/i686/cmov/libpthread.so.0(+0x5c39)[0xb78a7c39]
/lib/i386-linux-gnu/i686/cmov/libc.so.6(clone+0x5e)[0xb760e93e]
======= Memory map: ========
08048000-08235000 r-xp 00000000 08:04 20017      /usr/bin/python2.6
08235000-08284000 rwxp 001ed000 08:04 20017      /usr/bin/python2.6
08284000-0828d000 rwxp 00000000 00:00 0 
08b3c000-0bbe4000 rwxp 00000000 00:00 0          [heap]
af4fe000-af4ff000 ---p 00000000 00:00 0 
af4ff000-afcff000 rwxp 00000000 00:00 0 
afcff000-afd00000 ---p 00000000 00:00 0 
afd00000-b0500000 rwxp 00000000 00:00 0 
b0500000-b05ff000 rwxp 00000000 00:00 0 
b05ff000-b0600000 ---p 00000000 00:00 0 
b0b22000-b0d23000 rwxp 00000000 00:00 0 
b0e24000-b0e25000 ---p 00000000 00:00 0 
b0e25000-b1bfe000 rwxp 00000000 00:00 0 
b1cb1000-b23ff000 rwxp 00000000 00:00 0 
b23ff000-b2400000 ---p 00000000 00:00 0 
b2400000-b2c00000 rwxp 00000000 00:00 0 
b2c00000-b2d00000 rwxp 00000000 00:00 0 
b2d7d000-b2d7e000 ---p 00000000 00:00 0 
b2d7e000-b357e000 rwxp 00000000 00:00 0 
b357e000-b3582000 r-xp 00000000 08:04 16174      /lib/i386-linux-gnu/i686/cmov/libnss_dns-2.13.so
b3582000-b3583000 r-xp 00004000 08:04 16174      /lib/i386-linux-gnu/i686/cmov/libnss_dns-2.13.so
b3583000-b3584000 rwxp 00005000 08:04 16174      /lib/i386-linux-gnu/i686/cmov/libnss_dns-2.13.so
b35b1000-b366c000 rwxp 00000000 00:00 0 
b366c000-b382b000 rwxp 00000000 08:04 19603      /usr/share/file/magic.mgc
b382b000-b38e6000 rwxp 00000000 00:00 0 
b38e6000-b397c000 r-xp 00000000 08:04 65546      /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans-Oblique.ttf
b397c000-b3a21000 r-xp 00000000 08:04 63682      /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans-Bold.ttf
b3a21000-b3b97000 rwxp 00000000 00:00 0 
b3c00000-b3cb4000 rwxp 00000000 00:00 0 
b3cb4000-b3d00000 ---p 00000000 00:00 0 
b3d7b000-b3d97000 r-xp 00000000 08:04 14758      /lib/i386-linux-gnu/libgcc_s.so.1
b3d97000-b3d98000 rwxp 0001b000 08:04 14758      /lib/i386-linux-gnu/libgcc_s.so.1
b3d98000-b3d99000 ---p 00000000 00:00 0 
b3d99000-b50bd000 rwxp 00000000 00:00 0 
b50bd000-b50dc000 r-xp 00000000 08:04 25356      /usr/lib/libjpeg.so.62.0.0
b50dc000-b50dd000 rwxp 0001e000 08:04 25356      /usr/lib/libjpeg.so.62.0.0
b50f0000-b50f4000 r-xp 00000000 00:14 1386718    /media/0/kumatux.org-apps/gdk-pixbuf_2.23.3_jj_2011-04-27-002509_debian-unstable_2.6.38-1-686_gcc-4.6.0_i686_macbook-pro-5.2/lib/gdk-pixbuf-2.0/2.10.0/loaders/li
bpixbufloader-jpeg.so                                                                                                                                                                                            
b50f4000-b50f5000 rwxp 00004000 00:14 1386718    /media/0/kumatux.org-apps/gdk-pixbuf_2.23.3_jj_2011-04-27-002509_debian-unstable_2.6.38-1-686_gcc-4.6.0_i686_macbook-pro-5.2/lib/gdk-pixbuf-2.0/2.10.0/loaders/li
bpixbufloader-jpeg.so                                                                                                                                                                                            
b50f5000-b51a5000 r-xp 00000000 08:04 63683      /usr/share/fonts/truetype/ttf-dejavu/DejaVuSans.ttf
b51a5000-b51a7000 r-xp 00000000 08:04 32692      /usr/lib/pango/1.6.0/modules/pango-basic-fc.so
b51a7000-b51a8000 rwxp 00001000 08:04 32692      /usr/lib/pango/1.6.0/modules/pango-basic-fc.so
b51a8000-b51a9000 r-xs 00000000 08:04 261540     /var/cache/fontconfig/c05880de57d1f5e948fdfacc138775d9-le32d4.cache-3
b51a9000-b51af000 r-xs 00000000 08:04 271570     /var/cache/fontconfig/945677eb7aeaf62f1d50efc3fb3ec7d8-le32d4.cache-3
b51af000-b51b1000 r-xs 00000000 08:04 261521     /var/cache/fontconfig/ea47318ec9849e1a71e80a5d69d13859-le32d4.cache-3
b51b1000-b51b2000 r-xs 00000000 08:04 261402     /var/cache/fontconfig/e3fa16a14183b06aa45b3e009278fd14-le32d4.cache-3
b51b2000-b51b4000 r-xs 00000000 08:04 261398     /var/cache/fontconfig/b5ea634b0fb353b8ea17632d1f9ef766-le32d4.cache-3
b51b4000-b51b8000 r-xs 00000000 08:04 261360     /var/cache/fontconfig/6eb3985aa4124903f6ff08ba781cd364-le32d4.cache-3
b51b8000-b51bc000 r-xs 00000000 08:04 261351     /var/cache/fontconfig/515ca1ebc4b18308bea979be5704f9db-le32d4.cache-3
b51bc000-b51c3000 r-xs 00000000 08:04 271552     /var/cache/fontconfig/6d41288fd70b0be22e8c3a91e032eec0-le32d4.cache-3
b51c3000-b51df000 r-xs 00000000 08:04 32186      /usr/share/mime/mime.cache
b51df000-b51e0000 ---p 00000000 00:00 0 
b51e0000-b59e0000 rwxp 00000000 00:00 0 
b59e0000-b59f9000 r-xp 00000000 08:04 19598      /usr/lib/libmagic.so.1.0.0
b59f9000-b59fa000 rwxp 00018000 08:04 19598      /usr/lib/libmagic.so.1.0.0
b59fa000-b59fc000 rwxp 00000000 00:00 0 
b59fc000-b59fe000 r-xs 00000000 08:04 261325     /var/cache/fontconfig/ddd4086aec35a5275babba44bb759c3c-le32d4.cache-3
b59fe000-b5a0b000 r-xs 00000000 08:04 261341     /var/cache/fontconfig/d52a8644073d54c13679302ca1180695-le32d4.cache-3
b5a0b000-b5a0e000 r-xp 00000000 08:04 19969      /usr/lib/python2.6/lib-dynload/_json.so
b5a0e000-b5a0f000 rwxp 00002000 08:04 19969      /usr/lib/python2.6/lib-dynload/_json.so
b5a0f000-b5a26000 r-xp 00000000 08:04 19939      /usr/lib/python2.6/lib-dynload/_ctypes.so
b5a26000-b5a29000 rwxp 00016000 08:04 19939      /usr/lib/python2.6/lib-dynload/_ctypes.so
b5a29000-b5a2a000 ---p 00000000 00:00 0 
b5a2a000-b622a000 rwxp 00000000 00:00 0 
b622a000-b62b5000 r-xp 00000000 08:04 17124      /usr/lib/libsqlite3.so.0.8.6
b62b5000-b62b8000 rwxp 0008a000 08:04 17124      /usr/lib/libsqlite3.so.0.8.6
b62b9000-b62bf000 r-xs 00000000 08:04 271525     /var/cache/fontconfig/e13b20fdb08344e0e664864cc2ede53d-le32d4.cache-3
b62bf000-b62c1000 r-xs 00000000 08:04 261345     /var/cache/fontconfig/7ef2298fde41cc6eeb7af42e48b7d293-le32d4.cache-3
b62c1000-b62c5000 r-xp 00000000 00:14 1386700    /media/0/kumatux.org-apps/gdk-pixbuf_2.23.3_jj_2011-04-27-002509_debian-unstable_2.6.38-1-686_gcc-4.6.0_i686_macbook-pro-5.2/lib/gdk-pixbuf-2.0/2.10.0/loaders/li
bpixbufloader-png.so                                                                                                                                                                                             
b62c5000-b62c6000 rwxp 00003000 00:14 1386700    /media/0/kumatux.org-apps/gdk-pixbuf_2.23.3_jj_2011-04-27-002509_debian-unstable_2.6.38-1-686_gcc-4.6.0_i686_macbook-pro-5.2/lib/gdk-pixbuf-2.0/2.10.0/loaders/li
bpixbufloader-png.soAborted
 
白い熊 is offline 白い熊
2011-07-30 , 19:58
Posts: 451 | Thanked: 334 times | Joined on Sep 2009
#795
Well, the crashes seem to be arbitrary... After a series of crashes at roughly 10 thou, I've had a download running now, which is 50 thou and going...
 
kureyon is offline kureyon
2011-07-31 , 05:14
Posts: 992 | Thanked: 738 times | Joined on Jun 2010 @ Low Earth Orbit
#796
1) Track Logs: Any particular reason why they're stored under /opt? Also their file permissions are owned by root.

May I suggest that all user "generated" data and maybe config files are stored in the home directory or in MyDocs.

2) Another thing is modRana doesn't seem to reload the tracklogs directory. Eg I export a track from GPSJinni and copy it to the tracklogs directory whilst modRana is running. I need to restart modRana before it sees the copied file.

Thanks!
 
白い熊 is offline 白い熊
2011-07-31 , 09:41
Posts: 451 | Thanked: 334 times | Joined on Sep 2009
#797
This downloading is a serious problem. I'm trying to download a 40km area just at the lowest level of zoom that OSM will go to, that's around 350k tiles, but it keeps crashing at 15k, which I already have.

How to get around it?

Tried switching it to just save the tiles, so that I'd later import them into the sql db, but the same thing, just bombs, WTF?
Last edited by 白い熊; 2011-07-31 at 09:45.
 
MartinK is offline MartinK
2011-07-31 , 13:07
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#798
Originally Posted by kureyon View Post
1) Track Logs: Any particular reason why they're stored under /opt? Also their file permissions are owned by root.
This was for legacy reasons - Rana, the modRana predecessor was developed for the Neo FreeRunner, where everything run under root. So it didn't matter where they were stored.

Originally Posted by kureyon View Post
May I suggest that all user "generated" data and maybe config files are stored in the home directory or in MyDocs.
Well, I've just sent V0.27-2 to the autobuilder, that does just this
check out the updated data storage article for details about the new paths.
In short:
  • map data - nothing changes, still MyDocs/.maps
  • POI- nothing changes, still MyDocs/.maps
  • tracklogs are now stored in MyDocs/tracklogs, old tracklogs should be migrated there automatically
  • configuration files are now in ~/.modrana

Originally Posted by kureyon View Post
2) Another thing is modRana doesn't seem to reload the tracklogs directory. Eg I export a track from GPSJinni and copy it to the tracklogs directory whilst modRana is running. I need to restart modRana before it sees the copied file.
Thanks!
OK, I'll change it to reload the directory once it is listed from the GUI.

Originally Posted by 白い熊 View Post
Same problem with Japanese POIs.
I'm looking at this right now.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
 

The Following User Says Thank You to MartinK For This Useful Post:
MartinK is offline MartinK
2011-07-31 , 13:56
Posts: 1,548 | Thanked: 7,510 times | Joined on Apr 2010 @ Czech Republic
#799
Originally Posted by beermad View Post
Hi Martin,

From looking through the database code and structure to produce the migration script, I think I've found a few points where the code and schema could be more efficient.
  • When you're retrieving tiles, you're pulling the epoch_timestamp from the databases. Except when you're checking if you need (or want) to download an updated tile, this is unnecessary, so just wastes CPU cycles.
Good catch.
Originally Posted by beermad View Post
  • After retrieving details from the lookup table, it seems inefficient to find the right tile in the store table using x,y,z when you've already done that on the lookup table. It would be more efficient to have a single primary index in the store table, referenced in the lookup table entry.
Well, I'm not really a database architect - I basically just thought quite a long time about how to make an universal schema and asked a friend who works with databases a bit. Well, like this, the lookup and store databases are independent - so when the lookup one is corrupted, it should be possible to regenerate it just form the stores.
Originally Posted by beermad View Post
  • Moving on logically from the previous idea, it might be possible to make the lookup table more efficient. At the moment you need to work through three indexes (x,y,z) to find the right tile. It might be more processor efficient to have a single index on this table which is either a combination of the x,y,z parameters or a hash based on them (neither option is simple, but would probably speed things up a worthwhile amount).
Yeah - say we have x=1 y=2 and z=17 -> 1217, also x=1,y=21,z=7 -> 1217 ...
A hash with separators might work though: 1,2,17 vs 1,21,7 - would something like this be usable ?

Also, would it be possible to maintain backward compatibility by adding this new indexes and still storing the old info ? Converting all the existing database files users might already have would be quite a headache and also some developers might be already working on supporting the format in its current form (IIRC the CloudGPS developer, maybe also some others).

There is a version filed in the schema, so it would be possible to do something like this:
  • 1 = current version
  • 2 = old info + new indexes
  • 3 = just new indexes


Originally Posted by 白い熊 View Post
This downloading is a serious problem. I'm trying to download a 40km area just at the lowest level of zoom that OSM will go to, that's around 350k tiles, but it keeps crashing at 15k, which I already have.
I remember getting this some time ago. I'll run a few large downloads to check if I can still reproduce it.
Still, this seems more like a bug in Glib that is being triggered by modRana. There is even a post mentioning a similar behaviour (glib and long lists).

Originally Posted by 白い熊 View Post
How to get around it?
First, try the new version that was just released - some unrelated changes might have fixed it.
If this does not help, you can try some other batch download software, like Gmapcatcher and then importing the tiles with the SQLite import script.

Originally Posted by 白い熊 View Post
Tried switching it to just save the tiles, so that I'd later import them into the sql db, but the same thing, just bombs, WTF?
Interesting - looks like it is independent from the storage method. So the long lists + threaded access to them might really be the issue.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
 
lsolano is offline lsolano
2011-07-31 , 15:16
Posts: 235 | Thanked: 163 times | Joined on Dec 2008 @ Costa Rica
#800
I think it would be better to store everything under MyDocs/Modrana.

So, we'd have:

MyDocs/Modrana/tracklogs
MyDocs/Modrana/pois
MyDocs/Modrana/config (not sure about this one, maybe it is better ~/.modrana)

Of course, maps should be kept at MyDocs/.maps to share it with other applications.
 
Reply
Page 80 of 207 « First 707879 80 818290 Last »

Tags
bada rox, martin_rocks, modrana, navigation, openstreetmap, the best, wehasgps

« Previous Thread | Next Thread »
Thread Tools

 
Forum Jump


All times are GMT. The time now is 20:51.

Contact Us - Home - Archive - Top