Active Topics
-
Sailfish OS for the Motorola Moto G7 Power (XT1955-5) - (ocean) (3)
to SailfishOS by edp17 - 12 hrs, 47 mins ago -
The N900 is now 15 years old! (6)
to Nokia N900 by Macros - 1 day, 16 hrs ago -
F(x)tec Pro1x (QWERTY) for sale (0)
to Buy & Sell by aln00000 - 2 days, 5 hrs ago -
n770 youtube video (0)
to General by nmdv - 6 days, 4 hrs ago - more...
| The Following 8 Users Say Thank You to MartinK For This Useful Post: | ||
|
2012-03-15
, 13:58
|
|
Posts: 1,548 |
Thanked: 7,510 times |
Joined on Apr 2010
@ Czech Republic
|
#962
|
Originally Posted by jafd
Oh, now I get it - good point.
The countries are not a problem. But usually you don't want blank data right after the state border, that's the point.
Originally Posted by jafd
Good luck!
I also have this idea of using a clustered setup. I also don't want my Twisted knowledge to rust. The company I'm working at announced an R&D lab for noncommercial, spinoff and hobby projects, so maybe someday I would talk them into building the most efficient OSM rendering farm in the world.
Originally Posted by jafd
What actually happened with it ? A few days ago it just vanished...
Seeing demise of t@h, OTOH, I'm a bit skeptical about this one.
Originally Posted by jafd
Oh, sorry, it should have been "default layer" not "detail layer" - I really need to read more carefully what I write.
I don't know a thing about layers. All tiles are single layer, with all details already burnt-in. The point is that in OSM default stylesheet has better contrast overall, and maps rendered with it have proper city/village/whatever boundaries rendered, which my maps don't.
Originally Posted by jafd
Interesting !
imposm, on the other hand, handles data much more efficiently on your average desktop/laptop hardware, my humble entry-level Thinkpad would crunch all of Europe in around 4 hours which is fast. The problem is, tables, tags and so on as produced by this tool, are totally different from what osm2pgsql provides.
So, in fact, I had to google "imposm osm.xml style compatible" and so on, getting irrelevant things at the top, until I've found OSM-bright and Carto. Then, the stylesheet produced makes what you see on the screenshot.
The OSM XML is so big that I don't have the faintest idea what to tweak so it would work.
Or maybe I would leave osm2pgsql running for a weekend with the latest Europe data, set up a cron job to incorporate differences as they flow in (osm2pgsql data allow this, imposm don't, AFAIK), and then render particular countries (so they have consistent data at the borders and so that anyone would be able to merge tilesets as he wishes).
Originally Posted by jafd
Well Navit manages realtime rendering from OSM data on quite lowend hardware, so it is in some way possible.
rendering would become so complex that your phone would die of overheating / battery drain very fast.
Originally Posted by jafd
Thats how most offline rendering/routing programs do it - they take an *.OSM file as input and give a (efficiently indexed) binary file as output. I'm don't know much about OSM PBF, but even if it could not be directly used, it should at least speed up parsing speed quite nicely.
Another way is to use OSM data (maybe in PBF format) to do the same, but I'm wondering how much study one has to invest into it, and how efficient would it be in the end. Wouldn't it be that in addition to quite big OSM datafiles there would need to be an even bigger index to efficiently search for needed data fragments?
Originally Posted by jafd
I would say that one should probably start with with off-line tile rendering first anyway. But realtime map rendering can be also nice, for things like adaptive labels that are not inverted when the map layer is rotated (IMHO Nokia Maps does that, but their map is also quite basic though).
Tiles are simple, on the other hand, and straightforward. You don't need to muck around with projections all the time, resorting instead to simple Cartesian arithmetic.
Originally Posted by jafd
Nice ! If it's backward compatible, I can merge it in right away.
By some clever use of normalization and a few lines patch for ModRana I have been able to slim dataset for the whole of Poland with zoom level up to 15 and select cities up to 18, from 4792M to 4589M. That's the whole 203 megabytes less, with no performance
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
| The Following User Says Thank You to MartinK For This Useful Post: | ||
|
|
2012-03-15
, 14:54
|
|
Posts: 242 |
Thanked: 97 times |
Joined on Sep 2009
|
#963
|
Originally Posted by MartinK
Hi, thanks for the reply, I have set main map and background map both as "google maps" and the tile appearing problem is now resolved. I will like to download these as offline so I have been trying to use the download feature, and its not working. On that screen where it says "total size of tiles is unknown (click to compute)" I tap the screen to start the calc process, it does start but never tells me the size, its always 0MB. Then if I tap the "Start" button for download, it always says "downloads failed".
There are several thinks to try:If even all of the above does not help, start modrana with:
- check general Internet connectivity - are sites in Microb loading, etc.
- try another map layer, in Menu->Options->Map->Map Layers
- turn off map overlay in Menu->Options->Map->Map overlay
- if nothing of the above helps, try to delete/rename the file options.bin in /home/user/.modrana/
And post/pastbin the output you get.Code:modrana
__________________
so i guess the the lesson learned is: "if you want a thing done well, do it yourself"
so i guess the the lesson learned is: "if you want a thing done well, do it yourself"
|
|
2012-03-15
, 16:03
|
|
Posts: 1,548 |
Thanked: 7,510 times |
Joined on Apr 2010
@ Czech Republic
|
#964
|
Originally Posted by abubakar
Unfortunately, Google doesn't really like batch downloading of their tiles and blocks tile downloads, if it detects too many requests in a short time.
Hi, thanks for the reply, I have set main map and background map both as "google maps" and the tile appearing problem is now resolved. I will like to download these as offline so I have been trying to use the download feature, and its not working. On that screen where it says "total size of tiles is unknown (click to compute)" I tap the screen to start the calc process, it does start but never tells me the size, its always 0MB. Then if I tap the "Start" button for download, it always says "downloads failed".
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
|
|
2012-03-20
, 11:30
|
|
Posts: 29 |
Thanked: 26 times |
Joined on Oct 2009
|
#965
|
So, I managed to get speedup (not giving any numbers, I was too lazy to recheck) for generate_tiles.py which comes with mapnik.
First, I've made it save tiles directly to SQLite database.
Second, I've made it check if a tile already exists before attempting to render it.
Third, for each tile render request, an area of (x+8, y+8) more tiles is rendered and saved into SQLite. This is vital because in the stock renderer, where map was 256x256, a gutter of 128 pixels was used so that labels are not mispositioned, which gives us actually a 512x512 image, 75% of which is discarded.
Fourth, I poked around map stylesheet and changed all that crazy light gray on white coloring to black.
After filling the data up, the normalization is done as follows: tile_id (integer) is added to lookup, and it references the ROWID of tiles table in the store. x, y, z and other columns beside ROWID and tile are dropped from the store. Also, I've seen no use for timestamp column, and dropped it altogether. As for store filename, I made it "store number" and use it to calculate the filename, like 'store.sqlite.'||store_number. All backward compatibility is achieved by using views.
I think matters could be improved further by having map size table in the lookup database, which would be used to translate planar coordinates into linear ones, and use those linear ones to be the tile_ids. Maybe even dropping x and y columns as a result.
So, on my todo list:
1) Make generate_tiles.py accept input files which tell what coordinates at what zoom levels should be generated;
2) Try even more space optimization tricks;
2.5) As a part of this, I suspect that some areas are still needlessly rendered and sliced, so investigate, it would save a few days' work;
3) Put the tile generator on github for all to see;
4) The generation process is very long and boring, so use urwid to visualise the progress in some way.
By the way, the most time-consuming task is PNG encoding. Unfortunately, we cannot encode first, then slice. However, we can employ some parallelizing tricks, as this task is very multiprocessing-friendly.
Last edited by jafd; 2012-03-20 at 11:37.
First, I've made it save tiles directly to SQLite database.
Second, I've made it check if a tile already exists before attempting to render it.
Third, for each tile render request, an area of (x+8, y+8) more tiles is rendered and saved into SQLite. This is vital because in the stock renderer, where map was 256x256, a gutter of 128 pixels was used so that labels are not mispositioned, which gives us actually a 512x512 image, 75% of which is discarded.
Fourth, I poked around map stylesheet and changed all that crazy light gray on white coloring to black.
After filling the data up, the normalization is done as follows: tile_id (integer) is added to lookup, and it references the ROWID of tiles table in the store. x, y, z and other columns beside ROWID and tile are dropped from the store. Also, I've seen no use for timestamp column, and dropped it altogether. As for store filename, I made it "store number" and use it to calculate the filename, like 'store.sqlite.'||store_number. All backward compatibility is achieved by using views.
I think matters could be improved further by having map size table in the lookup database, which would be used to translate planar coordinates into linear ones, and use those linear ones to be the tile_ids. Maybe even dropping x and y columns as a result.
So, on my todo list:
1) Make generate_tiles.py accept input files which tell what coordinates at what zoom levels should be generated;
2) Try even more space optimization tricks;
2.5) As a part of this, I suspect that some areas are still needlessly rendered and sliced, so investigate, it would save a few days' work;
3) Put the tile generator on github for all to see;
4) The generation process is very long and boring, so use urwid to visualise the progress in some way.
By the way, the most time-consuming task is PNG encoding. Unfortunately, we cannot encode first, then slice. However, we can employ some parallelizing tricks, as this task is very multiprocessing-friendly.
Last edited by jafd; 2012-03-20 at 11:37.
| The Following 3 Users Say Thank You to jafd For This Useful Post: | ||
|
|
2012-03-20
, 15:50
|
|
Posts: 29 |
Thanked: 26 times |
Joined on Oct 2009
|
#966
|
Some news from competition.
Installed MapDroyd on Android. They render vector data from some unbelievably efficient data files called MicroMap (http://www.onestepahead.de/index.php...omap&Itemid=62). Rendering is mindbogglingly fast. Maps take ridiculously little space. Borderlines look like sh*t at any zoom level: when far, they look like what you would draw on a whiteboard when asked to sketch a map of world. At close levels they look like what you would draw when asked to draw a map of world as precisely as you can with a pencil and no eraser.
Verdict: we still need ModRana on Android, like it or not.
Installed MapDroyd on Android. They render vector data from some unbelievably efficient data files called MicroMap (http://www.onestepahead.de/index.php...omap&Itemid=62). Rendering is mindbogglingly fast. Maps take ridiculously little space. Borderlines look like sh*t at any zoom level: when far, they look like what you would draw on a whiteboard when asked to sketch a map of world. At close levels they look like what you would draw when asked to draw a map of world as precisely as you can with a pencil and no eraser.
Verdict: we still need ModRana on Android, like it or not.
| The Following 2 Users Say Thank You to jafd For This Useful Post: | ||
|
|
2012-03-22
, 12:28
|
|
Posts: 1,548 |
Thanked: 7,510 times |
Joined on Apr 2010
@ Czech Republic
|
#967
|
Originally Posted by jafd
Nice !
So, I managed to get speedup (not giving any numbers, I was too lazy to recheck) for generate_tiles.py which comes with mapnik.
Originally Posted by jafd
Well, I'd like to eventually use the timestamp column (or file modification date if tiles are stored as files) for batch-tile update, so it should be available in the main tile database/cache.
After filling the data up, the normalization is done as follows: tile_id (integer) is added to lookup, and it references the ROWID of tiles table in the store. x, y, z and other columns beside ROWID and tile are dropped from the store. Also, I've seen no use for timestamp column, and dropped it altogether. As for store filename, I made it "store number" and use it to calculate the filename, like 'store.sqlite.'||store_number. All backward compatibility is achieved by using views.
On the other hand, if we consider the concept of tile read only tile-packs, it makes sense to gut every redundancy and make them as small & and as fast to generate as possible.
How it could look like
- each pack resides in the corresponding map layer directory, in its own folder, named with the "pack_" prefix, for example: "pack_poland"
- in its folder it has the optimized lookup db and numbered stores
- there are some additional meta-data (in a table in the lookup db or a separate file ?)
- created timestamp as a Unix epoch
- (approximate ?) bounding box (so that modRana knows which packs are relevant for the current fetch operation)
- pack format version
- other ?
- modRana detects the tile pack and uses tries to fetch tiles in this order: pack/-s, main tile cache, download
Originally Posted by jafd
Looking forward to it!
So, on my todo list:
1) Make generate_tiles.py accept input files which tell what coordinates at what zoom levels should be generated;
2) Try even more space optimization tricks;
2.5) As a part of this, I suspect that some areas are still needlessly rendered and sliced, so investigate, it would save a few days' work;
3) Put the tile generator on github for all to see;
4) The generation process is very long and boring, so use urwid to visualise the progress in some way.
By the way, the most time-consuming task is PNG encoding. Unfortunately, we cannot encode first, then slice. However, we can employ some parallelizing tricks, as this task is very multiprocessing-friendly.
Originally Posted by jafd
Its good to see that realtime map rendering is indeed quite doable.
Some news from competition.
Installed MapDroyd on Android. They render vector data from some unbelievably efficient data files called MicroMap (http://www.onestepahead.de/index.php...omap&Itemid=62). Rendering is mindbogglingly fast. Maps take ridiculously little space. Borderlines look like sh*t at any zoom level: when far, they look like what you would draw on a whiteboard when asked to sketch a map of world. At close levels they look like what you would draw when asked to draw a map of world as precisely as you can with a pencil and no eraser.
Verdict: we still need ModRana on Android, like it or not.
Unfortunately as even the map format is proprietary, there don't seem to be any directly usable components. Still, there are quite a few open geodata storage formats already:
OSM Protocol Buffer Format
Navit binary format
Monav binary format
the old Rana binary format
McNavi binary format
Garmin reverse engineered binary format
Many of these formats seems to have the similar concept of binary map data tiles "tiles", probably to the same advantages normal graphical map tiles have (easy coordinates -> tile x,y conversion; easy access to neighboring data, parallel processing, etc.). Well, there was even a binary tile data server for Rana.
There are also data sets already available in some of these formats hosted online (for Navit, Monav, even McNavi).
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
| The Following 3 Users Say Thank You to MartinK For This Useful Post: | ||
|
|
2012-03-22
, 18:03
|
|
Posts: 29 |
Thanked: 26 times |
Joined on Oct 2009
|
#968
|
Crap, I just have wasted a week of time trying to re-render tiles with the new stylesheet.
Guess what, nowhere in Mapnik docs (nor in __doc__ strings) does it say that Mapnik.Image.view() should be given (x, y, width, height) and not (x1, y1, x2, y2).
So I assumed the latter and only now, when the database all of a sudden grew to 90G at zoom level 15, did my suspicions rise. I checked a tile, and it was 3 megs in size.
Hence. I was ultimately wrong about PNG encoding taking the most time. Rendering does take the most time, unless you let bugs into your code, like I did.
Guess what, nowhere in Mapnik docs (nor in __doc__ strings) does it say that Mapnik.Image.view() should be given (x, y, width, height) and not (x1, y1, x2, y2).
So I assumed the latter and only now, when the database all of a sudden grew to 90G at zoom level 15, did my suspicions rise. I checked a tile, and it was 3 megs in size.
Hence. I was ultimately wrong about PNG encoding taking the most time. Rendering does take the most time, unless you let bugs into your code, like I did.
| The Following 2 Users Say Thank You to jafd For This Useful Post: | ||
|
|
2012-03-28
, 21:32
|
|
Posts: 1,548 |
Thanked: 7,510 times |
Joined on Apr 2010
@ Czech Republic
|
#969
|
Short progress update
ModRana source code on Github
I have migrated the source code from SVN to a Github hosted Git repository:
https://github.com/M4rtinK/modrana
Pull requests & patches are welcome !
NOTE: The TRAC on www.modrana.org is still fully functional and will be switched from SVN to a Git mirror of the Github repository in the coming days.
Where to submit new issues/bugreports/feature requests ?
I'll be watching both modrana.org and the Github issue pages. Please try not to post duplicates or at least link to the relevant issues on the other site.
Last edited by MartinK; 2012-03-28 at 21:34.
- configuration file upgrade support done
- confiuration files updated in regards to T@H quiting and OpenCycleMap url changing
- qtmobility support is working
- tile caching support for the QML GUI is about working, still doing some tweaking
- preview packages coming hopefully in a few days
ModRana source code on Github
I have migrated the source code from SVN to a Github hosted Git repository:
https://github.com/M4rtinK/modrana
Pull requests & patches are welcome !
NOTE: The TRAC on www.modrana.org is still fully functional and will be switched from SVN to a Git mirror of the Github repository in the coming days.
Where to submit new issues/bugreports/feature requests ?
I'll be watching both modrana.org and the Github issue pages. Please try not to post duplicates or at least link to the relevant issues on the other site.
__________________
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
Last edited by MartinK; 2012-03-28 at 21:34.
| The Following 8 Users Say Thank You to MartinK For This Useful Post: | ||
|
|
2012-03-29
, 14:32
|
|
Posts: 482 |
Thanked: 550 times |
Joined on Oct 2010
|
#970
|
Hooray! I have an N900 again and installed ModRana on it! It has come a long way. I do have a suggestion: when you search for POIs and you select one, there should be a "Route to Here" button on there, rather than having to go through two more layers of menus.
| The Following User Says Thank You to skykooler For This Useful Post: | ||
 |
| Tags |
| bada rox, martin_rocks, modrana, navigation, openstreetmap, the best, wehasgps |
«
Previous Thread
|
Next Thread
»
|
All times are GMT. The time now is 01:48.
So I disabled it, fixed tile loading and started to work on all other broken & missing features (POI, online routing, turn-by-turn navigation, batch tile download, map overlay & rotation, tracklog support etc).
The source code of the renderer (pyrender) is actually still inside modRana BTW. I took a quick look at the code after about two years - and it actually doesn't look that bad. If I understand it correctly it takes an *.osm file (it can even download the OSM data for given tiles), loads it to memory using an XML parser and then renders a set of predefined tags as points or (poly)lines. There seems to be even some support for textual labels. It uses Cairo or PIL (Python Imaging Library) as the drawing backend.
Looks like that it might be a quite good base for a simple map renderer + there is a lot of room for various optimizations (pre-filtering OSM files, marshaling/caching parsed XML data, QPainter backend,...). Any interested developers ?
When I'm at it, I'll also take a closer look at the Kothic realtime python map renderer.
modRana: a flexible GPS navigation system
Mieru: a flexible manga and comic book reader
Universal Components - a solution for native looking yet component set independent QML appliactions (QtQuick Controls 2 & Silica supported as backends)
Last edited by MartinK; 2012-03-15 at 13:30.