![]() |
How to download large maps with high resolution with mappero
Say you want to go to a (large, as in Turkey for me) foreign country where mobile data is expensive, find out that Ovi maps (that can be downloaded like this - use the ones marked to work for 3.04) are of no use and turn to Mappero and Google Maps:cool:.
The problem is as follows. Mappero stores tiles by default in the folder /home/user/MyDocs/.maps/ which is a FAT32 partition. On N900, this means that each file takes up 64kB space no matter how small it is (some tiles Mapper downloads are just 177 bytes small, but they still take up 64kB). This information comes from here. We will solve it by creating a loopback device (a filesystem in a file) that overcomes this limitation. We will be using ext2 filesystem - N900 wouldn't mount ext3 or ext4 for me. As we will be storing the file on N900 itself (and therefore on a FAT32 filesystem), the file can be only 4GB in size, but it should be hopefully enough. If you need more, buy a microsd card, use some saner filesystem then FAT32 on it and store the file there. This can be (I guess) done on N900 itself, but I used my Ubuntu system (but any linux should do). First create a file Code:
touch .mappero_image Code:
dd if=/dev/zero of=.mappero_image bs=1024 count=4190208 Code:
mkfs.ext2 -i 1024 -L mappero_image_label -m 0 -b 1024 .mappero_image -i switch specifies, that we will have one inode per each 1024 bytes. Since for every file we need one inode, this means we will be able to have around 4 million files in this device. This is needed because lots of files are smaller then 1024 bytes. The value could be set to 2048 or maybe even 4096 (to have 2M or 1M inodes respectively) which would probably free some more space, since inodes take up space too. You can play with that. -L is a label, you can set it or ignore it -m 0 specifies that superuser will not have any allocated space. He or she does not need the space anyway in this file, so we might as well use it. -b specifies that a block size will be 1024. ext2 (or ext 3 or ext4) does not allow smaller block sizes. Now copy .mappero_image (or whatever you call the file) to the device and mount it with Code:
mount -o loop -t ext2 /path/to/.mappero_image /home/user/MyDocs/.maps If everything works as it should, here is how we make it work automatically even after reboots. Since N900 recreates /etc/fstab on each boot, we cannot just put a line there. There are basically two ways - modify the script that autocreates fstab (which is scary, so we will not be doing that) or write a one-purpose upstart job, which is what we will do. Create a file in /etc/event.d/ and put the following into it: Code:
start on started rcS-late Code:
#!/bin/sh The script just waits 40 seconds and then mounts the file. I am sure this can be done better, but I do not know hot to do actions in upstart based on what is mounted (ideally, it would work like "start when /home/user/MyDocs is mounted). But this works. If it does not, try to increase the sleep time (i.e. 30 did not work for me). Now, we can proceed to downloading maps. To download maps beforehand for later offline use, proceed as follows: Quote:
If number of maps is large (more than 10 000 or so), N900 might start to be very unresponsive or appear to be dead. But it probably is not and just is very busy downloading the maps. You can check over SSH on how it is doing. I found out that around 250 000 (most of Turkey on zoom 7, or area between 36° and 42° latitude and 30° and 45° longtitude) maps takes up about 450MB, which is about 15% of the file. It took 6 hours to download it (I think the bottleneck was N900 itself) so you might want to plug the device to an adapter because it is very battery intensive. So you could get around 1 500 000 maps onto single file I gues (but YMMV). If that is still not enough, I guess you can use more such files and symlink them in a smart way. Please note that it seems at the moment that Google Satellite does not work with mappero in zooms below 8 r so. If it works for you, I would be very happy to hear about it. Also, if something does not work or could be done better, say so! Update: fat filesystem allows lower block size so maybe it is actually better. You create it with (dd command holds): Code:
mkfs.vfat -I -n mappero_image_label -S 512 .mappero_image (.debs are just archives) and put the file mount onto N900. Move it to /usr/bin/ and name it mount2 and make it executable (chmod +x mount2) Put this instead of the version for ext2 to /etx/event.d/mount2 (I know, my naming sucks): Code:
mount2 -t vfat -o gid=root,uid=user,loop,rw /path/to/.mappero_image /home/user/MyDocs/.maps |
Re: How to download large maps with high resolution with mappero
Good idea, however did you perhaps consider the use of a sparse-file to keep the tile storage FS on? This would allow it to 'grow' upto 4GB without using unneeded space.
Something like: Code:
dd if=/dev/zero of=.mappero_image bs=1024 seek=4190208 count=0 |
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
Holy ****, fat32 does not support sparse files! GGGGAAARRRGGGHHHH screw fat32 back to the last century. Sorry, no can do! |
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Oh man!
Why didn't You start this thread a few days earlier? :) I fiddled around with that since last week and came to almost the same solution (on my own after long long heavy trials). Here are some remarks: I started also with ext3,then ext2 bs=4096, then 2048 but this was not ideal du told me 4GB used and du --apparent-size said 2.5GB, so still 1.5GB lost then I decided to use FAT (.maps is originally n a FAT drive and we do not need any rights) using a block size of 512 now du says 2.6G and du --apparent-size says 2.7G, so almost nothing lost I had heavy problems with dd. My device always rebooted when making such heavy file transactions. Do not know why, but found using truncate helped me out (but still I had sudden reboots). Yes, FAT does not support sparse files, a shame, right? For the automatic start, this is absolutely perfect. At first I also wanted to use rcS-late. But I started using start on MOUNTS_OK, but this was triggered three times. then switched to ke-recv, but also herein we do need some time delay, see Code:
description "An upstart event file for Crash Reporter" But nevertheless, thank You. ;) |
Re: How to download large maps with high resolution with mappero
Quote:
use to tune2fs to reduce the reserved blocks percentage tune2fs -m 0 <your partition> |
Re: How to download large maps with high resolution with mappero
Folks, how do we use Mappero???
I was travelling last weekend and was in desperate need for a GPS device and my N900 FAILED!!! I had both the Ovi maps and Mappero and both couldn't work. The GPS could not even locate me (I was on a highway, and not in some desert or jungle). Please, someone, how do we setup Mappero and some simple user guide to create destinations? Thanks. |
Re: How to download large maps with high resolution with mappero
I've been using ModRana for the past year, since Mappero (last I looked) was no longer being maintained. It supports multiple download sources, and can store things in sqlite database format, which is both compressed, and gets around losing chunks of space on every tile based on FS granularity.
Also: Google has become quite a pain when it comes to tile cacheing. If it detects you're doing it (too many too quickly) it cuts you off. Silly, since it means you're probably going to lessen the load on their servers over time, but then they want to control when/where/if tiles update. :P Thanks for the refresher/tutorial on creating loopback file systems though. I'm sure several people have uses for such things! |
Re: How to download large maps with high resolution with mappero
Modrana looks nice, that is true and sqlite seems sweet. Just checking it out.
|
Re: How to download large maps with high resolution with mappero
Hm, I do not know. It does not download from google (mappero does), it cannot download based on gps coordinates (ie.e from 42° to 56° or something) and the interface is not as powerful (even though zooming with volume keys is nice, I miss that in mappero).
|
Re: How to download large maps with high resolution with mappero
Whats the best gps software please? (with voice nav)
|
Re: How to download large maps with high resolution with mappero
Quote:
Dow did you mount a vfat file so that mappero could write on it? I am trying to mess up with uid and guid options but I think N900's mount does not understand them:-(. |
Re: How to download large maps with high resolution with mappero
if i understand correctly this will just give you access to the maps but without routing ...i have never used it but but marble uses monav i believe which will also give access to offline routing as well as offline maps
|
Re: How to download large maps with high resolution with mappero
Mappero does routing, but must be online (it uses google).
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Hey sup,
standard mount is strong enough. I am mounting with: Code:
echo /home/user/MyDocs/.maps.vfat.img /home/user/MyDocs/.maps vfat loop,noauto,nodev,noexec,nosuid,noatime,nodiratime,utf8,uid=29999,shortname=mixed,dmask=000,fmask=0133,rodir 0 0 >> /etc/fstab second line mounts with Maemo standrad mount (busybox enhanced installed. but I think I installed it later and it worked with default bb mount also) But hey, if your mount2 works too ... Here is my full automatic start/mount in /etc/event.d/ Code:
description "starting my own mounts and python server for Nokia Maps" |
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
All you have to do is add a line to ke-recv to remount it when you disconnect a USB mass storage mode. |
Re: How to download large maps with high resolution with mappero
Hi,
I was just thinking about reformating /home/user/MyDocs/ partition to ext* because of the tile's size (also because of directory linking, permissions...). On n900 I have 9.4G of OSM tiles (du -h) the same on my PC with ext3 takes 1.4G and apparent size on PC is 1.1G And my PC has 4096 block size. Interesting. I did all in the first post on my PC (with 1024 blocks) and the first thing I see is that the empty 4G file reports 3G free space after mount (on PC). After tile-files copy - du says 1.2G and apparent size is 1.1G. So - (before copying this 4G file to n900), the main problem is the size 3 out of 4G. This I dont understand. It looks like one block takes 256bytes? EDIT: 4096 block says 3.8G free.... man page says inode takes 256bytes, you can change with I=128...testing it EDIT2: test of different block sizes, all with -I 128 inodes: 4096 block : avail 3.9G; size 1.4G apparent 1.1G (0.1G;+~30% waste; tot=~38%) 2048 block : avail 3.7G; size 1.2G apparent 1.1G (9%;+ 16% waste; tot=23.6%) 1024 block : avail 3.5G; size 1.2G apparent 1.1G (15%; +10% waste; tot=23.5%) For me, the 1024 or 2048 blocks are the same and I simply prefer 2048. After copy to n900 it seems to work. |
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
You see why I came to use vFAT? I did/do not see any advantages here for an ext FS. Just for storing map tiles.
4GB file ~ 4294967296 bytes Filesystem 1K-blocks Used Available Use% Mounted on /dev/loop0 4129759 3199535 930225 77% /home/user/MyDocs/.map so available are: 1k*4129759 ~ 4228873216 bytes Overhead seems to be only 66094080 bytes ~ 64MB My disk usage: du -xsh MyDocs/.maps 3.1G MyDocs/.maps du -xsh --apparent-size /home/user/MyDocs/.maps 2.9G /home/user/MyDocs/.maps to be more precise: du -xs MyDocs/.maps 3199535 MyDocs/.maps ~ 3276323840 bytes du -xs --apparent-size /home/user/MyDocs/.maps 3008856 /home/user/MyDocs/.maps ~3081068544 bytes So loss for storing around 3GB map tiles is only about 191MB. On ext there needs to be stored the inodes and also there is some space reserved for the super-user (man mkfs.ext2): -m reserved-blocks-percentage Specify the percentage of the filesystem blocks reserved for the super-user. This avoids fragmentation, and allows root-owned daemons, such as syslogd(8), to continue to function correctly after non-privileged processes are prevented from writing to the filesystem. The default percentage is 5%. 5% of 4GB is quite much (204MB) space lost on ext (excluding inodes space). --edit About blocks and inode size: 1k block size and -i bytes-per-inode Specify the bytes/inode ratio. mke2fs creates an inode for every bytes-per-inode bytes of space on the disk. The larger the bytes-per-inode ratio, the fewer inodes will be created. This value generally shouldn’t be smaller than the blocksize of the filesystem, since in that case more inodes would be made than can ever be used. Be warned that it is not possible to expand the number of inodes on a filesystem after it is created, so be careful deciding the correct value for this parameter. -i 1024 (and, the default, -I 256) will lead to this huge amount (1GB) of inode storage. |
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Hey,
which "script" do you refer to? The two lines you posted above? Of course you need to be root to be allowed to edit/modify /etc/fstab and also you need to be root for mount/umount. Install rootsh or sudser from extras-devel, but be aware that these kinds of modifications may brick your device (if you are a noob, as you described yourself ;)). If you are already root, then maybe you do not have created the file /home/user/MyDocs/.maps.vfat.img (the vfat storage file) and/or the directory /home/user/MyDocs/.maps (the mount point). |
Re: How to download large maps with high resolution with mappero
Yes, I was referring to that one ... but you fixed it anyways :-)
I deliberately deleted the folder .maps because I thought it would interfere with mounting the image file to that directory. Btw., how can I see that everything worked and .maps is a mounted file instead of an ordinary folder? And one other thing: Now that everything probably works as expected I cannot access MyDocs via USB any more. Only my SD card gets mounted on my windows machine :-( Apart from that I'm very happy that people like you come up with crazy ideas ;-) |
Re: How to download large maps with high resolution with mappero
OK, as you confirmed my problem with the USB mass storage mode, I do know that it is the mounted file on a (also) mounted device.
I am afraid we have to live with that, But there is a workaround: before putting N900 into mass storage mode, type in x-term Code:
sudo umount -l /home/user/MyDocs Code:
sudo mount /home/user/MyDocs/.maps |
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
/home/user/MyDocs/.mappero_disk 4182016 1685044 2496972 40% /home/user/MyDocs/.maps |
Re: How to download large maps with high resolution with mappero
Quote:
Power Search with "mount vi_" lead the way ;j http://talk.maemo.org/showthread.php?t=80562 Quote:
|
Re: How to download large maps with high resolution with mappero
Quote:
|
Re: How to download large maps with high resolution with mappero
No, not strange at all.
Just look at the first post's date sup 28th Jun 2011 , 10:29 vi_ 2nd Dec 2011 , 11:23 So, only "little" chance for you to detect that ;) |
Re: How to download large maps with high resolution with mappero
Oh yeah, I saw 12-02-11 - that crazy american way of indicating time will always puzzle me:-).
|
Re: How to download large maps with high resolution with mappero
I get /dev/loop0 4086976 8276 4078700 0% /home/user/MyDocs/.maps
Looks like it is working pretty well :) Hm, I think I will just copy things to my SD card and move it to internal memory if necessary :) |
Re: How to download large maps with high resolution with mappero
Quote:
1/ the rights - I want to have all odd-numbered subdirs for mappero readonly - to keep only every second zoom tile - with the actual zoom level the maps details are too faint to view (I didnt manage? under fat). 2/ I can imagine to link some more map directories from SD card anytime when needed. |
Re: How to download large maps with high resolution with mappero
OK.
Point 1 is not to beat (at all). If you like/need read/write access to set explicitly, there is no other chance than ext (1024 bs ?). The vFat image itself may be mounted write protected, but only the complete imgae (not down to single directories). And I do not mind anyone to use ext, my personal favorite is vFat. Point 2 may also be achieved with whatever file system you use. )ust map the single Map Providers as storage files (e.g. OSMI and OSMII) to as many directories you need under /home/user/MyDocs/.maps/xxx. |
All times are GMT. The time now is 01:48. |
vBulletin® Version 3.8.8