maemo.org - Talk - How to download large maps with high resolution with mappero

maemo.org - Talk (https://talk.maemo.org/index.php)

- Applications (https://talk.maemo.org/forumdisplay.php?f=41)

- - How to download large maps with high resolution with mappero (https://talk.maemo.org/showthread.php?t=74415)

How to download large maps with high resolution with mappero

Say you want to go to a (large, as in Turkey for me) foreign country where mobile data is expensive, find out that Ovi maps (that can be downloaded like this - use the ones marked to work for 3.04) are of no use and turn to Mappero and Google Maps:cool:.

The problem is as follows. Mappero stores tiles by default in the folder /home/user/MyDocs/.maps/ which is a FAT32 partition. On N900, this means that each file takes up 64kB space no matter how small it is (some tiles Mapper downloads are just 177 bytes small, but they still take up 64kB). This information comes from here.

We will solve it by creating a loopback device (a filesystem in a file) that overcomes this limitation. We will be using ext2 filesystem - N900 wouldn't mount ext3 or ext4 for me. As we will be storing the file on N900 itself (and therefore on a FAT32 filesystem), the file can be only 4GB in size, but it should be hopefully enough. If you need more, buy a microsd card, use some saner filesystem then FAT32 on it and store the file there.

This can be (I guess) done on N900 itself, but I used my Ubuntu system (but any linux should do).
First create a file

Code:

touch .mappero_image

Then make it 4GB large (it is small enough to fit on FAT32)

Code:

dd if=/dev/zero of=.mappero_image bs=1024 count=4190208

Now create an ext2 filesystem on it:

Code:

mkfs.ext2 -i 1024 -L mappero_image_label -m 0 -b 1024 .mappero_image

(based on this and mkfs.ext2 man page)
-i switch specifies, that we will have one inode per each 1024 bytes. Since for every file we need one inode, this means we will be able to have around 4 million files in this device. This is needed because lots of files are smaller then 1024 bytes. The value could be set to 2048 or maybe even 4096 (to have 2M or 1M inodes respectively) which would probably free some more space, since inodes take up space too. You can play with that.

-L is a label, you can set it or ignore it

-m 0 specifies that superuser will not have any allocated space. He or she does not need the space anyway in this file, so we might as well use it.

-b specifies that a block size will be 1024. ext2 (or ext 3 or ext4) does not allow smaller block sizes.

Now copy .mappero_image (or whatever you call the file) to the device and mount it with

Code:

mount -o loop -t ext2 /path/to/.mappero_image /home/user/MyDocs/.maps

You may want to copy the files in .maps directory onto .mappero_image first - just mount it to some random directory, copy the things over and remount it. If mappero refuses to work, check permissions and that the file poi.db is present.

If everything works as it should, here is how we make it work automatically even after reboots. Since N900 recreates /etc/fstab on each boot, we cannot just put a line there. There are basically two ways - modify the script that autocreates fstab (which is scary, so we will not be doing that) or write a one-purpose upstart job, which is what we will do.
Create a file in /etc/event.d/ and put the following into it:

Code:

start on started rcS-late

script

        exec /etc/event.d/mount2

end script

This just starts script /etc/event.d/mount2, which we must create. The location and name of this script can of course be changed, that is what I used. Put this into the script:

Code:

#!/bin/sh

sleep 40

mount -t ext2 -o loop /home/user/MyDocs/.mappero_image /home/user/MyDocs/.maps

Do not forget to make the file executable!
The script just waits 40 seconds and then mounts the file. I am sure this can be done better, but I do not know hot to do actions in upstart based on what is mounted (ideally, it would work like "start when /home/user/MyDocs is mounted). But this works. If it does not, try to increase the sleep time (i.e. 30 did not work for me).

Now, we can proceed to downloading maps.
To download maps beforehand for later offline use, proceed as follows:

Quote:

It is indeed possible to load the maps in Mappero beforehand: while internet connectivity is available, start Mappero and scroll the map to the area you want to download the maps of. Try to zoom in/out so that only the area you are actually interested in is on the screen. Then, tap on the title bar, choose "Maps" -> "Manage Maps...", go to the "Zoom" tab and click the zoom levels you are interested in. If the number of maps is on the order of tens of thousands or more, you'd better deselect some zoom levels, or the operation might never end.

I usually download low resolution maps for the bigger areas, and the detailed maps (zoom levels 5 or 6) only for those areas where I know I'll actually be walking around.

If you already know your routes, you can also download the maps along the route (on the first tab of this dialog, on the bottom, you have a "Along route" option).

(by the author of mappero here)

If number of maps is large (more than 10 000 or so), N900 might start to be very unresponsive or appear to be dead. But it probably is not and just is very busy downloading the maps. You can check over SSH on how it is doing.

I found out that around 250 000 (most of Turkey on zoom 7, or area between 36° and 42° latitude and 30° and 45° longtitude) maps takes up about 450MB, which is about 15% of the file. It took 6 hours to download it (I think the bottleneck was N900 itself) so you might want to plug the device to an adapter because it is very battery intensive. So you could get around 1 500 000 maps onto single file I gues (but YMMV). If that is still not enough, I guess you can use more such files and symlink them in a smart way.

Please note that it seems at the moment that Google Satellite does not work with mappero in zooms below 8 r so. If it works for you, I would be very happy to hear about it. Also, if something does not work or could be done better, say so!

Update: fat filesystem allows lower block size so maybe it is actually better.

You create it with (dd command holds):

Code:

mkfs.vfat -I -n mappero_image_label -S 512 .mappero_image

Busybox mount is not smart enough to understand enough options to make the file writatable by mappero so download this, unpack it
(.debs are just archives) and put the file mount onto N900. Move it to /usr/bin/ and name it mount2 and make it executable (chmod +x mount2)

Put this instead of the version for ext2 to /etx/event.d/mount2 (I know, my naming sucks):

Code:

mount2 -t vfat -o gid=root,uid=user,loop,rw /path/to/.mappero_image /home/user/MyDocs/.maps

Re: How to download large maps with high resolution with mappero

Good idea, however did you perhaps consider the use of a sparse-file to keep the tile storage FS on? This would allow it to 'grow' upto 4GB without using unneeded space.

Something like:

Code:

dd if=/dev/zero of=.mappero_image bs=1024 seek=4190208 count=0

This would also save the need for creating a 4GB file THEN copying it to the device.

Re: How to download large maps with high resolution with mappero

Quote:

Originally Posted by vi_ (Post 1040628)

Good idea, however did you perhaps consider the use of a sparse-file to keep the tile storage FS on? This would allow it to 'grow' upto 4GB without using unneeded space.

Something like:

Code:

dd if=/dev/zero of=.mappero_image bs=1024 seek=4190208 count=0

This would also save the need for creating a 4GB file THEN copying it to the device.

I did consider a growing file but did not know how to do it. I will try this and update original post if it works (I hope it will:-)). Thanks.

Re: How to download large maps with high resolution with mappero

Quote:

Originally Posted by sup (Post 1040665)

I did consider a growing file but did not know how to do it. I will try this and update original post if it works (I hope it will:-)). Thanks.

Holy ****, fat32 does not support sparse files!

GGGGAAARRRGGGHHHH screw fat32 back to the last century.

Sorry, no can do!

Re: How to download large maps with high resolution with mappero

Quote:

Originally Posted by vi_ (Post 1040666)

Holy ****, fat32 does not support sparse files!

GGGGAAARRRGGGHHHH screw fat32 back to the last century.

Sorry, no can do!

Oh yeah, if it were not for FAT32, this would not be needed anyway...

Re: How to download large maps with high resolution with mappero

Oh man!

Why didn't You start this thread a few days earlier? :)

I fiddled around with that since last week and came to almost the same solution (on my own after long long heavy trials).

Here are some remarks:
I started also with ext3,then ext2 bs=4096, then 2048
but this was not ideal
du told me 4GB used and du --apparent-size said 2.5GB, so still 1.5GB lost
then I decided to use FAT (.maps is originally n a FAT drive and we do not need any rights) using a block size of 512
now du says 2.6G and du --apparent-size says 2.7G, so almost nothing lost

I had heavy problems with dd. My device always rebooted when making such heavy file transactions. Do not know why, but found using truncate helped me out (but still I had sudden reboots).

Yes, FAT does not support sparse files, a shame, right?

For the automatic start, this is absolutely perfect.
At first I also wanted to use rcS-late.
But I started using start on MOUNTS_OK, but this was triggered three times.
then switched to ke-recv, but also herein we do need some time delay, see

Code:

description "An upstart event file for Crash Reporter"

author "Eero Tamminen"



start on started ke-recv

stop on starting shutdown



pre-start script



# mmc(s) should be mounted after ke-recv has started, but currently

# started notification seems to be emitted before it's actually ready.

# this is a temporary workaround, we should fix this in ke-recv or c-r

# properly.



#        sleep 4

        sleep 30

end script

So, these are my 2 cents.

But nevertheless, thank You. ;)

Re: How to download large maps with high resolution with mappero

Quote:

Originally Posted by peterleinchen (Post 1044210)

so still 1.5GB lost
then I decided to use FAT

the lost freespace is something a lot of people forget is due to reserved blocks
use to tune2fs to reduce the reserved blocks percentage
tune2fs -m 0 <your partition>

Re: How to download large maps with high resolution with mappero

Folks, how do we use Mappero???

I was travelling last weekend and was in desperate need for a GPS device and my N900 FAILED!!! I had both the Ovi maps and Mappero and both couldn't work. The GPS could not even locate me (I was on a highway, and not in some desert or jungle).

Please, someone, how do we setup Mappero and some simple user guide to create destinations? Thanks.

Re: How to download large maps with high resolution with mappero

I've been using ModRana for the past year, since Mappero (last I looked) was no longer being maintained. It supports multiple download sources, and can store things in sqlite database format, which is both compressed, and gets around losing chunks of space on every tile based on FS granularity.

Also: Google has become quite a pain when it comes to tile cacheing. If it detects you're doing it (too many too quickly) it cuts you off. Silly, since it means you're probably going to lessen the load on their servers over time, but then they want to control when/where/if tiles update. :P

Thanks for the refresher/tutorial on creating loopback file systems though. I'm sure several people have uses for such things!

Re: How to download large maps with high resolution with mappero

Modrana looks nice, that is true and sqlite seems sweet. Just checking it out.