Notices

Not quite happy with neither MM1.x nor MM2.x map repository format, I decided to write my own one. It went on and off for more than a month, now it's basically done. BTW I ported MM1.4.7 to os2008 during the process, what an achievement! This is the first time I did coding on Linux platform, kind of brainstorming, lots of pain and fun as well.

The new map repository format is very similar to MM1.x one, but instead of storing one map tile in one file, the new system stores up to 8 map tiles in one file(the number can be changed, very scalable). In this way the disk file space is reduced significantly but the access speed is not compromised.

Compared with MM1.x repository format: The waste disk space is reduced to about 1/10 of the original MM1.x format, but access speed maintains the same or even faster(since it can read up to 8 map tiles at one time). Typically MM1.x system requires about 400-500MB disk space to store about 100MB map tiles on a SD card(32k allocation unit for SDHC card). Well the new map repository needs only about 120-130MB disk space.

Compared with MM2.x repository format: The access speed is greatly enhanced. Also the new format removes the 2G byte limit on a GDBM dbase file. The map repository size is limited only by the SD card capacity.

To demonstrate the new map repository file system, I modified the MM2.2 and MM1.4.7, made them available for both os2006-2007 and os2008 platforms. To port MM1.4.7 to os2008, the internet utility library is changed from osso-ic to conic. Now you have the Maemo Mapper 1.x and 2.x working on all platforms, and they have the same map file system. The new repository uses similar file structure as original MM1.4.7(z/x/y), but the map tile file name is changed from .jpg to .map, also changed is the numbering for directories and files. Their numbers are not 1-to-1 match with the original repository.

You can go to the download page to get modified Maemo Mapper executables.
http://code.google.com/p/mmpcmapper/downloads/list

Also a PC utility tool is developed to convert both the original MM1.x or MM2.x map repository format to the new one:
http://www.internettablettalk.com/fo...269#post144269


The new software will not change or erase anything of the original data and setup parameters, except it will create a directory in MM2.x card to store the map tiles. In MM1.x the POI icon may not work since the icon directory is changed to the same as the MM2.x structure, the icon files locate in the same directory of poi.db.

The MM2.2 Mod 1 includes all the features up to official MM2.3.1 release, removed parts are GDBM file system and in-memory cache. I found there is not much difference in speed between MM2.3.1 and M2.2 Mod, due to improved access time of the new file system. So the in-memory cache needs not to be implemented, however the new file system does use about 3M byte as the in-memory cache. To change from your official MM2.x to the Mod version, all your data on your SD card will be kept untouched. A new directory similar to M1.xx file structure will be created to store the map repository. It's recommended to backup your GDBM file on PC and delete it from the SD card to save the disk space.

The MM1.4 Mod 1 includes all the features up to official MM1.4.7 release, only file format is changed. Note that for os2006-os2007 version, the old osso-ic internet library is still used to keep backward compatibility, in os2008 version it uses the new conic library. But you actually can compile the os2008 version source code on os2006-os2007 platofrm, it works as well. The Mod version also changes the POI directory(POI icon directory), now the POI directory has a consistant location for both MM1.4 or MM2.x Mod versions. To change from your MM1.4.7 to the Mod version, it's suggested to backup your files to PC and delete your old map repository. The new file system will not erase any of your data but they will mix up with the original MM1.x file system, since they're quite similar.

Nutter,

Looking forward to trying this soon. What do you recommend as the optimal allocation unit for the new repository format? I believe I used 1024b with MM1.4 on the 770, and that worked fairly well. Note that most of the maps will be Google Satellite types, so they're a bit larger than the street versions.

Thanks!

It's optimized for 32k allocation unit for the best speed and balanced waste space. The purpose I did that is I am still using MM1.4, and it consumes quite astonishing disk space. MM2.2 is slow unless you remove the rotation algorithm, which requires quite CPU resources.

If you use 1024B as the allocation unit, basically you don't have space wasted under this format.

Nutter,

It looks as if the standard 32KB allocation unit for FAT32 is far from the best choice for the new format.

I took my .db file for VEStreet (55.9MB), and converted it using MapsIn1 on my laptop. This resulted in 9059 files for 49.7MB that took up 71.4MB on an NTFS drive. 58869 files were copied from the .db.

Then I copied them over to the SD card, formatted as FAT32 with a default 32KB allocation unit. Same number of files, but they now consume 296MB of disk space. There are still LOTS of little files. 4459 files are less than 2KB.

I'm guessing my files are taking up a lot more room because they are not for a big contiguous area, rather they are scattered along roads or coastlines.

I'm guessing I should just bite the bullet and reformat the card with a smaller allocation unit, probably the same 1024KB I was using before.

Do you agree?

Now we're cooking!

I'll give it a go tonight. It sounds like I need to use you're modded MM2.2 executable then use your converter to convert my 4.78 GB of MM 1 maps for use with this new set-up...

Tre cool dude! The most attractive program that can run on my N800 is MM running hybrid satellite maps. It gets everyones attention. Especially if I'm driving using GPS... It looks like you're flying over the areas you're traveling through...

The first thing everybody tries when they see it is to look up their own address to get a view of their town. Unfortunately with the previous 2GB limitation, 9 out of 10 times I didn't have their town loaded. This got the "What good is it then?" eye roll and they handed it back to me.

If your mods works out, I can now load a large chunk (NJ, DE, half of PA, and NYC) on the sucker and it will blow the TomTom's and Garmins out of the water.

1.4 it is for me... It rocks on 2008. Thanks again!

****

Dude1: My TomTom can take me right to my house address!

Dude2: Well, my Garmin can tell me what side of the street your house is on!!

Me (The Dude): Oh yeah. My MaemoMapper can see your freakin' house!!!

Hi, I am really happy to hear that works on os2008, actually I never tested them on os2008 since I don't have it. Although I compiled them from the same source, only tested them on os2007. Anyway bring 1.4 back to life is my goal.

syzygy, I am really sorry for your case, maybe I didn't explain it clearly. The new format is indeed designed for big contiguous area, esp for the big cities. I tested it using 50M map files(for the big urban area I lived), 13000 tiles, the result shows the wasted space is 20%-40%, compared with 300%-400% on MM1.x before with 32k allocation unit. If you map repository is for large scattered area, consider using small allocation unit. And seems like that your 50k map tiles only take 50M space, most of them must be the water body or farm land kind of landscape.

Things are improving. I reformatted my 8G SD card with as FAT32 with 2048B allocation units. The aforementioned VEStreet files are now 49.4MB and use 58.4MB of disk space. This is less than 20% overhead, and should improve with larger, more contiguous tiles.

The GSat files are in the process of copying back after conversion, and we'll see how those work.

As an aside, MapsIn1 has now crashed a couple of times on the same .db file. The only error I get is the Windows "do you want to send an error report" message. The last file on the list was different in each case.

Like dude, I also tried 1.4 on OS2008, and it appears to work fine. There was a little confusion in setting up the map repository names and hardware keys that were copied from 2.3, but it was fine afterwards.

Thanks again...

Well, the crash of mapsin1 maybe caused by gdbm routine which keeps allocates/release memory(in number of 10s of thousands of times), or fetch a corrupted tile, Windows runtime is notorious weak in doing that. But it won't damage your map files. Just keep running it, the saved map files will be skipped.

I like to explain the new format in the more detailed fashion here. In MM1.4, the map file is stored in z, x, y as the path name, one file contains 1 tile. It doesn't have a centralized data table, which is very good for a SD card storage system. It's fast, scalable, spreaded writing in the whole SD card. But the drawback is the waste disk space, since each file only contains 1 tile.

I personally won't use GDBM file system on a SD card, which may damage or kill the card, with the only benefit of small storage space. That file system was designed 15 years ago and has no idea about flash storage system. It maintains the centralized storage table, the hash table(bucket). The MM2.x updates(sync) that table on per tile base, which means on a 32k allocation unit card, it writes about 3000 times to fill a table block(bucket). A SD card block has the limit on times of writing, and that makes the block very easy to be damaged permanently.

The new format inherites the MM1.4 file system idea, it uses z, x/4, y/2 as the file path name, thus each file stores 4x2 contiguous tiles. It doesn't have a centralized data table as well(which you should always avoid on a flash card), and provides the balance between speed and storage space, and no potential damage to your card.

Some thoughts:

Since the new format store 4x2 tiles, each file stores 2 tiles in y direction(4 in x, the way of this was designed considering tablet's screen resolution). It may not work ideally to store maps along road in North-South direction, in scattered manner, like California costal area. In this case, you should use small allocation unit. Actually I don't think small allocation unit under the new format will bring much penalty in access speed, because the new file system always tries to read/write the 8 map tiles together. It greatly reduces the read/write times on your SD card, thus extends its lifespan.

I am considering to use 3x3, 4x3, or 4x4 tiles per file as the more balanced storage format. Anything beyond that is not necessary and may bring acess speed penalty. This is basically an engineering experiment on MM. It's clear that we can find ways to improve it dramatically.

Nutter,

Thanks for the explanation. This really helps to justify going away from the GDBM format, and satisfies most of my needs. I really like the idea of being able to maintain repositories manually again.

Regarding the MapsIn1 error, I did find that rerunning two more times got everything transferred. I tried it on a different machine, and this gave a different error--something about dbgeng.dll could not be found. Hopefully, I'll only have to run this once!

The Google Satellite files have much better space utilization. After translation and transferring back to the SD, I now have 1.56GB of files that use 1.58GB of disk space. Excellent! And this was from a 1.78GB .db file. Not sure if something got missed in the translation--will recheck.