Maemo Mapper Mod - the new map file format
 

Not quite happy with neither MM1.x nor MM2.x map repository format, I decided to write my own one. It went on and off for more than a month, now it's basically done. BTW I ported MM1.4.7 to os2008 during the process, what an achievement! This is the first time I did coding on Linux platform, kind of brainstorming, lots of pain and fun as well.

The new map repository format is very similar to MM1.x one, but instead of storing one map tile in one file, the new system stores up to 8 map tiles in one file(the number can be changed, very scalable). In this way the disk file space is reduced significantly but the access speed is not compromised.

Compared with MM1.x repository format: The waste disk space is reduced to about 1/10 of the original MM1.x format, but access speed maintains the same or even faster(since it can read up to 8 map tiles at one time). Typically MM1.x system requires about 400-500MB disk space to store about 100MB map tiles on a SD card(32k allocation unit for SDHC card). Well the new map repository needs only about 120-130MB disk space.

Compared with MM2.x repository format: The access speed is greatly enhanced. Also the new format removes the 2G byte limit on a GDBM dbase file. The map repository size is limited only by the SD card capacity.

To demonstrate the new map repository file system, I modified the MM2.2 and MM1.4.7, made them available for both os2006-2007 and os2008 platforms. To port MM1.4.7 to os2008, the internet utility library is changed from osso-ic to conic. Now you have the Maemo Mapper 1.x and 2.x working on all platforms, and they have the same map file system. The new repository uses similar file structure as original MM1.4.7(z/x/y), but the map tile file name is changed from .jpg to .map, also changed is the numbering for directories and files. Their numbers are not 1-to-1 match with the original repository.

You can go to the download page to get modified Maemo Mapper executables.
http://code.google.com/p/mmpcmapper/downloads/list

Also a PC utility tool is developed to convert both the original MM1.x or MM2.x map repository format to the new one:
http://www.internettablettalk.com/fo...269#post144269


The new software will not change or erase anything of the original data and setup parameters, except it will create a directory in MM2.x card to store the map tiles. In MM1.x the POI icon may not work since the icon directory is changed to the same as the MM2.x structure, the icon files locate in the same directory of poi.db.

The MM2.2 Mod 1 includes all the features up to official MM2.3.1 release, removed parts are GDBM file system and in-memory cache. I found there is not much difference in speed between MM2.3.1 and M2.2 Mod, due to improved access time of the new file system. So the in-memory cache needs not to be implemented, however the new file system does use about 3M byte as the in-memory cache. To change from your official MM2.x to the Mod version, all your data on your SD card will be kept untouched. A new directory similar to M1.xx file structure will be created to store the map repository. It's recommended to backup your GDBM file on PC and delete it from the SD card to save the disk space.

The MM1.4 Mod 1 includes all the features up to official MM1.4.7 release, only file format is changed. Note that for os2006-os2007 version, the old osso-ic internet library is still used to keep backward compatibility, in os2008 version it uses the new conic library. But you actually can compile the os2008 version source code on os2006-os2007 platofrm, it works as well. The Mod version also changes the POI directory(POI icon directory), now the POI directory has a consistant location for both MM1.4 or MM2.x Mod versions. To change from your MM1.4.7 to the Mod version, it's suggested to backup your files to PC and delete your old map repository. The new file system will not erase any of your data but they will mix up with the original MM1.x file system, since they're quite similar.

Re: Maemo Mapper Mod - the new map file format
 

Nutter,

Looking forward to trying this soon. What do you recommend as the optimal allocation unit for the new repository format? I believe I used 1024b with MM1.4 on the 770, and that worked fairly well. Note that most of the maps will be Google Satellite types, so they're a bit larger than the street versions.

Thanks!

Re: Maemo Mapper Mod - the new map file format
 

It's optimized for 32k allocation unit for the best speed and balanced waste space. The purpose I did that is I am still using MM1.4, and it consumes quite astonishing disk space. MM2.2 is slow unless you remove the rotation algorithm, which requires quite CPU resources.

If you use 1024B as the allocation unit, basically you don't have space wasted under this format.

Re: Maemo Mapper Mod - the new map file format
 

Nutter,

It looks as if the standard 32KB allocation unit for FAT32 is far from the best choice for the new format.

I took my .db file for VEStreet (55.9MB), and converted it using MapsIn1 on my laptop. This resulted in 9059 files for 49.7MB that took up 71.4MB on an NTFS drive. 58869 files were copied from the .db.

Then I copied them over to the SD card, formatted as FAT32 with a default 32KB allocation unit. Same number of files, but they now consume 296MB of disk space. There are still LOTS of little files. 4459 files are less than 2KB.

I'm guessing my files are taking up a lot more room because they are not for a big contiguous area, rather they are scattered along roads or coastlines.

I'm guessing I should just bite the bullet and reformat the card with a smaller allocation unit, probably the same 1024KB I was using before.

Do you agree?

Re: Maemo Mapper Mod - the new map file format
 

Now we're cooking!

I'll give it a go tonight. It sounds like I need to use you're modded MM2.2 executable then use your converter to convert my 4.78 GB of MM 1 maps for use with this new set-up...

Tre cool dude! The most attractive program that can run on my N800 is MM running hybrid satellite maps. It gets everyones attention. Especially if I'm driving using GPS... It looks like you're flying over the areas you're traveling through...

The first thing everybody tries when they see it is to look up their own address to get a view of their town. Unfortunately with the previous 2GB limitation, 9 out of 10 times I didn't have their town loaded. This got the "What good is it then?" eye roll and they handed it back to me.

If your mods works out, I can now load a large chunk (NJ, DE, half of PA, and NYC) on the sucker and it will blow the TomTom's and Garmins out of the water.

Re: Maemo Mapper Mod - the new map file format
 

1.4 it is for me... It rocks on 2008. Thanks again! http://www.clicksmilies.com/auswahl/ernaehrung004.gif

****

Dude1: My TomTom can take me right to my house address!

Dude2: Well, my Garmin can tell me what side of the street your house is on!!

Me (The Dude): Oh yeah. My MaemoMapper can see your freakin' house!!!

Re: Maemo Mapper Mod - the new map file format
 

Hi, I am really happy to hear that works on os2008, actually I never tested them on os2008 since I don't have it. Although I compiled them from the same source, only tested them on os2007. Anyway bring 1.4 back to life is my goal.

syzygy, I am really sorry for your case, maybe I didn't explain it clearly. The new format is indeed designed for big contiguous area, esp for the big cities. I tested it using 50M map files(for the big urban area I lived), 13000 tiles, the result shows the wasted space is 20%-40%, compared with 300%-400% on MM1.x before with 32k allocation unit. If you map repository is for large scattered area, consider using small allocation unit. And seems like that your 50k map tiles only take 50M space, most of them must be the water body or farm land kind of landscape.

Re: Maemo Mapper Mod - the new map file format
 

Things are improving. I reformatted my 8G SD card with as FAT32 with 2048B allocation units. The aforementioned VEStreet files are now 49.4MB and use 58.4MB of disk space. This is less than 20% overhead, and should improve with larger, more contiguous tiles.

The GSat files are in the process of copying back after conversion, and we'll see how those work.

As an aside, MapsIn1 has now crashed a couple of times on the same .db file. The only error I get is the Windows "do you want to send an error report" message. The last file on the list was different in each case.

Like dude, I also tried 1.4 on OS2008, and it appears to work fine. There was a little confusion in setting up the map repository names and hardware keys that were copied from 2.3, but it was fine afterwards.

Thanks again...

Re: Maemo Mapper Mod - the new map file format
 

Well, the crash of mapsin1 maybe caused by gdbm routine which keeps allocates/release memory(in number of 10s of thousands of times), or fetch a corrupted tile, Windows runtime is notorious weak in doing that. But it won't damage your map files. Just keep running it, the saved map files will be skipped.

I like to explain the new format in the more detailed fashion here. In MM1.4, the map file is stored in z, x, y as the path name, one file contains 1 tile. It doesn't have a centralized data table, which is very good for a SD card storage system. It's fast, scalable, spreaded writing in the whole SD card. But the drawback is the waste disk space, since each file only contains 1 tile.

I personally won't use GDBM file system on a SD card, which may damage or kill the card, with the only benefit of small storage space. That file system was designed 15 years ago and has no idea about flash storage system. It maintains the centralized storage table, the hash table(bucket). The MM2.x updates(sync) that table on per tile base, which means on a 32k allocation unit card, it writes about 3000 times to fill a table block(bucket). A SD card block has the limit on times of writing, and that makes the block very easy to be damaged permanently.

The new format inherites the MM1.4 file system idea, it uses z, x/4, y/2 as the file path name, thus each file stores 4x2 contiguous tiles. It doesn't have a centralized data table as well(which you should always avoid on a flash card), and provides the balance between speed and storage space, and no potential damage to your card.

Some thoughts:

Since the new format store 4x2 tiles, each file stores 2 tiles in y direction(4 in x, the way of this was designed considering tablet's screen resolution). It may not work ideally to store maps along road in North-South direction, in scattered manner, like California costal area. In this case, you should use small allocation unit. Actually I don't think small allocation unit under the new format will bring much penalty in access speed, because the new file system always tries to read/write the 8 map tiles together. It greatly reduces the read/write times on your SD card, thus extends its lifespan.

I am considering to use 3x3, 4x3, or 4x4 tiles per file as the more balanced storage format. Anything beyond that is not necessary and may bring acess speed penalty. This is basically an engineering experiment on MM. It's clear that we can find ways to improve it dramatically.

Re: Maemo Mapper Mod - the new map file format
 

Nutter,

Thanks for the explanation. This really helps to justify going away from the GDBM format, and satisfies most of my needs. I really like the idea of being able to maintain repositories manually again.

Regarding the MapsIn1 error, I did find that rerunning two more times got everything transferred. I tried it on a different machine, and this gave a different error--something about dbgeng.dll could not be found. Hopefully, I'll only have to run this once!

The Google Satellite files have much better space utilization. After translation and transferring back to the SD, I now have 1.56GB of files that use 1.58GB of disk space. Excellent! And this was from a 1.78GB .db file. Not sure if something got missed in the translation--will recheck.

Re: Maemo Mapper Mod - the new map file format
 

It doesn't matter if it writes to the same place or not, it's the number of writes that matters - the writes are wear-levelled over the card so there's no difference if you re-write the same VFAT block or not (not taking into account possible wear-levelling sectioning of the card, which is up to the vendor).

Maybe that's what you meant anyway, just thought I should mention it to avoid confusion.

Re: Maemo Mapper Mod - the new map file format
 

That's what I mean, the number of writing, but has to be to the same location.

The GDBM uses default allocation unit size as the hash table(bucket) size, normally it's 32k on a SDHC card.

Each data item uses about 12 bytes, so each bucket contains about 3000 entries. For every entry update it writes disk to sync it, so it writes about 3000 times to one block just to fill it. Sounds like a killing machine to SD card.

syzygy, I am glad you even make the repository smaller. Honestly I didn't dig deep into the detail implementation of the gdbm. It's a very old piece of junk. I can't find even a working version on Windows platform, none works actually. I ported it to Windows but didn't do any clean up. Time to move on to rid of it.

Re: Maemo Mapper Mod - the new map file format
 

What I dislike most is the GDBM header, it contains all the important information and locates in just one allocation unit. So each tile update it needs to write to the header once, you have 50k tiles, it write 50k times.

If the operating system doesn't have the dynamically changing allocation support for SD card(I don't know if maemo has or not), it pretty much ensure a kill of that block some time down the road.

Re: Maemo Mapper Mod - the new map file format
 

I still think there may be a misunderstanding here - if by dynamically changing allocation support you really mean what I called 'wear levelling'. That's a feature of the SD card itself, not of the operating system (unlike the built-in 256MB flash. That's why the internal flash is using JFFS2, as that's a file system with wear-levelling built in. That's not necessary for cards.)

Nevertheless, I'm still very interested in your experiments with alternative map storage formats.

Re: Maemo Mapper Mod - the new map file format
 

My understanding is that it needs operating system support, although I am not very sure. If it's up to the card controller support, then that's something even we can't know, since none of the manufacturers specify it.

Anyway, Traditional GPS doesn't have this problem since they rarely do real-time downloading maps, the maps are pre-installed. I don't want to bet on the chance.

Re: Maemo Mapper Mod - the new map file format
 

@nutter: I'm relatively sure that it happens on the SD card itself. If I understand correctly, even the SD controller on the N800 doesn't have to know anything about the actual mapping of physical blocks to logical blocks. AFAIK, the only thing really needs to be avoided are lots of small writes, since every time 1KB is written a whole block gets erased and written. Better to wait and do 1 64KB write, than 64 1KB writes. BTW, thanks for the work on tile based storage.

-John

Re: Maemo Mapper Mod - the new map file format
 

Nutter, I think you may be somewhat confused about how the system works. 30,000 writes to an SD card takes a substantial amount of time. I think those writes you're talking about take place in RAM, and then are written once to the card. You can change the contents of a file almost indefinitely in RAM, but you don't have to write the whole database to a file each time. It's trivial to manipulate files in memory indefinitely, and only write to storage once when finished. That's the way almost every program works.

Re: Maemo Mapper Mod - the new map file format
 

Nutter,

I'm in love again with MM1.4, on my N800 and OS2008. VERY snappy performance overall, and as good or better file sizes compared to MM2.2.

I've encountered a couple of minor problems, and was wondering if you had any clue as to what is going on.

First, I believe you were correct that MapsIn1 crashed when it encountered corrupt tiles during translation. These showed up as a black 4x2 tile area when viewed. I tried to correct this by checking the 'Overwrite' button, and downloading the area again. Same result--there was still a black area. I then manually deleted the offending .map files, navigated to the area, and new ones auto-downloaded properly. I also encountered a crash of MM when trying to download a larger area, but I don't think this had anything to do with the corrupt file issue. It's important to have the overwrite function work properly, especially for updating maps.

Second, something is strange with the hardware keys. With MM2.2 I had Zoom In and Zoom Out mapped to the up and down rocker buttons on the front panel. These will not work as zoom buttons, but are fine for other functions. I reset the buttons and reselected, but no go.

Ideas?

Re: Maemo Mapper Mod - the new map file format
 

hi nutter,

it seems that you made sth. very useful for all maemo mapper users.
Did you talk to the maintainer of maemo mapper, john costigan?
I think it would be the best idea to integrate your great improvements
directly in maemo mapper releases.
So people get the speed and storage improvements when they simply update
maemo mapper.

So - did you have contact to john - will he integrate your code in the main release of maemo mapper ?

Thanks for your mod again and again ....

Regards,
asys3

Re: Maemo Mapper Mod - the new map file format
 

I checked some resources, TA-t3 is right about the flash card features. Operating system can support wear levelling, also SD controller supports it as well. But not all SD card support that, many cheap ones don't. As for MM2.x writing to flash card, I checked the source code, it does write that many times. Since for every tile downloading, the code does a "sync", that's the flushing the file to the disk. It will write to header, hash table and data section.

syzygy, I guess your hardware key problem is due to that you use MM2.x before, and its key configuration method/value may be different from the MM1.x, and downgrade may confuse the MM1.x. BTW, I didn't change anything in both MM1.x and MM2.x except replacing their file system, so the problem may well be from the incompatibility of the two. I need to look into the source code to see how old MM1.4 and new MM2.2 handle the hardware key and replacing tile issue, to find some resolution for you.

BTW, since this is the engineering experiment to probe the way to improve MM, so I tried to limit the code change as small as possible.

Re: Maemo Mapper Mod - the new map file format
 

Thanks again, Nutter.

I think the hardware key problem is a bug that goes back to 1.4.7 and OS2007. I see the same problem with this configuration on the 770. If it's an easy fix for you to do, I would appreciate it. John may not want to resurrect a version that has been put on the shelf, so to speak.

Re: Maemo Mapper Mod - the new map file format
 

LOL, that's right. I see John has little interest in touching MM1.4 again.

My Mod is basically to bring MM1.4 back to "modernization" to be useful . MM2.2 is just not for me, it's slow and keeps confusing me by rotating the maps. I'll check and see what I can do, I think 1.4 is kind of free to do any Mod.

Re: Maemo Mapper Mod - the new map file format
 

It's possible that the card is being written to that many times, but it's incomprehensible to me that anyone would do that. It slows the whole app down by orders of magnitude. That's so inefficient that I still have doubts, but I'm not so interested that I'm willing to take the time to go through all the source code to verify it one way or the other.

I'm unsure of what you mean by rotating the maps. They don't rotate on mine unless I tell it to, and I prefer just keeping north up, so the maps are readable. Maybe you mean something else.

Re: Maemo Mapper Mod - the new map file format
 

I just have the new mod version of MM1.4 done.

1. "Temporarily" fixed the hardware zoom key issue syzygy asked, now you can config MM1.4 zoom key to any keys. It's indeed an original MM1.4.7 bug.

2. Changed and optimized the memory allocation method during map tile downloading. The old method is really a crude one, only for concept proving, and it may cause crash during large area downloading.

3. Fixed a fatal issue in MM1.4.Mod.1 os2008 version, I did a dumb thing that I accidently delete the bluetooth DBus init code when cleaning up the ossoic DBus init routine, so the old 1.4 mod 1 version can't find GPS reciver. This problem is fixed now. os2006-2007 version doesn't have this issue.

To download the code, go to
http://code.google.com/p/mmpcmapper/downloads/list

I think we're close to there now. I will refine the file system to make it more "bullet prove", more robustic against "rouge" data. Then we will have a "perfect" MM1.4 working version.

Re: Maemo Mapper Mod - the new map file format
 

I'm curios about that - is the "sync" (presumably an fsync(2) call?) in the gdbm code? Or in the MM application? That's not good to have on a flash fileystem - without it the operating system would be very much more efficient (as sgosnell already said earlier).

Re: Maemo Mapper Mod - the new map file format
 

It's in the MM2.x application code. The old gdbm always sync the data with disk no matter how small change it is. The gdbm 1.8.3 seems to give user some choice and provides a sync command(basically fsync), you can choose to sync at your time.

The MM2.x does sync for each tile downloading. The structure of gdmb requires you to update data, hash table and header, not very good. But nevertheless, even you don't issue sync command, the gdbm will do a lot of flushing disk also since it has very little cache allocated.

Re: Maemo Mapper Mod - the new map file format
 

As long as there's no explicit fsync(2) call (directly, or indirectly through e.g. fflush(3)), then the operating system itself is what determines when the data will be written to disk, and the application's/library's internal caching doesn't matter much (all disk writes go through the kernel's buffer cache, and when it's being physically written is determined by configured kernel parameters as well as by amount of available RAM).

Re: Maemo Mapper Mod - the new map file format
 

Well, this is the source code:

in MM2.x(display.c)

if(++_curr_download == _num_downloads)
{
#ifndef MAPDB_SQLITE
if(_curr_repo->db)
gdbm_sync(_curr_repo->db);
#endif

In gdbm 1.8.3

void
gdbm_sync (dbf)
gdbm_file_info *dbf;
{

/* Initialize the gdbm_errno variable. */
gdbm_errno = GDBM_NO_ERROR;

/* Do the sync on the file. */
fsync (dbf->desc);

}

I may made a mistake by saying that it sync disk for each tile downloading. It may be for each downloading.

Re: Maemo Mapper Mod - the new map file format
 

That looks like it does a sync when the download is complete, which it should do. I can't say for sure without seeing the complete source code in context, but ISTM that "if(++_curr_download == _num_downloads)" means don't do anything until the current download is the final download. One write per download is a far different thing than one write per tile. I don't think any competent coder would do a file write for every map tile.

Re: Maemo Mapper Mod - the new map file format
 

That's probably one of the issue panning is slow. Even you downloading couple of tiles, it does a sync, which I really don't like.

Another thing MM2.x does a rotation calculation(even it doesn't rotate) for each display update, it slows down the speed as well.

Re: Maemo Mapper Mod - the new map file format
 

If you're unsatisfied with the performance of downloading/saving maps while panning, then perhaps you should consider turning off auto-downloading and pre-downloading the maps that you need. Or using no Map Cache database at all. Or a faster MMC card. The disk sync is done in a separate thread, though, so it shouldn't freeze the UI.

And yes, sgosnell was right, it only syncs after a set of downloads is complete, so if you're panning many times consecutively, that would be a single file sync. If it didn't sync at all, then you could lose data if Maemo Mapper or your device crashes.

When it pans, it accesses new maps, and (of course) all maps use north-is-up orientation. How is Maemo Mapper supposed to draw newly-accessed maps at the correct orientation without rotating them?

That, however, shouldn't matter to you if you're always using the north-is-up orientation. In that case, "map rotation" is essentially a no-op - very little--if any--actual computation is being done. This is very easy to see for yourself - map redrawing is much faster when you are at a rotation of 0 degrees (north is up).

Re: Maemo Mapper Mod - the new map file format
 

Quick question: is any effort made to simplify rotation at the other cardinal directions? They should be fast vs. intermediate angles, though I didn't notice any difference in scrolling regardless of angle. Maybe I'll try it while paying attention...

Re: Maemo Mapper Mod - the new map file format
 

That's basically what I did. I never download maps using N800 actually, all my map tiles downloading, poi...are managed on PC. But even that, MM2.x can not be comparable with MM1.4 in the speed.

I did some tests as well, actually putting display in the separate thread slows down the speed. I tested to use the direct call, and it turns out much fast. But I know I can't do that in the real world since if downloading runs into trouble, it will block and freeze.

I am using modified MM1.4.7 and quite happy with it. I don't need rotation, etc functions. In the new format my 129M map tiles only take 135M disk space, very good.

Re: Maemo Mapper Mod - the new map file format
 

The chance for a crash that actually loses data is very very small though. It would basically have to be a kernel panic (i.e. a kernel error or some fatal hardware problem). Other kind of crashes (e.g. UI crash because of low memory, bug in MM, watchdog reboot) will still sync the data as part of the shutdown. On a desktop computer you can accidentally pull out the power cable, but on a battery-powered device you have to actually remove the battery.

The most likely way of losing data would probably be to yank out the memory card so fast that it can't get to sync it between when you open the door/remove the magnet (on N800) and when you rip out the card.

Re: Maemo Mapper Mod - the new map file format
 

You have to write the data sometime, though, and it may be easier and clearer to do it explicitly when the download is finished, to minimize the possibility of data loss and to make maintenance easier. It's a design choice, and one I agree with. I don't think writing as soon as the download is over slows things significantly.

Re: Maemo Mapper Mod - the new map file format
 

nutter,
thank you for bringing v1.4.7mm to os2008. I have used mm for several years, and have loaded mm with custom maps that I have scanned in and then tiled with mapcruncher. When you ported mm to os2008 you changed the map format to a .map file for better storage to optimize the space. This however broke my ability to edit the individual map tiles which were .jpg's. The .map files cannot be edited, they throw errors. When I scanned in a map I would fill in the 'empty' space that would happen with the tiles on the edge. This would prevent mm from crashing on what it though was a corrupted tile.
I find it very difficult to use v2.2 mm for the custom maps as accessing the tiles is troublesome, and your port would be fine, if I was able to edit the .map files or even better I had the option to keep the old .jpg format of the os2007 mm, this would be best as it allows easier editing of individual tiles. I am not sure but what would be the possibility of having a choice between the space saving .map format, which I would use for 'standard' maps, and the .jpg which could be used for 'custom' maps, where storage is less of less import?
If not maybe you could help with why the individual .map files are not editable.
thanks for your time to consider this...

Re: Maemo Mapper Mod - the new map file format
 

Nutter, two questions.

1. Have you talked to the maemo-mapper author about switching database formats and/or integrating your mod into his program?

2. Have you considered (since you're already halfway there) creating a program for Windows and/or Linux that automatically downloads the map files for a given area and automatically builds the database? I ask this because if you've already done this, or are part way there, then there's no need for me to do my script which would essentially be duplicating your work. If not, could you consider it? :) It'd save me the **** of trying to learn GDBM.

Re: Maemo Mapper Mod - the new map file format
 

Gnuite (above) is the author; as you can see, they've talked, and seem to not quite agree entirely. ;)

Re: Maemo Mapper Mod - the new map file format
 

Well poo. :(

Re: Maemo Mapper Mod - the new map file format
 

Anyone know the status of this project mod? How does it compare to the current repository version of M.Mapper? Any dependency issues if I install this?

I'm just in the process of caching large areas and finding M.Mapper is getting very ssssslllllloooooowww while panning around. This idea is intriguing, but it's circa 2008, with nothing new posted...

Anybody using it?

Edit: So, I tried this and it works very well for my needs. I've cached pretty much everywhere I'm ever going to go and it isn't any slower in panning around than when I started. The only annoyance I've run into is that I have to remember to not upgrade to the latest version of MM, which a trivial issue. Don't know, maybe the new version of MM has solved the slow panning issue, but I'm not inclined to try it. Nutter's version works very well for me.