Problem with Maemo Mapper 1.4 and SD's
 

I installed the magnificient MM on my N800 and decided to collect map data from Europe, by zooming and moving around, all the while MM updated map data through my home WiFi, brilliant!

After playing a while with the app, I got a write error to the 4 GB SDHC - the external card seemed empty in Filemanager. I had some trouble with the card earlier, so thought it is some problem with the "experimental" SDHC or the card itself.

Took another card, a 2 GB regular SD. Was all over virtual Europe for a couple of hours... collected some 10's of MB's map data. Took the card to my PC, backed up the files and reformatted the card with FAT, to dedicate it to map data for the time being. Copied map files back to the card, ve-ery slow but eventually done.

Went back to N800 and MM, and continued getting zoom levels and areas. Stopped to check I'm not running out of capacity - no problem there. Again, back to MM and further surfing around the map, while talking on the phone with a friend. I used the Virtual Earth street maps repository.

Then, I got a write error again, ext. card in Filemanager is 0 kB cap, 0 kB used, 0 kB everything. Restarted the N800, ext. card does not appear to be inserted. Reinserted, to no avail. Took the card to the PC and it's Windows Explorer started acting like it sometimes does with damaged CD's - not responding, but not quite freezing either. Finally it tells me there's an I/O error with the card. Format is no-go, tried several card readers and OS's.

So, my other card is trashed too. Now I am afraid to start over with another, this time a 1 GB card, or to buy another 2 GB SD / 4 GB SDHC, and use them in connection with MM...

What might be the cause of the mishaps? Maybe the N800 external drive got pissed off with my attempting to format the non-functional 4 GB SDHC, and it thus destroyed the next best card? :D

Now seriously; Maybe the external drive is going bad? Maybe MM 1.4 with it's gazillion small files simply overstrained the controller and it flushed the FAT or something?

Does anyone have similar experiences? What could be done, except wait for John's version 2.0 of MM with dbase instead of numerous files?

Is it safe to continue collecting map tiles to other cards that I may get?

Later: I decided to try the 1 BG card - for a while everything went ok, then I got the same error, writing a file failed. Now, the card still contains the files but Filemanager tells me 0 kB available -- is it possible that FAT has run out of entries or what?

Well, I bought a new 4 GB Sandisk, but dare not use it for MM maps - what a drag! I have to wait for the 2.0 version it seems.

I did buy the Navicore package though so I can use the N800 for navigation, but Navicore really sucks a bit. No possibility to save routes, only POI's. It is therefore not very practical for longer journeys, in addition to it's inability to search for destinations on other maps than the presently used - so I cannot plan a trip through Europe but have to change the map for Germany, France, Benelux, Scandinavia.

I'm really looking forward to the MM 2.0 or else, maybe I will keep my "old" navigator, the Navigon 7000T which has better integration and can save routes, use a map of whole of Europe at once...

Re: Problem with Maemo Mapper 1.4 and SD's
 

I'll reply my own post, at least a little: it seems that the maps files, 89 MB of them, take up 431 MB space on disk... interesting. After deleting some 500 MB of mp3's from the a.m. 1 GB card I can download map data again. The problem seems to be, if I want to get larger areas cached, I must a) take only a few zoom levels b) delete some areas every now and then c) get a card that can hold all the maps I need, bloated to 5 times their actual size by stupid Windows FAT arrangement.

Well, should I reformat the card in N800 and use 512 bits allocation size or what? My WinXP only offers default allocation size for FAT/FAT32.

Re: Problem with Maemo Mapper 1.4 and SD's
 

Just wait a little (?) bit for MaemoMapper 2.0 : maps will be hold by a database : no loose space !

Fred

Re: Problem with Maemo Mapper 1.4 and SD's
 

I would recommend reformatting with fat32 with 512 allocation size. The default is 4 KB, I believe, which results in a lot of space wasted due to slack space:

http://www.wikistc.org/wiki/Slack_space_data

Re: Problem with Maemo Mapper 1.4 and SD's
 

Rocketman: That link is misleading as it discusses FAT12/FAT16 but not FAT32. In fact the default FAT32 cluster size is 16KB for a 1GB SD card and 32KB for anything larger (up to the 8GB SDHC currently available).

Since FAT32 also has a limit of (approximately) 4 billion clusters, this places a restriction on minimum cluster size that affects larger cards. You can have 512 byte clusters only on volumes up to 2GB. The limit becomes 1KB on a 4GB volume, 2KB on an 8GB volume, and so on.

lite: You can't "run out of FAT entries" (allocated by the formatter) before the volume is physically full. I currently use two 8GB cards in my N800, one with default 32KB clusters containing mostly MP3s, the other with 2KB clusters dedicated to Maemo Mapper files - literally millions of them. I'm sorry that I can't offer an explanation for your write errors, or more generally, for why my installation works and yours doesn't.

Re: Problem with Maemo Mapper 1.4 and SD's
 

I ran into this problem as well (about 500M filling up a 2GB card), and formatting the card for FAT32 fixed it for me.

Re: Problem with Maemo Mapper 1.4 and SD's
 

Using a database container doesn't automatically mean no wasted space;-)
It still depends on the data organisation, the access methods, and other things.

For a memory-limited device as the 770, it could also make sense
to store the tiles in an optimized database AND have them zipped beforehand.

Ray

Re: Problem with Maemo Mapper 1.4 and SD's
 

On which system? On most systems default FAT32 cluster size is 4KB no matter the size.

Are you sure about this? is this result of some practical test?

While I'm not sure anything below 4KB block is supported with FAT32 at all, pure math says limits should be different. 4GB fits inside 32bits even when you address it by individual bytes, here you are addressing by clusters so why you can't have 512 bytes sized cluster with 4gb card? That should give you 2^32(=4294967296)/2^9(=512)= 8388608 clusters which should definitely fit into 32 bits of FAT32 limit. So in theory you could have 2^(32+9) bytes big card for 512 cluster size. Or if inforamation here http://www.cknow.com/ckinfo/f/FAT-Fi...tionTable.html or here http://en.wikipedia.org/wiki/File_Allocation_Table is true (and it probably is) and only 28 bits are used is still should allow 2^(28+9=37)=128GB card to have such small cluster.

Anyway making the cluster size small will make FAT table quite big. Generally it may not be very good idea to choose size below 4KB on gigabyte sized cards. Check this
http://www.storagereview.com/guide20...partFAT32.html
4GB card with 1KB block would give you FAT table size of 16MB and 512 bytes cluster will take 32MB.
Since most devices are not caching writes and keep FAT table in RAM and update it on every write this could make such card very slow and even not working in devices like PalmOS handhelds or digital cameras (due to device RAM limit).

But maybe linux is clever enough and handle big FAT tables sensibly so in N800 this could work?

Re: Problem with Maemo Mapper 1.4 and SD's
 

I'm happy with using the internal flash for maps, they don't take at all as much space as I feared. Internal flash is jffs2, so a) it avoids the vfat cluster problem, and b) it compresses, so it avoids any 'padding' problems and wasted space (the actual png and jpg obviously can't be compressed, but any other wasted space will be).

Re: Problem with Maemo Mapper 1.4 and SD's
 

Ray is right. No data structure, especially flat ones (i.e. disk-based), are without some amount of overhead. In Maemo Mapper v2.0's case, the overhead is in storing the "keys" that are used to find and access a given image in what is otherwise a sparsely populated matrix. This overhead, however, is significantly less than that of the internal fragmentation of typically formatted memory cards.

The images will be stored in the database as they are retrieved (from Google Maps, Virtual Earth, etc.). This is usually in either PNG or JPG format, neither of which reacts at all to zip compression, because they themselves are already heavily compressed.

Re: Problem with Maemo Mapper 1.4 and SD's
 

I've mentioned this observation before, and although I haven't been bored enough to try myself, I imagine formatting an entire memory card with JFFS2 instead of VFAT (or even ext2/3) would pretty much eliminate the internal fragmentation issues. Plus, it would introduce wear-leveling to extend the life of the memory card.

Has anyone tried this? Can JFFS2 even support file systems that large? I guess the only (consumer) problem is a lack of Windows-accessible JFFS2 formatters...

Re: Problem with Maemo Mapper 1.4 and SD's
 

I was speaking specifically about SD card formatting on WinXP SP2; sorry for the omission. Feel free to try it yourself, the OS apparently treats these cards (and presumably other removable media types) differently from hard drives, where 4KB is the default FAT32 cluster for volumes from 257MB through 8GB. Then it's 8KB up to 16GB volumes, and 16KB up to 32GB volumes.

Win2K and XP accept FAT32 volumes >32GB but will not create them. However, NTFS defaults to 4KB clusters for anything between 2GB and 2TB. Maybe this is what you were thinking of, since it's what we find on most large Windows hard drives these days.

I assume there is some architectural reason for treating flash cards differently, since the industry supplied SD card formatters also use larger clusters than the hard drive defaults. For example, if the internal write logic always uses large blocks, writing sub-blocks becomes a very wasteful read-modify-write operation.

As you are no doubt aware, the OS implementation trumps pure math. ;-) In other words, the designers might have imposed the artificial limit on cluster count (which can be confirmed by an MSKB search) due to the efficiency concerns you cite. Or maybe they just clamped the lid because of complaints about the 16-bit Windows 98/ME ScanDisk, which yields the same constraint.

One could write an alternative formatter (or use the Win98/ME version) that produces a consistent FAT32 volume outside the Win2K/XP limits, and such a volume would probably work in practice (mostly). On the other hand, it can be expected to confuse other system components that were designed around the official constraints. Come to think of it, I might dust off an old Win98 SE startup disk and experiment, or just try hacking the version check code in its FORMAT.COM ... and there's always FREEDOS ...

In any case, I agree with your concern over the downside of large FAT tables, especially on SD cards. I consciously accept this tradeoff for my mapper files because it works well for that purpose, even though loading the card is painfully slow. As I mentioned, I have another large card containing mostly MP3 files, where I use the default 32KB SDHC clusters.

Re: Problem with Maemo Mapper 1.4 and SD's
 

Isn't that done transparently by the card firmware?

Re: Problem with Maemo Mapper 1.4 and SD's
 

That depends on the card itself, I think. You're right, though, most memory cards implement wear leveling at the hardware level.

Re: Problem with Maemo Mapper 1.4 and SD's
 

Now that this thread has diverted a bit from the original, I'll add some details...

First, I formatted my 4 GB SDHC card in sfdisk while doing the "start from MMC" routine and tried formatting to FAT32, by using a different flag for fs type - I used option b in place of 6 (see posts regarding starting from MMC) and got a FAT32 - File manager showed 3,7 GB at first. But I had to reformat it in Win because N800 only could use 2 GB of the partition - after reboot File manager told 2 GB...

Now I have loaded some 900+ MB of maps (1,45 GB used on disk) onto it and the system starts to get real slow. Would it be possible to use an ext2 fs for the map files? How would that be accomplished, other than repartitioning and such in sfdisk? Would it help, as in not making Canola crawl like it does now?

Also, how to tell MM that the maps should go to a folder that cannot be seen in the File manager? It is a nuisance that the OS only knows to show FAT disks in the manager, but I'm not going to try and install KDE to fix it... :-)

Re: Problem with Maemo Mapper 1.4 and SD's
 

Thanks for correction.
Yes, that may be one reason (together with speed and RAM limitations of various devices). But since this works transparently you never know. For example even if you choose bigger blocks then due to partition table and other alignment, beginning of such big block may not align with beginning of internal block so you'll get multiple writes and read-modify-write anyway. I guess there is probably some sophisticated controller on the card to handle wear leveling and bad blocks and there is probably also some spare area with unused blocks so I guess it is same like with harddisks now - you don't know the real geometry, it is just array of blocks and any guesses may be wrong or may even change after card is used for some time.

Re: Problem with Maemo Mapper 1.4 and SD's
 

Yes, it should be possible, check this
http://www.gossamer-threads.com/lists/maemo/users/10585
Repartitioning should not be needed, just reformatting (mke2fs) and some modification of system files.

Maybe :-)
It should be possible to see such ext2 formatted card in file manager so this is not needed.

Re: Problem with Maemo Mapper 1.4 and SD's
 

You don't technically have to use the "Browse..." button - it's there for convenience. You can enter arbitrary text in the Repository Directory field. Just find out the pathname of the directory you'd like to use (e.g. using xterm), and enter that pathname into the field.