[Under consideration] More efficient and flexible use of internal flash
 

https://maemo.org/community/brainsto...internal_flash

about 27GB of the N900 internal flash memory are reserved for a VFAT (FAT32)
partition which is mounted on /home/user/MyDocs and exported as USB mass storage.
The VFAT filesystem is an ancient, slow, inefficient (cluster size, bad for many small files) non-POSIX compatible (no symlinks, permissions etc) filesystem and only used for maximum compatibility when exporting it via USB mass storage.

It would be nice if it would be replaced by a ext3 or any another modern POSIX filesystem and merged with the /home partition so that all advantages of those file systems can be used. This would remove the 2GB limit for the home partition and make a more standard Linux home directory layout possible (eliminating MyDocs, new directories Pictures, Documents etc. in the home directory).

For USB mass storage mode file system images with any file system (e.g. ext3, NTFS, HFS+ depending on the users desktop OS) and arbitrary size could be stored on the partition and exported using loop devices.

Re: More efficient and flexible use of internal flash
 

This is an awesome idea. Just some questions:

1) Do you think we could have some sort of script to backup and restore the whole setup before and after applying a firmware update? (I know that until Nokia actually changes N900 default filesystem layout we will have to use some sort of workaround)

2) What would be the best file system for a flash-based disk? I'm a little bit worried about XYZ wearing the disk too much because of too much updates to some sectors. Anyway, FAT32 updates the FAT too much, anyway.

3) I also agree with your last points about having a more standard Linux directory structure. It would make it a lot easier to use a tool like rsycn or Unison to synchronized your desktop files with the N900.

4) Do you think we could also throw into this mix some sort of selective Cryptography (for example: keep a subfolder encripted)?

5) Finally. I'm assuming you have already managed to get your idea working on your device. What are your thoughts about it? Anything unusual happened?

Thanks,
Luis

Re: More efficient and flexible use of internal flash
 

See http://talk.maemo.org/showthread.php?t=35122

There's a few different repartitioning solutions there.

Re: More efficient and flexible use of internal flash
 

Hi Luis,

IIRC a firmware reflash did only overwrite the separate root flash memory but did not change the MMC flash at all. So there is no need to perform a backup only for the reflash (you should always do backups, nonetheless). If you want to backup the paritition table you could simply use sfdisk or save the first sector of the flash.
apparently UBIFS (the fs for the root fs) is not possible.
Nokia decided that ext3 with writeback cache is a good idea for /home and I guess it was a informed choice?
that's actually a good point! you could use your N900 as a backup device for your desktop home and vice versa.
everything which is possible on Linux should be possible on the N900 as well.
for instance, you could have a encrypted loop device using a file on the ext3 partition
and bind the directories into your home.
no problems so far. My next plan is to run some benchmarks on the root, the ext3, the vfat partition and vfat image to check how much overhead the loop device causes.

I wish Nokia would get rid of the MyDocs partition so that applications don't have to deal with a non-POSIX fs but can use a large POSIX fs.

Re: More efficient and flexible use of internal flash
 

I voted against. So what use is to have a ext3 MyDocs partition? Most people are already storing their documents in FAT32 USB drives already. I don't need symlinks nor fancy features to store my documents, and in fact it is mounted noexec for a reason.
Also, $HOME is already ext3.

Don't think that making it ext3 will instantly remove the 2GiB limit in /opt and $HOME. Do you want them unmounted when connected through USB?

And I can't see what are the benefits of having a 25GiB FAT32 loop file in a 27GiB ext3 partition instead of a 25GiB FAT32 partition and a 2GiB ext3 partition. Before you say "dynamically resizable!" consider that for me the only difference is that with the loop file method I won't have to use e2resizefs (but I'll have to use a tool to resize the FAT32 filesystem either way).

This is not such a simple issue.

Of course, if you don't care about windows or exporting through USB.... but that's not the common use case here. Still, I agree it has to be possible to do this on your own device.

Re: More efficient and flexible use of internal flash
 

Thanks for elaborating your concerns.

For people who
* use a single-user Windows with a FAT fs (no NTFS, no compression or encryption, and only few small files)
* and will never install more than 2GB apps
* and who will never need more than 2GB in their $HOME (e.g. only smal email folder)
my solution is neither helpful nor harmful.

But for tech-savy people (which I believe a high percentage of the Maemo community is) who would like to
* use different modern file systems (ext3, NTFS, reiserfs, zfs, HFS+ etc)
with their extra features (POSIX, permissions, compression, efficient storage for small files, encryption - important for a mobile device, versioning or backup etc)
to store or sync files with their desktop
* need are large $HOME
* use the flash efficiently (see later)
* export different filesystem images depending on the context (e.g. FAT image for compatibility, ext3 at home)
my solution offers them much more flexibility.

Please have a look at my loop-device solution in the brainstorm.
It already works flawlessly on my N900 and it would be great it was default
so that users would not have to go through the hassle of repartitioning their MMC.

Basically you would have a single ext3 partition on the MMC flash mounted as /home.
On this partition you would not only store /opt and $HOME but also virtual disk images
with arbitrary size and filesystem.
The images are sparse files, i.e only allocated sectors in the disk image are actually
allocated on the ext3 fs and free space is not wasted.
This way you could have at virtual 20GB FAT image (with 10GB used) and your
ext3 fs would still have 22GB free (32 minus 10GB).
It is possible to create several images and to select which one is unmounted and exported for USB mass storage mode.
it's not easy to use e2resizefs on /home as you'd have to umount it first.
AFAIK online resizing works only for growth.
Its much easier to resize an optional disk image than a crucial partition which
holds most of your apps and your home.
see above. It already works perfectly with Windows and USB mass storage mode.

Re: More efficient and flexible use of internal flash
 

Please don't vote against it. If you don't care about all the missing features just let it go. It won't affect you. If you would like to have a very large FAT32 partition you could just set it to the available free space, giving you more potential space than with the current partition scheme.

What you seem to be missing are many things that would be interesting if all that default space was not wasted on one of the worst file systems of the last +-20 years. For example, if we could have a default installation with just a decent and appropriate fs:

• We could use rsynch or Unison (or any other tool) to synchronize files with a desktop installation without the risk of overwriting files because FAT is not case sensitive.
• We could preserv all fs attributes
• We could use "ln -s" to symlink things
• We could avoid wasting some much space as FAT does
• We could have modern filesystem niceties, such as journaling (if we wanted), good performance, better indexing and so on
• We could have much better security, because we could use space in the /home/user partition to place an encrypted folder that doesn't get exported as mass storage
• We could even have, if we think appropriate, different VFAT partitions (loop devices ponting to files, actually) that are exposed as mass devices
• We could indeed not have to worry about only having 2GB for the OPT partition
• We could even try (it seems to be possible) to use AUFS to stack the root partition and the 32GB partition together, allowing the system to only write to the 32GB partition, either getting rid of the 256MB limit problem or minimizing it. When you would install some files to the /usr folder it would get transparently written to the other partition, where the limit is much higher.
• I think this list could go on.

Anyway, what is been proposed here is NOT an one or zero solution. You would not lose MyDocs (I, to be sure, would love to get rid of that). You can still use it, it can still grow to 22GB or even more than that. It would still be exported. It could still be mounted noexec. Whatever. The idea is that for those that care, we could have a better partitioning scheme.

Re: More efficient and flexible use of internal flash
 

Here is a quick one: if I had titan's partition scheme I would try to use AUFS, bind mount or other techiniques to transparently make better use of disk space. That done, i would probably have the FULL debian repository at my disposal, without some of the IO problems that the easydebian people are trying to solve. All this in a nicely, secured, scructured manner.

Don't you think this would be nice?

Re: More efficient and flexible use of internal flash
 

Seriously. I don't see how Nokia shipping a larger _ext3_ partition by default will help users who want to use zfs.

No, it doesn't work flawlessly. It's a bit slower. And it's sparse, but not smart. If you full format it/fill it ONCE you'll have a 20GiB FAT image _FOREVER_. You'd have to implement vmware-like shrinking capability, etc. Good luck selling that to the average user.

I though your idea was to create one big ext3 partition only.

Re: More efficient and flexible use of internal flash
 

You can very easily do it yourself on the device. I think the ability to still do that and choose whatever filesystem you want is MUCH MORE IMPORTANT than Nokia changing something that works pretty well _right now_.

I can do most of these now, in the 2GiB ext3 partition and the 256 MiB OneNAND. And if the sizes worry you, you can repartition, which works _better_ than the loop file. And you don't get to confuse users when they ask where their 32 GiB are.

Again, what's the difference between a 25GiB MyDocs file in a 27GiB ext3 partition and a 25GiB MyDocs partition with a 2 GiB ext3 partition?
So far, I only see "No need to e2resizefs", which is something I could consider a point if it weren't because a user like you would usually just shrink the MyDocs partition to 2GIB and then enlarge the ext3 partition ONCE and be done with it.

You aren't free to do that now?