Re: More efficient and flexible use of internal flash
 

Yep, it seems to be broken in Fremantle, since all I get when I set MYDOCSDIR is that file open/save dialogs break (the file manager is a "big" open/save dialog) -- instead of showing the usual "Device" it shows a "MyDocs" icon that points to /home/user/MyDocs, but creates the tmp folder in the $MYDOCSDIR.

This should be fixable though since hildon-fm is OSS (you can rename the safe folders if you want, too).

Re: More efficient and flexible use of internal flash
 

1) Is it possible to replace the standard FM with another one?

2) Would it be possible to implement a standard wrapper around a replacement FM so that when the standard open/save dialog is required by an app, the new FM would provide that?

Re: More efficient and flexible use of internal flash
 

@titan @ ruskie
I would like to repartition the internal flash as soon as Nokia releases the firmware upgrade. Do you have updated any of your docs on how to acompiish that?

Re: More efficient and flexible use of internal flash
 

If the main aim of replacing the filemanager would be to allow access to /home an easier way that seems to work (I've done no testing at all!) would be to bind mount /home to a place under MyDocs, e.g.

If this seems to work OK you could add a line to your /etc/fstab file to automate this.

WARNING! I guess this could break in strange and unexpected ways upon attaching to a computer in mass storage mode. I'll have a look at an automatic umount but if you try this out in the mean time it's probably worth manually doing it before connecting to a computer.

Cheers,
Jan

Re: More efficient and flexible use of internal flash
 

Actually, the main aim would be to have more features. In addition to be able browse all the file system, I would like the following features (that are available in other FMs, as I'm told):

• I would like to see whole filenames but it hides the extension (why does it, anyway?)
• I would like to be able to see hidden files and directories (maybe, a button to toggle its visibility)
• I would like to be able to define the starting folder
• Maybe integrate with samba and sshfs to be able to browse the network
• Maybe integrate with samba and sshfs to be able to browse the network

Re: More efficient and flexible use of internal flash
 

it doesn't work. that was one of the first things I tried on my N900 when I found out about
the stupid MyDocs constraint.

Re: More efficient and flexible use of internal flash
 

fortunately some of the things you can do in the open source GPE file manager.
It seems that Nokia has tried to make the N900 as simple as possible and even simpler
(!) so that average joe would not get confused but the actual users are mostly tech-savy
people...FAIL

Re: More efficient and flexible use of internal flash
 

update:

I've been working on adding support for loopback files, porting GNU parted etc
lately, but I didn't have time to test it yet (I have to make a complete backup first...)
I have uploaded modified version of ke-recv and upstart (system-services) with some changes:
* only generate fstab if it is missing or requested using /etc/fstab.regen
* run optional /etc/init.d/pre-mount-once once before mounting
* support for mounting loopback devices (options in /etc/default/mount-opts)

http://www.maemory.com/N900/mmc/ke-r...1tt1_armel.deb
http://www.maemory.com/N900/mmc/syst...1tt1_armel.deb
sources etc: http://www.maemory.com/N900/mmc/

feel free to test them at your own risk (backup first!) and you're welcome to help me.
parted still crashes on start. I have no idea why.


My plan is to run a script which performs repartitioning (incl. shrinking of ext3)
before mounting any partitions on the eMMC. If you just want to grow ext3 you can do it online using resize2fs.
I still need to work out a good solution for managing which partitions or files are exported per USB
and mounted as MyDocs or other directories.

Re: More efficient and flexible use of internal flash
 

I don't think they have changed anything that is related to paritition handling
but we'll see when the update is available...

Re: More efficient and flexible use of internal flash
 

ok, assuming that "storing my files" is also an application :)I think there should be as few as possible partitions as you cannot use the free space on other partitions - which would be wasted if you choose an inappropriate partition schema.

I think the cleanest solution would be:
/emmc - single ext3 partitions on the eMMC mounted on /emmc
/emmc/opt - directory for addons managed by system
/emmc/home - user directories
/emmc/loopback/ - directory for loopback file incl. MyDocs FAT file
If you really want to boot multiple systems you could create an appropriate
symlink to opt.<version> during boot.

loopback files can be easily resized online without dangerous repartitioning
and you could keep multiple FAT images or, e.g. NTFS images.

Re: [Under consideration] More efficient and flexible use of internal flash
 

my commment on Solution #3: Use NTFS

NTFS is common filesystem read by every OS.
In fact, the Linux kernel does only provide read support (limited write support)
and you would need FUSE for full access.
With loopback files the user can exported any filesystem she wants.

Re: More efficient and flexible use of internal flash
 

It works for me! Might be worth trying again. I can save to and open from /home at will. Perhaps you originally tried a "normal" mount, without bind? I guess that could fail.

Soeiro, I agree that more features would be nice, but this has solved the only huge inconvenience for me. Next on the list would be to show the full extension, but that's probably going to take a bit more time to sort out.

Cheers,
Jan

Re: More efficient and flexible use of internal flash
 

really? that would be great!
What I tried was to bind-mount DCIM and .sounds etc to directories in /home/user.
I could open media files if the media player was called from the GPE file manager
but not from the media player itself. The camera app did not work at all.

Re: More efficient and flexible use of internal flash
 

I hadn't looked at the DCIM folder, but that (half) works for me as well. The camera app seems happy enough storing pictures on ext3, but the Photos app doesn't seem happy indexing them (at least it doesn't do so immediately).

I suppose the next thing is to try bind mounting /home/user/ext3MyDocs or something to /home/user/MyDocs and see what happens...

Cheers,
Jan

Re: More efficient and flexible use of internal flash
 

OK, I think this should be solved by modifying /home/user/.config/tracker/tracker.cfg to add the new folders to WatchDirectoryRoots. Haven't tried yet though. Will report again when I do.

Cheers,
Jan

Re: More efficient and flexible use of internal flash
 

I seem to be in a conversation with myself! :D

Well in case anyone is interested, I've now got this running almost as I want it. Here are some observations.

Most important observation: Strange and interesting things can occur if you mess with things that seem designed not to be messed with (as below). As this is only worth doing if you've repartitioned the eMMC I'm assuming that you're well prepared for whatever may befall you!

If you have media stored outside of the standard places it won't be seen by some Nokia applications, including Photos and Media player. This is easily fixed by editing /home/user/.config/tracker/tracker.cfg and adding the directories to WatchDirectoryRoots. You may need to reset your tracker processes using:

as user (NOT root!).

The locations shown for the hard-coded Documents, Music, Pictures, Videos and Camera folders in hildon-fm are stored in /home/user/.config/user-dirs.dirs. I got slightly odd behaviour when I tried changing these directly to directories on ext3, and changing the camera directory to a directory on ext3 seems to not work at all.

I think the easiest way of explaining what I did is to just give the commands. So here they are:

This process has given me the functionality that I want - media is accessible, but is taking up space on the ext3 partition instead of the VFAT one, the camera stores images to ext3, and I can access everything under /home from the file manager. It's obviously a right faff, but overall probably worth it.

To make the mounts last after reboots, and the mount --bind instructions to /etc/event.d/rcS-late, just before the line that says "#We can safely continue booting now".

I don't think I will ever use my device in mass storage mode, but this will almost certainly break that, because you won't be able to umount MyDocs unless you first umount the bound directories. I'm sure that could be fiddled with, but I'm not going to do it!

I'm sure this is clear as mud, so if you'd like any further clarification just ask.

Cheers,
Jan

Re: More efficient and flexible use of internal flash
 

may I join you ;)thank you! that sounds very promising. I'll have to try it myself.

Can you confirm that you could not find any other way to get it working,
except for the VFAT MyDocs mount?

In that case we could implement the following solution for people
who want to keep a USB exported VFAT partition:

* create a minimal VFAT loopback file and mount it under MyDocs

* create a hidden (VFAT) directory in MyDocs, e.g. MyDocs/_home

* bind mount ~/ to MyDocs/_home
and fix .config/user-dirs.dirs accordingly
(would be nice if MyDocs/_home/Documents instead of .documents works)

* mount the real VFAT partition or image somewhere else,
e.g., /home/extern

* in ~/.documents/ (or Documents/) create a symlink to the external folder
ln -s /home/extern/Documents/ ~/.documents/extern
and the same for the other media folders.
This way one can work with the ext3 folder and also include
the external files if the external VFAT is mounted.
If symlinks don't work use bind mounts and fix the mounting scripts to bind/unbind the directories for USB mass storage mode.

I hope that it will work.
It's a ugly workaround for the bogus VFAT requirement in the Nokia apps
but it's much closer to what I want.

Re: More efficient and flexible use of internal flash
 

Of course!

No problem. I found that aspects of what I wanted were achievable using other methods,e.g. changing XDG_DOCUMENTS_DIR in .config/user-dirs.dirs to /home means that following Documents in the file manager takes you to /home. This however caused some strange side-effects, nothing too serious, but enough to make me decide against this approach.

Even should this work, if you change MYDOCSDIR to a non-VFAT partition you don't even get as far as being able to choose documents - the "Jan's Nokia N900" link disappears.

I think that the Camera is not going to be happy storing images to anything but a VFAT partition (though it seems it can be "tricked" using the bind mount). Changing NOKIA_CAMERA_DIR in .config/user-dirs.dirs to /home/user/DCIM did not work, even after chown-ing it to user:root and chmod-ing to 777.

I can't see any reason why changing .documents to Documents wouldn't work. Changing it to /home works after a fashion! The only issues I have found are that for some reason the icon for /home/user looks strange and if you navigate from this to /home/user/MyDocs it's replaced with "Titan's Nokia N900".

I guess that your other plans would work too, though perhaps if you add a "filesystem" bind mount within your MyDocs you wouldn't need to do any linking, or direct mounting to /home/extern, as this would already be pretty accessible when mounted.

I think the best long-term plan is going to be to modify hildon-fm to just show us the whole filesystem! Then the only remaining issue would be the seeming unwillingness of the Camera to save files except via a VFAT partition.

Cheers,
Jan

Re: More efficient and flexible use of internal flash
 

Bad news.

It turns out that perhaps Nokia had good reason for forcing the Camera to only write to a VFAT partition. I don't know why, but if you "trick" the camera to save to the ext3 partition using the methods above, the resulting picture has a strange "cloudy" effect.

This effect (which completely ruins the photo) disappears when switching the camera dir back to a directory which is physically on the VFAT partition. This effect does not occur when the picture is on the VFAT transiently and then copied to the ext3 partition.

I have no idea what is causing this, but I'll have a fiddle and see what happens. I guess a possible solution here would be to monitor the DCIM directory on VFAT, and immediately move any new files to the ext3 partition. More faff!

Cheers,
Jan

Re: [Under consideration] More efficient and flexible use of internal flash
 

I resolve CAMERA's problem!!!!
Now my camera can took pictures on ext3 and vfat!!!!

I resolved it by modification ke-recv (I will try too put patch somewhere...).
But camera has bug too.....

Bug is:
When camera start it is checking on presence folder DCIM on target partition. If it doesn't exist camera create folder DCIM. Here is a bug!
DCIM created with root:root (group and user) but should be user:users.

Like way around (until nokia will fix it) could be change owner:group by command:

chown -R user:users DCIM
chmod -R 664 DCIM

After this camera make picture on ext3 partition!!!
Quality of pictures absolutely same!

full script for this:
1. Turn on Camera
2. Change Location for saving picture on internal memory
3. Make picture. Camera will be closed with error message.
4. in terminal type:
chown -R user:users DCIM
chmod -R 664 DCIM

After this camera will work.

Re: [Under consideration] More efficient and flexible use of internal flash
 

664? It's a directory, so surely 775 would be more appropriate. And why the -R if it's unable to create any files in there?

Re: [Under consideration] More efficient and flexible use of internal flash
 

You right 775 will be better. I really used ug+rw. About -R because it is doesn't matter with or without this....

Re: [Under consideration] More efficient and flexible use of internal flash
 

I'm getting a microSDHC card. Is it exported as USB mass storage? Can I format it as, say ext3 and mount it anywhere?

Re: [Under consideration] More efficient and flexible use of internal flash
 

yes, should be possible. Without modifications, the N900 does even export my
mounted large ext3 /home partition!

Re: [Under consideration] More efficient and flexible use of internal flash
 

Thanks. But an additional question:

Without modifications? I'm assuming you have repartitioned your device to have a large ext3 /home that occupies all of the eMMC. On top of that, you've have created a mount -loop file to use as MyDocs. What did you do to have you home partition experted? And do you mean NFS (or others, such as sshf)s or USB mass storage?

Re: [Under consideration] More efficient and flexible use of internal flash
 

Fremantle exports the full SD card, like Diablo (not single partition like Fremantle does on eMMC).

Re: [Under consideration] More efficient and flexible use of internal flash
 

For now I'm keeping a 27GB ext3 and 2GB vfat for easier reflashing (don't need to apply modifications every time) and for testing parted (as soon as the compiler problems are resolved).

ke-recv apparently hardcodes /dev/mmcblk0p1 as the partition which it exports via USB.
in my case that's the ext3 partition.
so basically I did nothing but reformat the vfat partition as ext3 and
ext3 as vfat.

Re: [Under consideration] More efficient and flexible use of internal flash
 

Titan:
When your partition exported to computer it is not unmounted from n900.
I used your repartition script before.
Is ext3 working well for parallel access from n900 and computer?
I had once corrupted files by simultaneous access from n900 and computer. I could be wrong and it could be other reasons...

Re: [Under consideration] More efficient and flexible use of internal flash
 

That is great news! My Sd card is ext2. I did chown on the DCIM directory and I can have the camera send pictures there.

Why do you need to modify ke-recv?

Re: [Under consideration] More efficient and flexible use of internal flash
 

Hi

please don't mount ext3 simultaneously from N900 and computer!
it will definitely lead to a corrupted fs.
The simplest workaround should be the modifications to
/usr/sbin/osso-usb-mass-storage-enable.sh I described in
http://talk.maemo.org/showpost.php?p...9&postcount=68
assuming that your ext3 /home is /dev/mmcblk0p1
and your vfat MyDocs is /dev/mmcblk0p2
simply set DEV=/dev/mmcblk0p2

Re: [Under consideration] More efficient and flexible use of internal flash
 

For camera you don't need to modify ke-recv.
You need it for normal export partition for computer when connect by usb. additionally without modification you will have problem to access files which was copied from computer.

in my script for change vfat to ext3 when you connect to comuter in mass storage mode partition will be unmounted on n900. After diconnect it will be mounted back.

p.s. For ext2 you will need little bit modify ke-recv. in my patch i add only ext3.

and look like kamera and ke-recv will be fixed in next update because bugs for them get nokia's internal bug tracking number...

Re: More efficient and flexible use of internal flash
 

This seems like the best solution to me. Perhaps now that the camera is working with non FAT partitions the loopback file is not necessary. Although mass storage mode might not behave without it.
Also, would ext2 be preferable to ext3?

What would be useful would be a guide or a script to set this up, I wouldn't really trust myself to do this properly. The closest thing I can find is this.

One other thing somewhat related. How trustworthy are programs with regards to putting large files in /opt? is it likely that one will ever run out of space on the 256mb NAND flash? Might mounting /usr on the internal card be necessary?

Thanks.

Re: More efficient and flexible use of internal flash
 

correct, the loopback file is only useful for mass storage mode (or until recently, to keep the camera happy).
regarding ext2: I guess the Nokia engineers have carefully evaluated both fs
and decided that the journaling in ext3 is not really an issue, but rather an advantage
as there is no fsck before mounting (see bugs #7572, 7708)

you can find HOWTOs in this thread and the other one, e.g http://talk.maemo.org/showpost.php?p...1&postcount=66
I agree we should definitely compile the information on the wiki.

I'm really busy with other stuff ATM and I'd appreciate if someone
could collect the links to all relevant posts and summarize the possible solutions
(e.g. default, single /home with optional loopback, swapped vfat+ext3, MMC partitioning,
what should be exported per USB, $HOME layout and integration of MyDocs).
Then we could modify the ke-recv and upstart packages to support all our solutions
and hopefully propagate them to Nokia, so that it works out of the box.

It is quite likely that you will run out of space on NAND if you install additional applications. The optification is still work in progress and I think there
are several design flaws which need to be fixed first (e.g. /var/lib/dpkg or apt should
be moved, /tmp is too small).
Moving /usr completely to eMMC is AFAIK not an option as it may contain some essential components which are required before mounting /home (but I may be wrong).

Re: [Under consideration] More efficient and flexible use of internal flash
 

I'm interested to know what effect on performance it has, to repartition and format the eMMC.

My understanding is that the eMMC works like a memory card. A while ago I tried formatting one to ext3, it worked at approximately half the original speed. Then when I formatted it on a PC back to FAT32, it was still about half the original speed. It was only when I used the manufacturers flash formatting software did it go back to full speed.

The clusters of sectors in the filesystem should be aligned with the physical blocks of the flash memory. Otherwise it results in multiple accesses, slowing everything down.

The flash formatting software put the start of partition at an offset from the start of the card memory, and had a different size for the FAT data when compared with the Windows partitioning & formatting.

It is necessary to get both the address of the start of the partition, and also the address of the first cluster of data (which follows the metadata) correct, to get the full performance.

This is a link I found to an explanation of cluster alignment:

http://www.hjreggel.net/cardspeed/cs_calign.html

Re: [Under consideration] More efficient and flexible use of internal flash
 

thanks. that's a useful observation and the same problem also affects new harddisks
with blocksizes > 512bytes (AFAIK the flash erase block size is typicall 64K).
I think the the eMMC has a different wear leveling strategy than consumer flash media
but I may be wrong.
I've just uploaded a bonnie++ port to extras-devel so that you can
start performing benchmarks with differents fs and devices (NAND, eMMC, MMC).

Re: [Under consideration] More efficient and flexible use of internal flash
 

some results of my bonnie++ benchmarks
(after disabling tracker "tracker-processes -k"
and using "/usr/sbin/bonnie\+\+ -n0 -f -d." on different media)
I had to disable the file creation benchmarks as they were extremely slow on fat.

fatpart=eMMC partition, fatMMC=on class 6 SD card, xLOOPy=x filesystem in loop file mounted on y on eMMC partition
ext2LOOPext3=easydeb image in home

raw results:
please replicate the tests on your device. there is noticeably variance.

Re: [Under consideration] More efficient and flexible use of internal flash
 

I tried this test with a new N900 with the default partitions as shipped from the factory.

The first time I ran the tests I got poor results, so I rebooted my N900, logged in with putty via ssh, and run the tests while the display was off.

I ran /usr/sbin/bonnie\+\+ -u root -n0 -f -d
with the following directories:

ext3-emmc /home/tmp/
nand /tmp2/
fatpart-emmc /home/user/MyDocs/tmp/
fatmmc-sd /media/mmc1/tmp/
ext2LOOPfat /.debian/tmp2/

The results are here:

raw data:

The microSD card I was using is an old 2GB I had lying around, which is very slow as you can see. What was the model & make of your microSD card?

It looks like you've lost some performance on the fat partition of the eMMC.

Re: [Under consideration] More efficient and flexible use of internal flash
 

it is probably a good idea to setup a standardized procedure for the benchmarks. How about:

a fresh, reflashed device with 51 firmware, with only ssh and gainroot installed,
no applications started, only Wifi, tests run after reboot as user via ssh.
run "tracker-processes -k" and the for each fs use
in addition to the results also report SD card specs, partition scheme and disk usage
for each tested partition.

mine is a Transcend MicroSDHC 8GB Class 6 SD card.

it would be interesting to see a comparison of ext2 vs. ext3 partition
to check whether journaling has a large effect.

Re: [Under consideration] More efficient and flexible use of internal flash
 

Titan, I think what you suggest will give the most accurate results. When I run the test I also thought about reflashing it, but I didn't want to lose my installation. So I ran 'top' to see if there were any busy processes first.

I did run tracker-processes -k

I've remembered now from my previous tests on a flash card a while ago that formating FAT32 to a cluster size of 32KB significantly improved performance. I think most flash cards and USB sticks are formatted to this (though I'm not sure how to see the parameters on the N900). ext 2 & 3 only support a maximum of 4KB per cluster (8 sectors).

I think ext3 journalling has an effect on writes only.

When you did your write testing, what parameters did you use?

Re: [Under consideration] More efficient and flexible use of internal flash
 

I also did use a "fresh" installation, so my results may be biased as well (but all processes were sleeping).
IIRC you need to set -S15 for mkfs.vfat
I used standard vfat options with -F32 and ca. 520MB free on a 2GB FAT partition.
journalling may also affect parallel read+write operations.

BTW, I realized that the NAND performance may be the best-case scenario
as it is a compressed FS and bonnie only writes zeros.
We should also benchmark copying from /dev/random (worst case) and from large typical files (e.g. libqt) from RAMdisk.