I did my own little research with flash access times. I did it via multiple usage of command

time dd if=src of=dest bs=64k count=3000

where 'src' and 'dest' were combinations of /dev/mtd5, /dev/mmcblk0p3, /dev/mmcblk1p2 (I set swap here) and /dev/null or /dev/zero.

The results are (time in seconds, measured with stock PR1.2 kernel):

MTD5 (flash-on-top-of-CPU, root FS):
READ -
wall time = 9.93,
sys time = 9.79
battery current=1400

/dev/mmcblk0p3 (regular N900 swap space, part of 32GB internal flash):
READ -
wall time = 9.87
sys time = 2.57
battery current = 1223, repeated=1870
battery current=1252
WRITE -
wall time = 11.27
sys time = 2.28

/dev/mmcblk1p2 (a partition on micro SD, class 6):
READ -
wall time = 14.94
sys time = 3.44
battery current=977, repeated=1620
battery current=1100
WRITE -
wall time = 19.23
sys time = 3.10

----------------------------------------------

So, the interpretation - the current usage of MTD5 as root file space is useless from many point of view - the Read performance is the same as internal 32GB flash BUT (!) - CPU usage is overtop!

Actually, I strongly suspect that MTD5 access performance is limited by CPU performance (due to compression in UBIFS?) - wall time == system time.

The move of root FS from MTD5 into 32GB may benefit not only in space but CPU usage and battery life. We need only switch off the disk block caching for file content - it seems that if kernel finds disk block in cache it copies block from that cache and that consumes MORE energy than DMA read from flash chip: repeated reads take more battery current than first one.

After root FS moved to 32GB flash the MTD5 can be used without compression for first order swap, it would greatly accelerate overall N900 performance (today swapout-swapin of huge behemoths like modest/gst-video-renderer may take up to 10secs during call answer). Or use it as file cache (my proposal #3).

EDIT: Battery current updated with more accurate measurement (N900 set offline). No significant fluctuation anymore.

What unit is your battery current figure in? How did you measure it?

The big difference for nand vs emmc isn't in sequential read/write.. Whereas the emmc can do something like 4-8Megabyte/s write per second sequentially, the built-in flash translation layer can not at all deal with random write (such as swap, /opt and MyDocs), and the speed for 4K random writes degrades to a few hundred kilobytes.

The OneNand has no FTL, so we can manage it ourselves, and we have a 600MHz cpu and 256 megs of ram, plenty of room to do something intelligent. As a result, the onenand with ubifs is a few magnitudes faster at serving IOs from a multitasking demand-paging operating system.

Reads are same magnitude in speed, as long as there are no writes interleaved.

sleep 15
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
time dd if=$1 of=/dev=null bs=64k count=3000
cat /sys/class/power_supply/bq27200-0/current_now

(bq27x00_power.ko module is from Matan's collection)


Swap is not an issue at all - it is read and written in big blocks and it is located NOW in 32GB flash (non NAND!). Also I tested files on /home/user Ext3 - the file read speed is about the same - 19MByte/sec. And reading uncachable /usr/lib/locale/locale-archive from UBIFS gives 18MByte/sec. I don't see a difference besides CPU resource and battery usage.

Short files from /etc and gconfig can land in kernel cache pretty soon. And we may combine file caching in MTD2 for that - it would get the advantage of fast short access if UBIFS implementation works fine for short files.

Besides of that it is possible to use not ext3 but some more flash-friendly FS - LogFS is released in May, for exam.

May be. But I don't see it now - my battery depletes too fast and I know the reason now. Personally, if there is a choice I prefer to have some slow disk access but much less CPU expenses, in my long UNIX experience with old and slow disk equipment it handles file access very well because of kernel buffer cache, especially in multitasking. But I don't think we would have a slow access.

I have a home server which is configured to have root FS read only besides some critical files for X11 which are symlinked to /var and the whole /var is located in Generic Compact Flash. It is done to prevent disk spinning and works very well, at least faster then N900

So, don't worry about writes... well, gconf should be killed or replaced because it writes on any bump. But we can symlink that stuff into OneNAND.

Just try gpodder on your N900 with a moderately big datbase (say 100-200 podcasts). When you start a download, the 8 megabits coming from 3g to MyDocs, the 4k random write/read swap activity by the 8mbit io pressure making linux swapout to increase cache, and the random read/write to ext3 to update databases results in half-minute long stalls..

My home server also runs on flash. With ext3, the squid proxy server stalls for half a minute at a time when the flash slows down. With nilfs2, which by accident is quite performance optimized for flash, there are no stalls and doing a backup of the squid cache dir is also about 8 times faster than with ext3.

We don't know the units, as they depend on the resistance of a specific resistor on the board. But the values are linear, so they are useful for such comparisons.

But the swap is actually on the eMMC, so I don't really get your point. In normal activity there is very little write access to the root filesystem.

Do you have some numbers for "a few magnitudes faster ..."? It sounds too incredible to be true.

Using root file system on eMMC is slower, even though the hardware is better, as those benchmark show, due to concurrency with swap access also on eMMC.

The best options I can think of are:

• Use a microsd card with root filesystem on eMMC and swap on uSD (or vice versa)
• Use root on eMMC and swap on NAND.

Both options should give better performance than the current setup and give a larger (configurable) root file system size.

The downside to first method is probably increased power usage due to uSD card.

The downside of the second is limiting of swap to 250MB rather than the current 768MB.

My measurements show the DECREASED power usage with uSD (see previous message). It is one of latest uSD Class 6 cards.

I tested the two swaps for single system, it works - just add another swap partition and kernel puts it as "secondary" priority swap. It works fine with automatic extension to the 2nd layer swap.

But I am afraid the usage of NAND has a disadvantage in battery performance. I observed high CPU usage for reading just raw 64KB data blocks from /dev/mtd5 and it seems that NAND performance is actually limited by CPU. Plus higher battery usage.

It may be because of block device emulation (not used in UBIFS) but this layer should be used for swap operations too.

So, it may have a sense to symlink or file-cache small files to NAND + UBIFS. For small file access it may be a good trade-off (Battery vs performance).

Caching takes care of all our concerns regarding small file access and so on, and also swap. However, this is of course only possible if we have enough RAM to cache things in ;-)

The few magnitudes claim comes from something called write amplification. The native flash block size is some 256kilobytes. Writing a 4k sector (as the OS often does) causes the emmc/sd to erase a new 256kb block from a pool of spares, (if there isn't one already erased available), read in 256kb, modify 4k of it, write it to the new block, put the old block into the spare pool.

Thus, a 4k write request from the OS was turned into a 256k read, 256k erase and 256k write. 256/4 is a factor 64 already

On the emmc/sd, 256k physical blocks are linearly mapped to 256k logical blocks.

With nand we can do it smarter, while the erase size is huge 256k, an erased block doesn't need to be completely written in one go. We can write 4k and fill in the rest later. We have the cpu to map blocks of 4k nonlinearly onto the 4k flash, so that even if the filesystem wants to update a table at the start of disk, a file entry in the middle, and some data elsewhere, we turn it into 3 writes into the same block.
This is roughly what modern SSDs also do, and they have ARM cpus and 64 or more megs of ram to do it, which explains why there are no sd cards doing clever stuff.

Now, normal filesystems are optimized for rotational media. There all kinds of non-sequential access becomes slow. So, they operate on the idea that data is read more often than written, and thus spend alot of effort on placing the data optimally, so that related data is close together. This is exactly the opposite thing you want to do on a flash based device.

nilfs2 and logfs, however, are very good for performance on emmc/flash
(and ubifs, but that can only be used on nand)

So yes, if we can use LogFS, and if we can have swap on its own device, or someone writes a new loop block device specifically to optimize swap on flash, having non-media system files on emmc would be doable..

Without logfs though, I suspect we get more of all sortsof "stuttering" people cmplain so much about even now

So I have a 32 gb I think class 10 card. Any chance I could use it for this? How, exactly?

I have been wondering about that myself, could anyone confirm if 32GB cards work without problems? The phone is officially only rated for 16GB cards AFAIK.

Edit: found a thread that verifies that 32GB microSD cards work:
http://talk.maemo.org/showthread.php...t=32GB+microSD

I faced a similar problem running a regular linux installation on top of a RS-MMC (damn slow, and too little for a regular linux distro)

I tried almost every solution available (unionfs, aufs, symlinked directories and so on) and finally I tried BTRFS with compression enabled and SSD mount opition.

The result is simply amazing, the system is incredibly fast, the bootime is almost a quarter than the one over a regular ext3 FS, and there is no space wasted as happens with the readonly fs overlaid by writable fs, trough aufs or unionfs.

So I vote for completely ignore the 256MB NAND and use a partition on 32 GB eMMC formatted with compressed BTRFS for the whole FS.

Eventually the 256 MB nand can be used for caching purposes, as suggested before.

The flashing problem can be eventually solved flashing not directly the emmc but the 256MB nand flash as happens now.

A script on the first boot can format the emmc partition and then copy everything on it (adjusting the configuration files when needed), so no worry for the user data on the vfat section.