metalayer-crawler
 

I wanted to be able to use the file manager to get at everything, not just the stuff in my home or on the cards. So, I put in a symbolic link (please don't do this :eek:):

cd MyDocs
ln -s / everything

However, eventually i rebooted the machine, and it became unbearably slow, even after reboot. Eventually, i ran 'top', and it said that metalayer-cralwer was using over 90% of the CPU. Probably wanted to use it all, but i was running 'top', right? Oh, and don't even talk to me about my crappy battery life. And i have two.

Some times, it was so unresponsive, that i'd click something, and it would be ten minutes before it actually did anything.

Removing the symbolic link, and rebooting, fixed the problem. Except, that i still want filemanager to be able to see the whole system. For now, i'll drop to an xterm and just do it, whatever it is. By the way, just removing the symbolic link is not enough. I also needed to reboot. And, the machine complained that some database was corrupt for a couple reboots. Presumably, metalayer-crawler had to rebuild it's database.

I recently reflashed, i'm now running osso-rx-44 20007-49-1

I thought this bug had to do with the roughly 7,000 new images, audio, etc., i'd just put on the machine. However, i looked up this bug with Google. Google can be really smart.

Bug 978
https://bugs.maemo.org/show_bug.cgi?id=978
talks about this bug.The bug report claims it's fixed.
And, Bug 1760 says it's fixed.
https://bugs.maemo.org/show_bug.cgi?id=1760
I'm sure some bugs were fixed, but, bug 1842 says it isn't.
https://bugs.maemo.org/show_bug.cgi?id=1842
The real problem is that metalayer-crawler should not follow symbolic links. That means it's gotta do an lstat on each file, check to see if it's a symbolic link, and if it is, ignore it.

The 'scp' utility also needs to ignore symbolic links. I think the 'busybox' version of 'find' suffers from this. I'd really like to ditch 'busybox'. For example, 'xargs' also pisses me off.

Again, the work around is not to have symbolic links that create filesystem loops. For example, i have handy short links to the SD cards in my home directory. These don't create loops (or problems).

What, exactly, does metalayer-crawler do for me? It's supposed to put the location of media files into some database somewhere. What looks at it? If 'Media Player' does, it's news to me.

If i wanted to contribute to fixing some of these bugs, how do i get started?

Re: metalayer-crawler
 

That's my understanding of it, too: It indexes media files so that you don't have to. Really quite redundant, since the system is already designed in a way that encourages organization (with clearly named folders on the user's home that every Maemo-targeted program knows to work with, and file transfer tools having shortcuts to choose those folders).

Personally, I ran apt-get remove metalayer-crawler and never saw it again. Note that this seems to permanently kill the thing; trying to reinstall results in apt complaing that it cannot find the package.

Having said that, it would probably be more useful if everything else (eg: Canola, the file browser) used its database. As is, I couldn't justify having it run, given that the things using it are so few.

Re: metalayer-crawler
 

I just remove metalayer-crawler

I also then, create a link,

cd /home/user/MyDocs/

ln -s / /root


Now when you open the file manager, you can access the whole filesystem. Also just because you can access everything, does not mean you can write everywhere.

This is a pretty well known issue, I am sure there were already existing threads you could have posted this in?


As far as find goes

find / -type l (to find links)
find / -type f (to find files)
find / -type d (to fid directories)

I have had lots of problems with metalayer-crawler in the past, it is usually relating to a large amount of file.

Old maemo mapper files, before it was kept in DB

Large amounts of music, photo, or roms can also cause metalayer-crawler to freakout.

Even installing KDE can cause metalayer to freak out, because it has so many files.

My files were usually under /usr/local which caused this problem.

Re: metalayer-crawler
 

metalayer-crawler can hog the system and without such general symlinks. I hope tracker will behave better. Not sure if it replace this one little bugger.

Re: metalayer-crawler
 

This is really a rant. Please ignore it.

I guess my main quibble with busybox 'find' is that it isn't GNU 'find'. It's not just that it does the wrong thing with symlinks. So, let's say you have file names with spaces in them, perhaps it's Pink Floyd's 'Comfortably Numb.mp3'. And, for some reason, the files aren't readable. No problem,

find . -type f -print0 | xargs -0 chmod a+r

will change every file in the current directory with all subdirectories, right? Except that the busybox 'find' doesn't know '-print0', and it's 'xargs' doesn't know '-0'. And if you try

find . | xargs chmod a+r

you get errors like:
chmod: ./Comfortably not found
chmod: ./Numb.mp3 not found

For the record, these options change the protocol from newline terminated to null termintated strings. I've no idea why 'xargs' never took each newline terminated line and made it a single argument to the given command without using a shell, but that's history. So 'xargs' with '-0' knows to exec 'chmod' with the whole strings, and not broken at shell break points, like spaces. In fact, if fixes lots of shell meta character problems.

But, you say, 'find' has '-exec', which can run 'chmod' directly. However, busybox's 'find' doesn't know '-exec', which was available in 'find' in Unix Version 7, certainly by 1981. So, i'm left without a solution. I regularly get files with spaces and other shell meta characters in them (and Unix has allowed them since forever), and i don't have utilities that can cope. Either fix 'busybox', or drop it in favor of the real apps.

In 1995, i had a 486/sx25 laptop with 16 MB RAM and 170 MB disk. I ran Linux with full Emacs, gcc, CVS, apache, browser in X, and had 70 MB disk free for my stuff. Now i have a 400 MHz Arm with 128 MB RAM, 18.25 GB disk, and 'ls' is crippled. We've clearly optimized the wrong thing. We should start with a complete system, not some minimized hack. I'm just saying it could be better. It could work at the early 1990's standards.

I saw somewhere that one can not uninstall 'busybox'. But, one can install the real 'grep', 'diff', 'find', 'xargs', etc., from .deb files and install them with 'apt-get -f' to force apt-get to ignore the obvious conflicts with 'busybox'. At some point, it's quite clear that the repository manager is working against us. And we continue to use it because it is sometimes convenient. This rant continues on about evil things like dependencies, shared libraries, and so on, but i'll stop now.

Re: metalayer-crawler
 

scp doesn't know about symlinks? use rsync instead.

Re: metalayer-crawler
 

So why didn't you just ask for what you wanted :D

http://penguinbait.com/find.gz

http://penguinbait.com/xargs.gz


GNU find and xargs

Re: metalayer-crawler
 

The new Media Player .m3u files are simple lists of file paths.
Instead of

file:///media/mmc2/...
its
/media/mmc2/...

If i'd made the change, i'd have let it parse either form. Backward compatibility is a good thing. And, the old form could be dropped... never.