maemo.org - Talk

maemo.org - Talk (https://talk.maemo.org/index.php)
-   Multimedia (https://talk.maemo.org/forumdisplay.php?f=32)
-   -   How tracker works (https://talk.maemo.org/showthread.php?t=61960)

Flandry 2010-09-09 03:03

How tracker works
 
For those that don't know, tracker is the utility that watches for new media and makes thumbnails of new images on your N900. It has been widely cursed and hated for its poor behavior and can cause sluggish response time while doing its thing -- which it tends to do right when you want to use your N900, like when taking a bunch of photos.

There is a configuration file for tracker in /home/user/.config/tracker/ called tracker.cfg. By default it has quite a few comments that can be helpful in deciphering the settings, but i find i'm still not really sure what the default behavior of tracker is. There's also an app called tracker-cfg that lets you edit these settings with a GUI.

There are three categories you can put individual folders into that determine how tracker treats them (please correct any errors here) (Edit: see this later post for links to more complete descriptions):
  • crawl - scan the folder only when it is first mounted and index any relevant files
  • watch - every time the folder contents change, reindex it
  • don't watch - don't ever index the directory

The way these are specified can be a bit confusing, because for example you specify a root folder to be watched (meaning tracker will automatically watch all its subfolders). It's not clear if it works the same way for the other two lists.

I'd appreciate a description of the behavior if anyone understands it. In particular, i'm wondering what the default rule is for folders not specifically included in any lists. and whether the "do not watch" list is for directory roots or just individual directories.

The current configuration on my N900 is supposed to be not watching a specific directory of web images, but i keep seeing them show up in the images listing after rebuilding the database. (tracker-processes -r)

Some of the other options are a bit confusing as well. Is there better documentation for this somewhere?

Thanks in advance for whatever insights you might have on the inner workings of tracker. I'd like to tame it! :eek:

quingu 2010-09-09 13:02

Re: How tracker works
 
You should check out the program called tracker-config. It not only lets you set all these parameters manually (thus making tracker actually, y'know, bearable), but it also helps on understanding how it works.

Rob1n 2010-09-09 13:16

Re: How tracker works
 
Tracker is all open source, so you can check out exactly what's being done from the code (the manual pages are the best place to start). According to the trackerd manual page, all three groups automatically include all (non-hidden) subdirectories.

msa 2010-09-09 13:17

Re: How tracker works
 
shouldnt it be so, that it indexes pictures when you open the photo-viewer or movie/song-files when you start the mediaplayer?

Netweaver 2010-09-09 13:43

Re: How tracker works
 
mmmmm
I would like it to index the new pictures/videos AFTER I've shot them, after LEAVING the application that made them. Not during...

In general, while I'm doing the creation, I would want as little other system activity as possible. Real multi-tasking on a limited resource device only goes that far ... Ask Apple and Google.

So while doing time critical stuff, I ideally want all non-critical stuff frozen.
That would help the stuttering while filming.

But that's asking too much I guess ...

OVK 2010-09-09 13:55

Re: How tracker works
 
Quote:

Originally Posted by Netweaver (Post 811685)
mmmmm
I would like it to index the new pictures/videos AFTER I've shot them, after LEAVING the application that made them. Not during...

In general, while I'm doing the creation, I would want as little other system activity as possible. Real multi-tasking on a limited resource device only goes that far ... Ask Apple and Google.

So while doing time critical stuff, I ideally want all non-critical stuff frozen.
That would help the stuttering while filming.

But that's asking too much I guess ...

I think that it should be possible to send some kind of suspend/pause command for tracker. This could be trickered by for example when the lens cover is opened (maybe this could be integrated to Lens cover reminder application?). I assume that Camera app is closed source so this functionality probably can not be integrated to camera itself (?).

Are there other non-critical processes than Tracker running that are so resource consuming that they should be stopped too while filming?

(Disclaimer: I am not a developer so this is mostly something that I have been thinking and don't have a slightest idea if this makes really any sense at all... :o )

toxaris 2010-09-09 14:06

Re: How tracker works
 
Something like this:
Pictures get indexed after camera app is closed, the same with videos.
Inside media player there should be a button to chose "Auto Indexing" (index always one) or "Manual Indexing".
When Manual Indexing is chosen you get another button that says "Scan for new media", that triggers the Tracker to index new music and videos. It works that way on Symbian S60 3rd if I dont remember wrong.

Flandry 2010-09-09 15:37

Re: How tracker works
 
Quote:

Originally Posted by quingu (Post 811659)
You should check out the program called tracker-config. It not only lets you set all these parameters manually (thus making tracker actually, y'know, bearable), but it also helps on understanding how it works.

Thanks. I actually mentioned it by name in my post. ;)

It is handy but i uninstalled it when i realized that the default configuration file is actually commented clearly and tracker-cfg is just a wrapper for a single user config file. It is nice to have the options to rebuild database &tc right in the app, but that's also easy to do manually.

Quote:

Originally Posted by Rob1n (Post 811667)
Tracker is all open source, so you can check out exactly what's being done from the code (the manual pages are the best place to start). According to the trackerd manual page, all three groups automatically include all (non-hidden) subdirectories.

Thanks Rob1n. I hear so many complaints about tracker i just assumed it was one of the closed bits of maemo. It does seem unduly hard to set it up to behave as desired, so i'll check out the manual.

One thing you can change that will have very apparent results is increase the throttle. If it's around 10 the slowdown will be much less noticeable.

Netweaver 2010-09-09 15:59

Re: How tracker works
 
If I'm filming I'm not interested in getting new email notices ... or IM messages. I want a good steady shot, no dropped frames, maybe even some help from the CPU in non-favorable light conditions.

If the hardware/software implementation is not powerful enough to do all at once, then I think things should be pragmatic.
I do think the Maemo ecosystem is too much desktop Linux driven, actually ahead of it's time, designed for devices with ample resources, allowing the walhalla of real-time multitasking at the expense of user experience and performance of the frontend tasks.

Lets hope Meego has better provisions for preserving User Experience and performance. But honestly, I think better HW is the only thing to keep all running as envisioned during design.

Flandry 2010-09-09 16:07

Re: How tracker works
 
Here's the man page for the tracker configuration file: http://linux.die.net/man/5/tracker.cfg

Here's the project page: http://projects.gnome.org/tracker

Now that i've read up on it, i'm more curious than ever at some of the options. AFAIK, there's no built-in global search capability in maemo, so all of the file content options in tracker configuration seem pointless, yet not disabled in the default tracker configuration. Does anyone know what use maemo makes of tracker's content indexing besides the very basic task of thumbnailing visual content?

nicolai 2010-09-09 16:16

Re: How tracker works
 
All media meta data like artist, album, ... in the mediaplayer is based on
tracker information.

nicolai

Netweaver 2010-09-09 16:24

Re: How tracker works
 
Certainly in Meego it actually stores info by itself (from the meego-dev) mailing list :

Quote:

From patrick.ohly at intel.com Wed Sep 8 04:32:25 2010
From: patrick.ohly at intel.com (Patrick Ohly)
Date: Wed, 08 Sep 2010 13:32:25 +0200
Subject: [MeeGo-dev] data storage and indexing in MeeGo - a survey
Message-ID: <1283945545.7176.302.camel@pohly-mobl1.ikn.intel.com>

Hello!

As those who I have been pestering with questions in the past already
know, I'm trying to get an overview of which data we store in MeeGo core
and how it can be accessed and/or searched. This email is an attempt to
close some gaps and extend the list that I'll present below. I also have
some general questions around Tracker.

Typically the answer consists of a domain specific storage and API
(mkcal + sqlite) plus Tracker for indexing the data. In some cases,
Tracker is also the main storage of the data (QtContacts +
qtcontacts-tracker). Sometimes there are additional portability layers
on top of the MeeGo platform component (QtMobility Messaging on top of
the Qt Messaging Framework (QMF)).

The next question then becomes how data items are referenced at the
various levels, and how the data content is mapped. For example, can I
take the Tracker search results and open the corresponding item using
domain specific tools and APIs?

This mapping is mostly only defined in the source code which implements
the Tracker indexing. I'll provide pointers below.

Finally, how can a permanently running app be notified of data changes?

Here's what I gathered so far - please extend and/or correct.
...
Very interesting and hardcore article on data and searching it. Most of it Meego oriented but interesting in the wider tracker context
http://lists.meego.com/pipermail/mee...er/005546.html

PathFinder@9GS 2010-09-09 17:06

Re: How tracker works
 
from what I understand..how about we juss write few scripts.. mk a desktop shortcut or qbw.. and turn on /off indexing/watchdog with the scripts..
if we cant merge those scripts with the apps like camera an all.. atleast we wil hv option to turn off indexin/watchdogs before using the camera... :D
another idea... will write a script fr qbw i guess

twaelti 2010-09-10 12:04

Re: How tracker works
 
One issue I have with the tracker.cfg is that there seems to be misunderstanding about the two Battery flags in it:

The config file itself and the help in the "Tracker Cfg" state that:
Code:

# Set to true to disable indexing when running on battery
BatteryIndex=false
# Set to true to disable initial indexing when running on battery
BatteryIndexInitial=false

However, the tracker.cfg man states that:
Code:

BatteryIndex=BOOLEAN
    Enable or disable indexing when on battery power.
BatteryIndexInitial=BOOLEAN
    Enable or disable initial index sweep (i.e. the very first indexing) when on battery power.

These two statements seem to be 180° contradictory.

For me, I now have set these flags to FALSE and expect that this DISABLES indexing while ON BATTERY.

Rob1n 2010-09-10 12:55

Re: How tracker works
 
Quote:

Originally Posted by twaelti (Post 812571)
One issue I have with the tracker.cfg is that there seems to be misunderstanding about the two Battery flags in it:

The config file itself and the help in the "Tracker Cfg" state that:
Code:

# Set to true to disable indexing when running on battery
BatteryIndex=false
# Set to true to disable initial indexing when running on battery
BatteryIndexInitial=false

However, the tracker.cfg man states that:
Code:

BatteryIndex=BOOLEAN
    Enable or disable indexing when on battery power.
BatteryIndexInitial=BOOLEAN
    Enable or disable initial index sweep (i.e. the very first indexing) when on battery power.

These two statements seem to be 180° contradictory.

For me, I now have set these flags to FALSE and expect that this DISABLES indexing while ON BATTERY.

They're not really contradictory - the manual page doesn't say which value enables and which disables. I'd be surprised if the details in the config file are correct though. Setting BatteryIndex to false to enable indexing while on battery is just so wrong - anyone who implements that deserves to be staked out on an anthill and smeared with honey.

Flandry 2010-09-10 17:31

Re: How tracker works
 
To follow up, i finally removed all but the two camera (DCIM) folders from my watch folders list to solve the problem of tracker indexing folders i don't want it to. It seems that either having the root of a child folder in the watch folders root list overrides the do not watch list, or the do not watch list is not recursive.

I added the music folders to the crawl list, because i only need those indexed when i (infrequently) change the music and can trigger a rebuild myself when i do that.

I tried increasing the delay time to 120 s. That seems to be a good way to work around tracker jumping in during the middle of a short filming or series of photos. However, it seems i'm not notified now of the reindexing countdown.

It may be something else that broke that, though.

teohhanhui 2011-02-25 07:28

Re: How tracker works
 
LowMemoryMode=true seems to cause the reindexing ETA not to display.

Can anyone confirm this?

vkv.raju 2011-03-10 03:52

Re: How tracker works
 
Can someone please tell me how to check the current version of tracker in our devices. I am running PR1.3+CSSU-Updates (latest is 13).

There is a current (hot) discussion going on at meego-dev miling list about the future of tracker. The point that came out (w.r.t tracker in maemo) is to which version we are running. Looks like we are running the 0.6.95.x version of Tracker. But their latest version is 0.10.1 and they claim that the latest version is way better than the one that maemo uses.

So, can someone please tell me how to check which version of Tracker are we running. 0.10.1 or older?

http://lists.meego.com/pipermail/mee...ch/482001.html
http://lists.meego.com/pipermail/mee...ch/481890.html (Thread start)

Thanks in advance,

dsawhney 2011-03-10 04:08

Re: How tracker works
 
Quote:

Originally Posted by vkv.raju (Post 964580)
Can someone please tell me how to check the current version of tracker in our devices. I am running PR1.3+CSSU-Updates (latest is 13).

There is a current (hot) discussion going on at meego-dev miling list about the future of tracker. The point that came out (w.r.t tracker in maemo) is to which version we are running. Looks like we are running the 0.6.95.x version of Tracker. But their latest version is 0.10.1 and they claim that the latest version is way better than the one that maemo uses.

So, can someone please tell me how to check which version of Tracker are we running. 0.10.1 or older?

http://lists.meego.com/pipermail/mee...ch/482001.html
http://lists.meego.com/pipermail/mee...ch/481890.html (Thread start)

Thanks in advance,

This should be the PR1.3 version - http://maemo.org/packages/package_in...-35maemo1+0m5/

vkv.raju 2011-03-10 04:28

Re: How tracker works
 
Quote:

Originally Posted by dsawhney (Post 964589)

So, from the link it looks like 0.6.95-35maemo1+0m5 is the version for tracker. But how to find out if CSSU has already updated it to the latest? If not, may be we can request the CSSU team to include the latest version of tracker in the CSSU?

dsawhney 2011-03-10 04:42

Re: How tracker works
 
Quote:

Originally Posted by vkv.raju (Post 964595)
So, from the link it looks like 0.6.95-35maemo1+0m5 is the version for tracker. But how to find out if CSSU has already updated it to the latest? If not, may be we can request the CSSU team to include the latest version of tracker in the CSSU?

This is the list of packages updated as part of CSSU, tracker isn't one of them

http://gitorious.org/community-ssu

javispedro 2011-03-10 04:55

Re: How tracker works
 
Quote:

Originally Posted by Flandry (Post 812865)
To follow up, i finally removed all but the two camera (DCIM) folders from my watch folders list to solve the problem of tracker indexing folders i don't want it to. It seems that either having the root of a child folder in the watch folders root list overrides the do not watch list, or the do not watch list is not recursive.

You might want to use my patched Tracker, because the realignore.patch that I made tries to fix that weird behavior ;)

vkv.raju 2011-03-10 05:19

Re: How tracker works
 
Just out of curiosity, so is it like not possible to use the latest tracker version directly over the PR1.3 or the reasons something else (compatibility, breaking, etc)?

If it is possible to update to the latest, then may be we can request CSSU team to include the latest tracker version in their future updates.

nicolai 2011-03-10 07:33

Re: How tracker works
 
AFAIK there are big API changes between tracker
0.6 and the recent version.
The whole 0.6 API is deprecated. The query language
used in the 0.6 was RDF and the 0.10 uses
sparql. So the chances that upgrading to a new
tracker version don't break anytihng and
or makes the performance better is very small.

Flandry 2011-03-12 19:39

Re: How tracker works
 
Quote:

Originally Posted by javispedro (Post 964601)
You might want to use my patched Tracker, because the realignore.patch that I made tries to fix that weird behavior ;)

Thanks for that.

It sort of works like i want it to now with the changes but it's good to know that it really does behave in an unexpected way by default. Is your patched version going to end up in CSSU?

javispedro 2011-03-16 17:44

Re: How tracker works
 
Quote:

Originally Posted by Flandry (Post 966457)
It sort of works like i want it to now with the changes but it's good to know that it really does behave in an unexpected way by default. Is your patched version going to end up in CSSU?

That is my plan. Well, that, or get the latest Harmattan 0.6.x series, which fixed two of the bugs that have been workaround on my patched version.

Unfortunately I am yet to get a proper schedule allocation to start contributing full time to the CSSU though, so if you think my patches are stable enough and want to push them to the CSSU, feel free :) .

JadeH 2011-03-20 23:21

Re: How tracker works
 
Quote:

Originally Posted by javispedro (Post 964601)
You might want to use my patched Tracker, because the realignore.patch that I made tries to fix that weird behavior ;)

Nice, I just installed it!

Fallingwater 2012-09-07 12:28

Re: How tracker works
 
I had problems with the tracker hogging the CPU, so I used tracker-cfg to disable everything I could see. I'm assuming this has disabled the tracker entirely, as I no longer get insane CPU activity after poweron or filesystem operations.
Question: is there anything inherently bad in doing this? I don't care about thumbnails and I'm not using the stock mediaplayer (Someplayer has its own indexing system, I think).

pichlo 2012-10-30 13:45

Re: How tracker works
 
Bump. I wanted to ask the same question as Fallingwater - except I would go even further and uninstall tracker altogether. Can anyone foresee any harm in doing either?

sixwheeledbeast 2013-01-09 16:59

Re: How tracker works
 
Quote:

Originally Posted by javispedro (Post 969070)
That is my plan. Well, that, or get the latest Harmattan 0.6.x series, which fixed two of the bugs that have been workaround on my patched version.

Unfortunately I am yet to get a proper schedule allocation to start contributing full time to the CSSU though, so if you think my patches are stable enough and want to push them to the CSSU, feel free :) .

Did tracker.cfg actually get updated?
Was this "patched" version stable enough to put it CSSU?
I know there was a mention of tracker doing odd things recently.

Currently can't get may device to watch the memory card properly.
It updated this morning after I added new music but just ripped another CD and it's not added the new tracks.

I'll try and force it to rebuild later.

marmistrz 2013-01-09 19:32

Re: How tracker works
 
Quote:

Originally Posted by nicolai (Post 964671)
AFAIK there are big API changes between tracker
0.6 and the recent version.
The whole 0.6 API is deprecated. The query language
used in the 0.6 was RDF and the 0.10 uses
sparql. So the chances that upgrading to a new
tracker version don't break anytihng and
or makes the performance better is very small.

I asked at the tracker mailing list and the API is different indeed. But they said too, that newer tracker is much faster.

Maybe (not a highest priority now) we should upgrade tracker and fix what's not working (guessing: image-viewer). there are already some replacements, so we might talk with their devs.


All times are GMT. The time now is 02:19.

vBulletin® Version 3.8.8