Well this should be possible with tracker without much difficulty. Does anyone know how to use its query API? A quick search on the internet only reveals sparql which I don't know the syntax.
I don't think that it indexes contacts but for that there is the osso api...