I fixed it!
In case someone else has this problem and came here by a search or something I will describe how I fixed it.
However it's not very likely, I have myself to blame for messing around to much when I'm tired and bored

You need to have a .vcf-file with your contacts backed up, I had one in ~/.osso-abook/db/backup.vcf

Solution:
Reinstall the addressbook:
sudo gainroot
apt-get install --reinstall osso-addressbook
Kill the addressbook soo you're able to start a new one, or something like that anyway:
killall osso-addressbook
killall osso-abook-home-applet
Move the old addressbook-files to another folder, so we can make a new one:
mv ~/.osso-abook/ ~/.osso-abook_bak
Import the backed up contacts:
osso-adressbook-backup -i PATH_TO_VCF-BACKUP
Start the adressbook from gui, should work.