I was able to improve it by executing (in terminal):

amixer cset numid=16 3

But it seems to only last for that 1 phone call, and may have undesirable effects on the speaker...