you could use # dpkg -P ttf-droid

problem is apt-get will probably complain afterwards.

This is what happens with debian-like systems. The package is compiled with some dependencies, packaged with some more, and then the user is stuck. Only solution is to compile yourself with the options you actually need/want.