I believe I ran into it as well. I had trouble tracking it down, but it seemed to go away after I installed some fonts. I really can't tell you the root cause though. My other thought is that it might be some left-over part of hildon interfering the xfce. Make sure you kill off hildon session and restart the x server, before starting xfce.

-John