All this made it seem like more of a hardware problem, and not just something in software. (Also, I haven't used anything like fm radio which can switch between the two in a really long time).
cat /sys/devices/platform/gpio-switch/headphone