In theory, it'd be possible to do just what Desktop Windows or Linux does: Have proper drivers for each bit of hardware, and abstract everything.
You should be able to write once, and have it work on most all systems powerful enough to use it - Just like with your Windows system(It doesn't matter if its a Dell, HP, or custom build. Whether its an AMD or Intel chip[both have x86/x86_64 compatible instruction sets])
If you have to tweak your application to make it run on different phones running the same OS... Something's wrong here - While you may have extra features in new ARM chips, you can always compile for a "lowest common denominator" instruction set, just like you compile for i386 on the Desktop.