This was a step in that direction nokia once did when they created the first smartphones, with the communicators...
The first communicators used a desktopos (GeOS, However crippled down a bit and without a PC compatible screen) with intel 386 and 486 processors.

But the way you are describing it with easydebian and such, is the way i WANT to use my N900...
-I want a pocketcomputer with a phone, and not a phone with apps...
and this approach was also what once upon a time made the communicators so legendary.
However, arm or x86 doesnt really matter, if it is running linux.