You do not need an xserver, you should be able to talk to the existing one. In fact given time I'd investigate what the easy arm debian chroot is doing.
Good luck. One thing I mean try try is touching a file before testing on a normal x86 system, then using find / -anewer /fileyoutouched to create a list of files for a very minimal chroot.
strace xterm 2>&1 | grep 'open("' | grep -o '"/.*",' | grep -o '[^"]*' | sort | uniq | grep -v "/dev" | grep -v "/proc" > list_xterm.txt