I still dont see why (the lowest common denominator)
running a "headless" X11 server running on your phone with a VNC display driver and a VNC input driver [which you locally connect to via a vnc client] shouldnot work.
It is not x11 or vnc it is: X11 via vnc.
Sure it is a overhead going through kind of framebuffer driver and local network but the VNC "last mile" gives you also input and display translation [from touchscreen to rightmouse click (+) drag and drop, zooming] which are sometimes needed for non fingerfriendly X11 applications.
[Or can you rebind the rightclick drag and drop gesture to shift + drag and drop on any open gui toolkit? Im hoping for emacs2020.]

This is done on https://www.youtube.com/watch?v=XfMLzlki9XE but the connection vnc server to vnc client is over wifi and not locally.
[But it shouldnt be laggy anyway, except he has a very very very bad wifi connection]