Does anyone know how to run a memory testing software like memtest86 on the 770? Or even some testing software to see how well the built in wifi radio is functioning.

My concern is how to tell if any part of the 770 hardware is malfunctioning and not buggy software that's causing lockups and crashes.

Well, nice timing. I'm just also having hardware instability issues and want to find a way to test what exactly is wrong. Check this and the followup messages in maemo developers mailing list:
http://maemo.org/pipermail/maemo-dev...er/005527.html

PS. Currently we suspect that this hardware instability might be caused by low battery charge.

Just my add:
Ive noticed that R&D mode will make my 770 more prone to random reboots. Since most of us have probably used the flasher I would wonder if it has anything to do with it.

Anyway, the only hardware testing program for Nokia 770 that I know about at this moment is memtester: http://pyropus.ca/software/memtester/

A binary for Nokia 770 is here: http://ufo2000.xcomufo.com/files/memtester.gz
It should be run as root and you need to specify the size of block you want to allocate and test from the command line:
# ./memtester 20

This way is allocates and tests 20MB of memory. That's only about 1/3 of all the memory used in the device but I don't know about anything better yet. If you do, please let me know.

Idealy if someone could make a bootable memtester that could load from USB boot or the flash card it should be possible to scan more memory.