It can be upgraded, but you have to install OS2007HE (The 'Hacker Edition' that brings the OS2007 from the N800 to the 770), which is not an easy task it seems and poses the risk of 'bricking' your 770. The current OS2007HE will add Flash 7. If you're patient, there might be a next OS2007HE, which could support Flash 9. I need to try it myself, as I'm getting tired of all the sites it can't browse due to only having Flash 6 as well.

Macromedia Flash is in no way open source. The Flash player in the 770 is licensed from Macromedia and paid for by Nokia. However, there is a reverse-engineering effort called Gnash that is making progress in running Flash content, that is open source.