Submarine patents are irrelevant. I.e. potentially anything can be patented by some unknown patents, and therefore by this definition everything is proprietary altogether? It's not how the definition goes however. If there are no known patents that cover some open source technology, it's free and non proprietary, until proven otherwise.

VP3 was originally encumbered indeed, as well as VP8 by the way. However it's long since it was opened by On2 themselves and became royalty free:

https://en.wikipedia.org/wiki/VP3

So Theora based on VP3 is by all means a FOSS and non proprietary codec, as well as VP8 which was opened by Google after they bought On2:

https://en.wikipedia.org/wiki/VP8