<shameless pet project promotion>
I've added some ebook reading support to pyqtoreader. Originally it's a python-qt based image viewer with strong on-line integration, but it can 'view' any item we can shove into a graphicsview, which is pretty much everything (raster, vector, text, html, etc). It's explicitly written with large documents in mind, so it handles all the dynamic loading/parsing/unloading of pages and items.

The idea of extandable pyqtoreader is that people should be able to make their own library reading modules to support any format they like. Originally, this was meant for integrating webcomics or image feeds from different sources, but, as proven with the recent txt/html extension, can be used for pretty much everything.

At this point, pyqtoreader is *really* raw in terms I wouldn't recommend it for daily use and the API is still shifting, BUT, if you're interested, it's certainly a lot less effort to add a format there than write a reader from zero. And, to develop a module that works with pyqtoreader you don't even need scratchbox ! Just an install of python+qt on your linux or windows and you're good to go.

I'll try to post a video how pyqtoreader looks like these days as the version in extras-devel is pretty out of date.
</shameless pet project promotion>

Edit: Not try to dissuade the OP, just outlining some possibilities. Also, it might be possible to write an extension for an existing project and then, if not satisfied with speed/options, just replace the core. Still a lot easier than the all-from-scratch approach.