Future maemo devices are in a very unique position. Currently the N900 has a high resolution camera, a fast processor, plenty of storage, and a very open environment. With all of these characteristics, this is a perfect platform on which to experiment with Computational Photography.

Some of the simpler effects may come in the form of photoshop-like filters. These can be simple filters like watermarks, motion blur, edge-detection, hue/contrast, etc, etc.

However, this is just the tip of the iceberg.... Imagine face-detection, augmented-reality using fiduciary markers, object isolation and recognition, in-photo optical character recognition, automatic HDRI splicing, 2D to 3D imaging, automatic panorama composites, image finger-printing, etc.

And then consider using these in combination with one another. Not only can the improve the quality of images taken, but provide shots that would be impossible with a traditional digi-cam alone. The best part? It's all software.

It would be nice to have a framework that could accept processing plug-ins, be built apon, and be used in apps as a dynamic library or from the command line. This implementation would make it easy to use from an application developer perspective, and very easy to contribute to (plugin-submission rather than patch).

If you have examples of CP in action, post them below! It's a growing field and I'd be interested to hear your input!

PS. Here's something that could use such a framework, though in this case, not for photography:
http://www.youtube.com/watch?v=zyWVH6jkDHg

}:^)~

Here's another neat idea, guessing device-orientation based on outdoor-shadow direction, the accelerometer, and time of day. If could be a very useful function in the absence of a compass. Perhaps not real-time, but still very useful in conjunction with GPS!

Additionally, a library like this may be useful for robots that use the N900 at its core, to process visual information. Of course, the plugins would have to be fast enough to be used in real-time!

}:^)~

Ok, last one for now!

Here's a VERY useful use: photo-copy.

Imagine taking a snapshot of a page, and having the library calculate the correct orientation, and compensating for page colour and lighting across the page, and then possibly running it through an ocr. Despite the page being bent (think thick book), or the lighting being different across the page (also thick book), or a slightly skewed angle (quick snapshot), the pages true orientation and colour could be found, as though it was a single, flat, colour-photocopied page.

This would be tremendously useful for snapping notes, business cards, reference pages, magazine pages, handouts, brochures, etc. Combine them with a tool like xournal and an upload engine, and you have TRUE magic.

Of course, something like this could be contained in an app, but it would be better to contain the functionality in a library, and then build the app using the library, so that many other developers can use the CP framework in weird, wild ways.

}:^)~

As I said in the other thread, at the summit we heard that it is planned to open the camera app - we should certainly either then, of in the meantime try to sort out a plug-in architecture so that photos can be post-processed in various ways.

In fact the camera app is probably easy enough to replace, other than the jpeg encoding on the DSP, which might take some work, so we could always get cracking with it now in anticipation.

This is something we are working on. Full GStreamer support in N900 allows you to have your own processing elements in Camera pipelines (there are three of them, viewfinder, photography and video recording). Post-processing with different effects for Image Viewer application is planned for Maemo 6, see Maemo Image Editor project on maemo.gitorious.org for our current code under development.

I also have been discussing the very same topic with people from Stanford University (Frankencamera) and others in community (Elphel, for example), and internally in Nokia. We are pretty much open to contributions on the topic.

One thing you would really need to understan is how all additional live processing is related to latencies and memory bandwidth that it would introduce. I'm not even touching the CPU load, it is secondary issue. Any additional frame copying while in viewfinder or recording video pipelines cuts down effective frames per second rate and contributes negatively to smoothness of the UX. Any additions need to be verified closely for such side effects.

I would like to be able to photograph a vinyl record (remember them?), and have the song come out of the N900's speakers.

This is very complex but achieveable, as the Digital Needle project showed. As you can hear from their sound files, it was a proof of concept only.

Regards,
Roger

To throw some oil in the fire: It will be probably possible one to use photoshop.com on N900 when the accelerated Flash 10.1 comes around Its more like the Lightroom, but nevertheless...

yesterday i tried a little piece of software (demo) which would be awesome on a N900:
http://people.cs.ubc.ca/~mbrown/auto...utostitch.html
I can allready see myself creating 4-desktop wallpapers directly on my N900...

thanks for moving interest on this subject, Capt

Maybe this thread needs to be merged with the HDR discussion thread?

Mike C

Great! I'm glad to see enthusiasm for this project. I believe not only can it be a big plus for Maemo and the N900, but a useful project for open source in general. I'm sure if the framework is robust enough, we can solicit the contributions of academia, or tinkerers world-wide.

Ok, enough of that. Any ideas? This is all new-territory for me, so forgive me if my suggestions sound silly.

Input plugins - For the N900, images can be taken from the camera, but having a generic input plugin makes this useful for other devices, and/or getting images from different sources (fs, internet, etc). Consider that the appropriate input plugin could handle vector imagery as well!

Output plugins - This is pretty straight forward. Having an extensible output system, the output could be xml commands for an autonomous bot. The output should not be limited to a graphic format. Also output to the internet becomes possible.

Temporary filesystem storage - For processing large amounts of data, or storing data for later use. It would be nice if this was an automatic extension of the memory allocation and access system. The benefit of not doing this is swap is a) a much larger scratch pad, b) the potential to optimize reads and writes based upon operations, c) one less thing to think of as the developer (and a pretty big one!).

Node-based architecture - This is a BIG feature, and IMO a must-have. We all know that pipes are useful on the command line. It would be nice to provide an in-code mechanism to easily chain plugins together. PLEASE Check out www.filterforge.com to get a good idea of the possibilities!

Resolution Independence - This seems a bit odd at first, but being able to automatically interpolate lower res images to make them size compatible for comparisons with higher res images would be useful for comparisons.

Language independence - This is a bit of an odd one, but it would be VERY nice to be able to program these plugins in any language. It would be doubly nice to have the library itself easily portable to any number of languages as well. I'm not sure how easy/possible this would be, so feel free to add or strike me down as a raving heretic.

CP (computational photograpy) functions - This is the bread and butter of the application. These CP functions take 0-n resolution independent images, 0-n data inputs, and compute 1-n outputs.

XML rule sheet - This file stores the node structure, which the library can use to load plugins, and determines how to pass information around. This is useful for using the library in a non-interactive, or code-driven way. This would also be necessary for the command-line app.

Plugin controller - It would be nice to be able to break the execution of a plugin after a completed stage, so that the app can prevent runaway situations for complex computations that would take hours on an N900, but would be well suited for a desktop. Perhaps the plugin architecture can allow for this?

Some extra thoughts:

- I think there should be designation for real-time vs slow plugins. Perhaps a different interface for each.
- I think that this should be standards based and avoid re-inventing the wheel where necessary
- I think that it should be an incredibly modular system to allow for disjoined development and easy integration into projects.
- We need a common interface for each category of plugin.

Anybody care to add anything?
Have any ideas for a name?
Any and all input is much appreciated!

}:^)~