n810 camera as mouse?
 

I've been looking lately at attempted to use the n810 camera as a mouse.
Here's the basic idea.

The n810 camera, while low quality, should work fine to notice and track gross finger movements across the screen.

This would be handy for scrolling motions such as in the browser/microB.

Here's how I would expect it would work. A background process would watch the camera and notice wipe like motions (up / down / left /right) and translate them into events, either keyboard or touch screen equivalents.

Has anyone seen a project like this in the past or have suggestions for how to proceed?

Thanks,
Jeff

Re: n810 camera as mouse?
 

There was some talk about this (ages ago) for the N800; the idea was to twist the camera to control scrolling. I don't think anyone's actually worked on it; probably very CPU intensive.

Re: n810 camera as mouse?
 

They got the camera working in Flash in Chinook (the hack broke in Diablo); this meant that you could play those silly flash games where you can break bubbles by waving your hands around...

But maybe that's not what you mean? ;)

Re: n810 camera as mouse?
 

You mean this?

http://www.infosyncworld.com/news/n/9165.html

I already asked, but no one seemed interested.

Its possible, and I think it would be cool....

Actually, I did something like this on Flash, AutoIT, and Windows. Really basic stuff, but it worked.

Re: n810 camera as mouse?
 

Should be easy enough to do, record data at full rate and perform a 2D correlation on each frame to work out which way it's moved (on average).

Try using the example code for the camera, or the maemo-barcode code as a starting point. I'm happy to advise/contribute to see if it can be made to work.

Re: n810 camera as mouse?
 

Yeah, but all the things you'd want to use it for are already hogging CPU.

My advice: Window a small chunk from the center of the previous frame, and do a correlation of that on the whole window, and keep resolution small, and it might work. But a basic approach is gonna be glitchy as imaginable when the CPU binds and you get no frames for a half second. Trying to accurately return results when there was no jump, and handle real jumps like that gracefully, is not gonna be easy.

If I were just looking for an arbitrary way of conveniently scrolling down webpages, I'd go with overall image brightness. Scroll up when there's lots of light, down when there's no light, and somehow (lux sensor?) detect when the user's thumb is not there and don't scroll at all. Modulate light with your thumb like one of those optical "theremins" on eBay.

Re: n810 camera as mouse?
 

What? Samsung Instinct? Don't understand. :confused: :confused:

Re: n810 camera as mouse?
 

I think cooler would be to use the front-facing (low resolution) camera as a hand gesture touchless control - like this demo
http://www.youtube.com/watch?v=7DikbVmYLR0
(check out the part at 1:16 mins where you control music playback with hand gestures)

And if this is true, then it works on a S60 platform, it should work on the more powerful Nokia IT at least.

Re: n810 camera as mouse?
 

Great replies. I can handle the image processing side pretty well, but haven't played around with Maemo/Linux event processing. Any thoughts on how to inject or emulate up or down dpad presses or emulate a upward or downward swipe on the screen?

Re: n810 camera as mouse?
 

I'd just use xte, for a POC at least. D-pad is just the arrow keys, so would be equivalent to d-pad taps. You could also implement swipes and long-presses of the keys, or page up/down...

Re: n810 camera as mouse?
 

You have to scroll through a bit to get to the part where it's using the camera as input.

I've seen a similar thing a while back about using stuff, CamSpace, but a group called Cam-Trax.
The possible uses for this seem very impressive. Hopefully we'll get an open source version :)

Re: n810 camera as mouse?
 

Perhaps, but then you simply return no mouse movement so it shouldn't be too strange.

DSP has hw acceleration for motion estimation, not perfect image correlation but would probably do the job well enough for this task.

Re: n810 camera as mouse?
 

JeffLuszcz I am quite interested in this application. I have been wanting to build it since I got my n810 (about 3 months ago). I am interested in a little bit more however. I would like to try and fully emulate an accelerometer with the cam.

There is a company that specializes in this (among other things) called gesturetek. see : http://www.gesturetekmobile.com/ Its a Canadian company and I only found out about it because I spoke to their CTO a few months ago about their multi-touch division.

I am a multi-touch researcher in general so I am no stranger to processing video as input, but I am a bit new to the maemo platform.
If you want to work together and set up a project on maemo garage I would be happy to work on this with you.

I have also read up on a guy who was trying to port openCV to maemo, but was mostly successful. ( I think)
see : http://www.nabble.com/Problem-using-...html#a16326280
and
http://andrew.daviel.org/n810-blog.html
But I think that openCV might be a bit to much bloat for this project anyway.

Re: n810 camera as mouse?
 

This will not be that CPU intensive. It does not need perfect image capture and recognition. It won't be as accurate as a accelerometer, but good enough for Up Down Left Right. I would like to see this implemented (since May in fact), as well as the motion gesture recognition.

Re: n810 camera as mouse?
 

I'm waiting to get my n810 back from warranty service since the camera stopped working!

Once I get it back (and reflash) I hope to start work.

Re: n810 camera as mouse?
 

I have searched about the thematic's on maemo.garage and found these
projects:

https://garage.maemo.org/projects/motion/
https://garage.maemo.org/projects/motiondetector/

I think these project's sources might be very useful for you ;)

greeez

Re: n810 camera as mouse?
 

I just thought i'd bump this thread and express my (end-user) interest in some kind of camspace- or freetrack-type webcam motion tracking. now that i have a keyboard for my n800, i often prop the n800 on a table, and it would be great to be able to control cursor movements without touching the device (or a mouse or the stylus).

freetrack is designed to read only IR. they say to cover the front of the webcam with 2-3 layers of film negatives so that only IR gets through. i think this reduces processor load. EDIT: but then, of course, you have to use a flashlight or something to move the cursor, which--although very cool--might not actually be any more convenient than using a stylus or a mouse....

Re: n810 camera as mouse?
 

really interesting thread: In theory a really rudimental motion detector system could be achived using a REALLY low resolution camera image (even with a 3x3 pixel wide image should be possible to have some informations to use) should be able to recognize when something move from one part of the 'grid' to another, the direction and the speed. Higher the resolution and higher should became the precision (but I suppose will became harder to manage all the extra informations): emulate the shake rock and roll should not be too hard: emulate an acceletometer, well, that sound harder :)

Re: n810 camera as mouse?
 

Wanted to revive this topic as I think it would be a really usable way to control the n8x0. Perhaps a new thread should be started because the way I am picturing it it would not function like a mouse.

Maybe it is just me, but I see this being useful to control the browser (for example, flick your finger/thumb vertically up or down across the camera to send a "page down" or "page up" command, or flick your finger/thumb horizontally left or right across the camera to switch between windows or tabs or workspaces even). I could see it controlling games very easily as well (example, draw a dot on your thumb and hold it in front of the camera, if needed put a cheap piece of clear plastic to hold your thumb on). It really has lots of potential if it can just pair keybinds with what the camera senses. If it can be done on s60, can't it be done on maemo? The things that can be done on a PC with openCV 1 2 , freetrack, camspace, or opengazer are impressive, but simple webcam tracking would be much easier I should hope. Make the thresholds less sensitive to save on processing power. If I am completely off base let me know, but seeing it working on a s60 phone makes me think it would be feasible.

Or maybe I have just had too much coffee today and am off in Pink Floyd land.

Re: n810 camera as mouse?
 

No, I still like the idea of this. I'll try to start something over the weekend (in MATLAB) to do some testing.

Re: n810 camera as mouse?
 

let me know if there is any way I can help.