Have exectly the same problem with both my N810 and my N900. It seems to me that there are certain thresholds for the ambient light sensor... So it can be that whenever the light in your room is around such a threshold but changes only slightly (maybe because you're movin the device a bit), the screen brightness will over-adjust noticeably.

And, as twisted said, it doesn't react immediately. It takes some time. I remember one day when I used my N810 indoors on a reletively sunny, but windy day, it would change brightness each time a cloud covered the sun... But it would become darker when the sun re-appeared behind the cloud and a little brighter again a few moments later. Funny thing was that I myself didn't even notice the change in ambient light, I only noticed the screen becoming darker and brighter.

The algorithm may need some tweaking.