This is definitely a hardware issue which is not caused by any extra software. You can check whether your N900 suffers from this issue or not by entering this command in Terminal:

You should really get "open" when nothing is covering the sensor. I personally always get "closed" when I am outside, which means my N900s proximity sensor is probably defective.

If you also experience this behaviour, then please vote for bug #7772 on bugzilla (https://bugs.maemo.org/show_bug.cgi?id=7772). Please remember not to use bugzilla as a forum - adding a vote is completely sufficient to show that you are also affected by the bug.