You probably have something either mechanically wrong in your IHF speaker in the phone, or some dust and dirt next to the component.

You could try going to a Nokia Care and find out if they could clean it for you. Mind you though, that dirt inside the device isn't covered by the warranty in most cases.