In practice, such imaging device doesn't exist. No-one will create it. Newer devices will use different connection methods, and suitable converters will not be made. The device size will not match.
In theory we have world peace and can travel in time.

In short, impossible. If someone were to try the upgrading starting from zero, acquiring all the needed knowledge, tools and parts, we would have take those into consideration when talking about the total cost of the mod. The price will be too high to justify the needed time and work for a private person...
Maybe Nokia have such a device already - At least all the money and man-hours spent in development of N900 would have to be counted in tens of millions in dollars or euro.

No. No software change can get more light into the 5MP sensor N900 has or make more sensitive - which would make better pictures. There are many more physical limits that software will not be able to solve.
The software to control the new imaging device would have to be changed - this would mean the whole pipe from controlling the camera to the final image produced would have to be reprogrammed for 12mpix images. Not going to happen due closed parts of that pipe.
Also, why stop at 12mpix? 50MP digital imaging devices exist, they are available as well, just use those when you start the upgrade process.
You can use your N900 as long as it will stay operational. Just take your pictures with something else, maybe the N8 if you want a 12MP camera. Ask for someone to write a bluetooth daemon for the N8 to transfer the images straight to your N900 after they are shot. That way you could keep fantasizing that the images were actually taken with your modded N900.