Mmm. Just trying to narrow this down...
There are 3 hardware parts - N900, cable and PC. Assuming you've tried a few cables, that's not the problem. Both Windows and 'n*x OSs say there's a problem, so assuming the PC end is good too. What happens when you use storage mode on a 'n*x PC? To confirm it's a problem at the N900 end, what happens when using HEN and a usb memory stick (screen msgs, kernel, whatever feedback you notice)?

Once the hardware related to the problem has been identified, the next job is to identify whether it's a hardware or software problem! One the face of it, it seems like a software problem because the connection seems live but a bit disfunctional, but we are talking about the N900 and its USB port! Fingers crossed...

Edit: just seen this in your post above:
- do you mean data lines on the PCB, or within your cable? I know it's risky from a damage point of view but those prongs that can pull the port off the PCB actually hold the plug in a better position - have you tried a cable which still has those prongs?

He mean, that everything seems like no data lines, despite fact, that when he probed them (from behind port) to testpads under battery, they've worked perfectly.

Can-opening a usb male plug or female port, connecting to teared-open microUSB cable (thus creating easy tet solution for socket itself) seems like good idea. If it isn't internal damage of usb port, then I have no clue wtf is going on with Your device.

/Estel

/Edit

Or even without tearing down usb plug - just cut open a microusb cable, then plug it into N900, and probe loose cables with testpads under battery.

Try a different USB cable. The same thing happened to me and it turned out the cable was faulty.

I have separate cables I use at home and work, and I have a few here at home that I've tried. They all work with my "test" N900 fine, so I'm happy to rule out the cables and PCs. It's either hardware on my N900 or software on my N900. I doubt the chances of corrupting not only NOLO's USB mode, but bootmenu's modules, AND the actual phone's proper modules as well.

...which just leave the hardware. I wonder if it could be a joint that has continuity but is starting to fail, making the resistance too high or the circuit too flaky for USB signalling to pass?

or some hardware chip died.

Apologies for the delay - it's been a bit hectic lately....and on top of all this my Raspberry Pi's started having HDMI issues. I just seem to be having all the bad luck lately! Anyway, I cracked open a dodgy old hub and tested the N900's USB port from the pads to the hub's connections. All a-ok.

FWIW, I get about 1.5Ω to 2Ω resistance on the data lines, ~2Ω on 5v (J2000 pin 3), and 0.5Ω on GND (J2000 pin 1), as detailed here.

All my cables have those prongs - I never removed any.

Interestingly, I recently had a 3-week trip to Japan, and I took my N900 for SMSes, and borrowed my brother's old HTC Desire so I could have a decent Google Maps experience using a data-only SIM.

Anyway, because of this issue, I had to charge my N900 using a power-only USB cable, and things were fine whilst I was away.

When I returned however, I needed to download my photos, some of which were on my N900. I figured I'd try plugging it in via USB as a random act of desperation...and my PC picked it up. It works normally again!

Tried it at work, and same...it picks it up and I can sync my calendar again

I'm a happy bunny. I've no idea how whatever was broken righted itself other than either the battery pack charger I was using managed to "unlatch" whatever minor internal fault was disrupting the USB data lines with higher current (the charger is a 2A one), or the tighter connector somehow improved the contacts on the USB plug? The always had continuity...but maybe the contacts weren't good enough for data? Who knows. All I care is that it works again

I point my finger at Windows XP. Had the very same issues myself.
Same cable works fine on my Win 7 laptop (and indeed my Ubuntu laptop), but Windows XP fails to recognise the device, but it charges OK.

So I go into Device Manager and delete the failed N900 connections, reboot and everything works fine after that...........until it does it again a few weeks later.

Interestingly enough, I had to go through the same shenanigans with my N95-8GB before I got my N900.