Yeah, you're right.
But in the same time, if we take a look at desktop PC and laptops, that had 20/30 years to mature, you can see this trend:
They are not a lot modular ! At least not as much as some here want the phone to be.

If I take my desktop :
* network, GPU, sound.... are all integrated in the motherboard. So I can NOT change them. What I can do however is to ADD another one thanks to the fact the desktop case is 5 times bigger than was it inside.
* For the CPU, yes I could change it, but will never do: the socket LGA775 is not used anymore (superseeded by at least 3 or 4 newer ones), so I can not find a recent CPU to fit here that could provide me more value.
* RAM : yes I added some RAM, it is nice ! But I need it to follow the speed that is availlable to the CPU and motherboard : the speed we had 5 years ago, not what is in the market now...
* storage/peripherals: I have now 3 drives (HDD+SSD) + a DVD. Here it is also the fact that the case is 5 times bigger than needed that allows that. I also have several USB external things that are always plugged (keyboard+mouse to start, bluetooth sometimes)

If I now take the exemple of my laptop:
* I can change memory, but it was already sold with the max possible...
* I can replace the hard drive, but can't have 2 drives (SDD+HDD for example...)
* I can add external things through USB, but that makes it a lot less portable if it needs to be always here (think of a second hard drive, of can you carry that !)
* And that's all ! I can not even replace the battery to a bigger one as they don't exist (I can only replace it by the same if it fails). I can replace the motherboard as the case is built around and no other one will fit. The processor may be changed, but it is not made to be easy with the cooling system. I can't replace the webcam, the screen size, ...

So, how do you think we can easily have much modularity in a phone that have far higher size constraints, as it is not really good in mature products like laptops. Desktops are better in that regards, but there are still limits.

The puzzlephone may have the balance here right. We'll see how it ends up.