Why VM wouldn't know about physical memory layout and architecture? There is not reason VM couldn't be clever like that. In the (in)famous "Java will be faster and C++" tells also that VM can also know how many and what kind of processors (how many registers, what is the size of L2 cache) the platform where it is running has. It can monitor page fault rate especially in multi-core platforms or if a program is interactive and sometimes is not doing anything.

On the other hand, fully compiled C++ code cannot be optimized so well because it has to fit to some minimum common architecture or have multiple compilations. Before also in this thread was an example why C++ compiler cannot do register allocations in some cases where Java JIT can. Same goes if there would be different number of registers on platforms the code is compiled for. Java VM can know how many registers and JIT the bytecode to use all available registers efficiently.

Of course C++ can be also JITed and interpreted, but then why use C++ after that when there is Java which seem to be more productive than C++, easier to maintain, easier to read, easier to fix and find bugs?

It is probably true, not yet for example Dalvik has all these fancy optimizations, but if it lives long enough it will, especially now when mobile platforms have started to get multiple CPU cores.