Interesting stuff...

Possibly preloaded applications -- iirc desktop will preload some programs like Clock when free memory is higher than a threshold defined in .desktop file . How is it calculated I don't remember but seems that just adding a new, empty swap area causes it to increase.

Note that phone will appear faster after swapoff/on cycle even without any swap backend related slowness present, since quite an "acceptable" number of open applications fit in RAM, specially after desktop unloads all the preloaded ones.
Of course if this issue is noticeable it should quickly manifest when trying to start extra ones.

I hope I get to test that next time I try to do an idle battery life test.

Btw, if writes are slowed down, wouldn't a heavy swapiness actually help -- as it would nice to keep as much free RAM as possible to minimize the amount of frames to page out when you're interacting with the device?