I think you are wrong. Only amount of swap data in use, i.e. what conky (cat /proc/swaps) shows, will be moved. So bigger swap parts will help to minimze swap refresh actions, as time until fragmentation starts, will increase. (did not test yet, but am pretty sure)