I think this is because freepascal compiled binaries are linked statically to the runtime. There is no such thing as separate libpascal.
If you try to compile programs with gcc statically then you will find out that resulting binaries are 5-10 times more fat than pascal compiled ones.
Anyway, this doesn't simplify your task.
I would suggest to use smartlinking (-XX flag, if libs are also compiled with smartlinking) and then UPX. You'll have binaries which are comparable, or even smaller than if you compile the same with gcc.