The "correct" way to do it is definitely to package them as separate .deb files, and add them to the dependencies. The alternative would be to restrict your package to building for armel (as you'll only be able to include libraries for one platform), add the libraries to the .tar.gz file, and install them via the debian/control file. You'll have to double check on licensing as well though (as you're then distributing without source).