I have a small problem:
For years, I have been fairly lax about backing things up. As such, I have many years worth of old files scattered across more than a dozen harddisks. Now, most of these files exist in more than one place, but not all exist in the latest incarnation.

I'm looking to get a 1.5TB harddisk and back all of these files up on it. I would like some sort of program that would allow me to keep a single copy of each of these files(No need to have three copies of a video, for instance), as well as a database/symbolic link tree that allows me to find any particular folder of files if I need.
I also need it to be able to handle multiple versions of a file, and keep one copy of each version.
Even better would be a tool that would backup based on CRCs & filesizes - I have a number of files that I've renamed at one point; no point in keeping multiple copies if the contents are the same.


The problem is, I'm not entirely sure what tools I have available; I hear rsync is incredibly useful, but I'm not sure how I would do something like what I want.
OS wise, I'm using Debian Linux(Squeeze+Unstable), so anything that's Linux based should work.


Any thoughts or ideas, guys?

Yes, sounds like rsync over SSH is the way to go.

I use it to run my nightly backups - everything from backing up my personal data from my server, backing up my websites and even my N900 too - all copied nice and safe to an external USB disk.

For the version history etc you'll need to do some scripting but there's loads of info around - for example http://www.mikerubel.org/computers/rsync_snapshots/

rsync is good for copying a bunch of files from one directory to another, but as a backup mechanism it is extremely rudimentary, not much above the level of cp

I don't have anything to recommend, but what I would do in similar situations is browse through sourceforge and freshmeat