Active Topics

 


Reply
Thread Tools
Posts: 692 | Thanked: 264 times | Joined on Dec 2009
#1
Hey all, thought I'd put together this guide to making easy full-system backups with rsync. You'll be backing up from your N900 to another Linux computer. My method doesn't require you to install an SSH server on your N900. In my case I'm backing up to my home file server running Xubuntu.

So a quick overview of what you need:

- An N900 (although this can be adapted for any Linux-based machine) with rootsh (duh), rsync and openssh (ssh client) installed. You can't get rsync through the Application Manager GUI, run "apt-get install rsync" as root.

- A Linux-based PC with sshd (SSH server) and rsync installed. You must be able to log in to this PC as a user that has sudo privileges.

Now I'm going to warn you a bit, rsync is one of the most potentially dangerous applications in Linux if not used carefully, right up there with dd and rm -rf. YOU CAN WIPE OUT TONS OF DATA IF YOU MAKE A MISTAKE SO BE CAREFUL. Pass an -n argument to rsync to do a "dry run" where no data is changed before doing any real tests.

Now make the following shellscript on your N900. Be sure to run it as root (I chown'd and chgrp'd mine to root, and set the permissions to 744) or the backup won't work properly.

In my example, I'm backing up to the PC at 192.168.254.100 as the user backupuser. I'm copying the root of my N900's filesystem (/) to /media/disk1/n900bak/ on the remote PC.

Code:
#! /bin/sh
stty -echo; ssh backupuser@192.168.254.100 "sudo -v"; stty echo
rsync -avn --delete \ #Change "-avn" to "-av" after testing
--rsync-path "sudo rsync" \
--exclude /proc/'*' \
--exclude /sys/'*' \
--exclude /syspart/'*' \
--exclude /dev/'*' \
--exclude /tmp/'*' \
--exclude /var/tmp/'*' \
--exclude /var/run/'*' \
--exclude /.debian/'*' \ #Excludes your debian chroot if you have one - recommended
--exclude /media/'*' \ #Excludes /media - remove this line to back up the microsd contents as well
--exclude /opt/'*' \ #Prevent backing up data twice due to bind mounting - this will be backed up under /home/opt
/ backupuser@192.168.254.100:/media/disk1/n900bak
Now run the script (as root of course) and make sure the backups are going where they're supposed to, for safety. You'll have to enter your login password, then your sudo password, then your login password again. The script is actually making a separate connection in the first line and running a remote sudo so that it won't ask for a sudo password while running the rsync command, which would cause the rsync job to hang - it has to run as root on the remote machine to preserve file permissions.

Once you're satisfied that it looks like it's working, change "-avn" to "-av" on the second line to do it for real. Your first backup could take a LONG time, mine took about 2 hours over a b/g wifi network, and I don't have a whole lot of data on my N900. The N900 got pretty hot too as it's processing flat out. So plug it in, open the keyboard, set it on it's kickstand (this will keep it cool and keep the battery from draining) and go watch a movie or something.

Once the first backup is complete (it should finish without errors), you can run the script again to bring your backup up to date. It will be MUCH faster, probably just a few minutes.

Now say your N900 fell in a sewer drain and you bought a new one. First bring it back up to the same firmware version you used to have, to make sure all component firmware versions are the same. Next, to restore your data, install rootsh+openssh+rsync, make a copy of your script, and reverse the arguments on the last line (be careful to include a trailing slash on the source directory this time). In my example that would be:

Code:
backupuser@192.168.254.100:/media/disk1/n900bak/ /
You're switching the source and destination just like with cp or mv. You might want to add the -n argument again to run a test before doing it for real.

Run the modified script as root and wait. When it's finished, reboot your phone and everything that was backed up should be restored. Be aware that with the --delete argument, any data on the N900 that isn't in the backup will be deleted.

Note: On some recent distros you may have to add a "defaults visiblepw" option to your /etc/sudoers (using visudo) on the server for this to work - this setting allows situations where passwords being entered could potentially be visible in the terminal, but this script is made to prevent this from happening anyways.
__________________
"Impossible is not in the Maemo vocabulary" - Caballero

Last edited by GameboyRMH; 2011-01-02 at 15:49. Reason: After testing, I no longer recommend compression. Also prevent redundant backup of /opt/, /var/run/, and add note about visiblepw
 

The Following 6 Users Say Thank You to GameboyRMH For This Useful Post:
Posts: 992 | Thanked: 738 times | Joined on Jun 2010 @ Low Earth Orbit
#2
... and go watch a movie or something ...
Just don't do it on the N900 while you're running the backup
 
Posts: 18 | Thanked: 2 times | Joined on Nov 2009
#3
Great tutorial. I use Grsync to do the same.

Does any one know of a good way to schedule a backup regularly?
 
Posts: 21 | Thanked: 8 times | Joined on Jun 2010 @ North America, west coast
#4
I second the use of rsync to do backups. A few notes:

- if you want to get rsync, you can do it by installing the program "grsync" (available in extras). However, if you don't know how to get rsync by command-line, you may not have enough expertise to create and run this shell script. But someone might want to create a GUI for the shell script and include the "apt-get install rsync" part or something.

- I do something similar, but I only back up my home directory, "/home/user/". I don't think we need to back everything up and then tell it to ignore the various system directories like "/proc/", etc. I figure I can always restore it to a known state.

- only backing up the home directory also obviates the need to connect and run as sudo, which makes it much safer.

- before I do a backup, I also back up my list of what packages I have installed. The shell script contains the command "dpkg --get-selections > /home/user/MyDocs/installed_pkgs". If your N900 gets wiped, you would restore using "dpkg --set-selections < /home/user/MyDocs/installed_pkgs"

- I also make use of the rsync options "-b --backup-dir=/path/to/backup/dir/", "--progress", "--partial" (you can replace "--progress --partial" with "-P") and of course as suggested by GameBoyRMH, "-v" so you can see what's happening rather than having rsync be silent for an hour leaving you wondering whether it's crashed.

- I use "rsync -e 'ssh'" to connect via SSH. In case you want to connect via rsync to a different port on your home Linux server (I think it's a very bad idea to use the default port of 22), you can say "rsync -e 'ssh -p 13579'", where you replace 13579 with whatever port you're using. Note that the entire 'ssh -p 13579' has to be quoted as the argument to "-e", because it is the ssh program itself that will be using port 13579.
I find myself unable to use shell variables to pass arguments containing spaces. So if I say
SSH_COMMAND="'ssh -p 13579'"
then if I say "rsync -e $SSH_COMMAND", it will say "I don't know what you mean by 'ssh", and then try to process "-p" and "13579'" separately.
Anyway, that's a minor point.

Thanks to rsync, I can make backups of my N900 while taking a walk in the park, and be reassured that the video I just took of my cute little baby will be backed up in case the N900 falls into the river before I get home.
 

The Following User Says Thank You to kwtm For This Useful Post:
Posts: 346 | Thanked: 271 times | Joined on Jan 2010
#5
Good tutorial but I think that restore data to a running system is a bad idea.
 
Posts: 692 | Thanked: 264 times | Joined on Dec 2009
#6
Originally Posted by Megaltariak View Post
Good tutorial but I think that restore data to a running system is a bad idea.
Nope it's totally safe in Linux, as long as you exclude the right directories.
__________________
"Impossible is not in the Maemo vocabulary" - Caballero
 
Posts: 31 | Thanked: 35 times | Joined on Jun 2010
#7
Originally Posted by GameboyRMH View Post
Code:
#! /bin/sh
stty -echo; ssh backupuser@192.168.254.100 "sudo -v"; stty echo
rsync -avzn --delete \ #Change "-avzn" to "-avz" after testing
--rsync-path "sudo rsync" \
--exclude /proc/'*' \
--exclude /sys/'*' \
--exclude /syspart/'*' \
--exclude /dev/'*' \
--exclude /tmp/'*' \
--exclude /var/tmp/'*' \
--exclude /.debian/'*' \ #Excludes your debian chroot if you have one - recommended
--exclude /media/'*' \ #Excludes /media - remove this line to back up the microsd contents as well
/ backupuser@192.168.254.100:/media/disk1/n900bak
Few comments:

Instead of all those excludes, I would use -x/--one-file-system switch and just list the mounted filesystems that need backing up (/ , /opt and /home/user/MyDocs probably).

I would investigate losing the "-z". One, ssh may already be doing some compression on the line already, and two, it is taxing on the CPU and if you're on battery that is not good.

Make sure the filesystem you are copying TO is capable of storing all the metadata and permissions. For example, backing up to VFAT is not ideal (not only that but VFAT truncates file times to even number of seconds so it'll always need resyncing).

Last edited by wotevah; 2010-08-19 at 22:31.
 

The Following 2 Users Say Thank You to wotevah For This Useful Post:
cfh11's Avatar
Posts: 1,062 | Thanked: 961 times | Joined on May 2010 @ Boston, MA
#8
Originally Posted by ugknite View Post
Great tutorial. I use Grsync to do the same.

Does any one know of a good way to schedule a backup regularly?
I would think if you have the script saved you could just schedule it to run through Alarmed
__________________
Want to browse streamlined versions of websites automatically when in 2g? Vote for this brainstorm.

Sick of your cell signal not reconnecting after coming out of a bad signal area? Vote for this bug.
 
Posts: 21 | Thanked: 8 times | Joined on Jun 2010 @ North America, west coast
#9
Originally Posted by wotevah View Post
Make sure the filesystem you are copying TO is capable of storing all the metadata and permissions. For example, backing up to VFAT is not ideal (not only that but VFAT truncates file times to even number of seconds so it'll always need resyncing).
Agree that you should be aware that the VFAT filesystem (for example, /MyDocs and /media/mmc1 which is the microSD card) does not store permissions.

But having said this, when backing up to VFAT all that happens is that a few errors get generated: rsync complains "I couldn't change the permissions", but the files themselves are backed up. If you replace the "-a" with "-rt", then the warnings go away. ("-a" is short for "-prtlgoD", if I recall correctly, so you're taking away the "-plgoD" part.)

The problem comes if you try to restore, and you need those permissions. Not such a big deal with backing up "/home/user/", but if you try to restore, say, "/bin" and the permissions are screwed up, then so are you. Another reason I just back up the home directory and let the system restore take care of the other directories.

Don't let the permissions/filesystems thing keep you from doing frequent backups to microSD. Better to have files backed up with wrong permissions than not to have a backup at all. You could do a rsync backup to microSD a few times a day, and at least have a safety net in case you delete some data file.
 

The Following User Says Thank You to kwtm For This Useful Post:
Posts: 16 | Thanked: 13 times | Joined on Jan 2009 @ Sofia, Bulgaria
#10
One warning and one hint from me for using rsync for backup/restore:

- If you have some packages which optify some parts from the rootfs (for example in my case I had pymaemo-optify) be sure to install this package in addition of the essential rootsh and rsync packages on the freshly flashed device ready to restore. Otherwise you will end up with full rootfs with 0 bytes free. You need to do this because otherwise you'll restore the contents of the dirs in rootfs BEFORE you had those dirs optified (on the next reboot with 'mount -bind') and you'll have real copies of all those files in rootfs hidden by binded mounts above them.

- And one hint: for quicker and more stable (not over wifi) restore procedure you can make one ext3 partition on some spare micro sd card and copy your backup there to do the restore with rsync.
 
Reply

Tags
backup, howto, rsync


 
Forum Jump


All times are GMT. The time now is 10:39.