Thread: Script for full N9's partitions diferential backup with btrfs and rsync
View Single Post
traysh is offline traysh
2013-07-04 , 14:47
Posts: 324 | Thanked: 739 times | Joined on Jun 2009 @ São Paulo, Brazil
#1
I known not many people use btrfs as it is not totally stable yet, but I've been using it for a while and made a script to backup all my N9's partitions (MyDocs, rootfs and home). This is a real backup, not fake as the official backup tool.

The advantage of this method over dd is it will not use the partition size of disk space for each backup. In the first backup, all files in each partition will be copied (and not the unused space as in dd). In the next backups, only the differences will be copied. Yet, you will have multiple btrfs subvolume snapshots with all the files but not multiple copies of the files due to this filesystem CoW capability.

real example:
Code:
traysh@japung ~/downloads/maemo/maemo6/malf/backups $ ls -l
total 8
-rw-r--r-- 1 root root  803 Jul  3 19:51 0_README
-rwxr-xr-x 1 root root 2725 Jul  4 11:04 backup.sh
drwxr-xr-x 1 root root   46 Jun 20 20:38 home_20130630
drwxr-xr-x 1 root root   46 Jun 20 20:38 home_20130703
drwxr-xr-x 1 root root   46 Jun 20 20:38 home_20130704
drwxr-xr-x 1 root root   46 Jun 20 20:38 home_last
drwxr-xr-x 1 root root  444 Dez 31  1969 MyDocs_20130703
drwxr-xr-x 1 root root  444 Dez 31  1969 MyDocs_20130704
drwxr-xr-x 1 root root  444 Dez 31  1969 MyDocs_last
dr-xr-xr-x 1 root root  206 Dez 31  1969 rootfs_20130628_quebrado
dr-xr-xr-x 1 root root  260 Dez 31  1969 rootfs_20130630
drwxrwxr-x 1 root root  350 Dez 31  1969 rootfs_20130703
drwxrwxr-x 1 root root  350 Dez 31  1969 rootfs_20130704
drwxrwxr-x 1 root root  350 Dez 31  1969 rootfs_last
drwxrwxr-x 1 root root  206 Jun 28 21:34 rootfs_limpo

traysh@japung ~/downloads/maemo/maemo6/malf/backups $ ls -l rootfs_20130703/
total 100
drwxr-xr-x 1 root  root    578 Jul  1 02:19 bin
drwxr-xr-x 1 root  root      0 Mai 26  2012 boot
drwxr-xr-x 1 root  root      0 Mai 26  2012 cdrom
drwxr-xr-x 1 root  root    550 Dez 31  1969 dev
drwxr-xr-x 1 root  root   2664 Jul  3 18:41 etc
drwxr-xr-x 1 root  root      0 Mai 26  2012 floppy
drwxr-xr-x 1 root  root      0 Jun 29 21:17 home
drwxr-xr-x 1 root  root      0 Mai 26  2012 initrd
drwxr-xr-x 1 root  root   3222 Jul  1 02:07 lib
drwx------ 1 root  root      0 Mai 26  2012 lost+found
drwxr-xr-x 1 root  root      0 Mai 26  2012 media
drwxr-xr-x 1 root  root      0 Mai 26  2012 mnt
drwxr-xr-x 1 root  root    324 Jul  3 18:16 opt
-rw-r--r-- 1 29999 29999  1926 Jul  3 15:27 pm-rescue.1.log
-rw-r--r-- 1 29999 29999 43091 Jul  1 03:47 pm-rescue.2.log
-rw-r--r-- 1 29999 29999 43982 Jul  1 03:47 pm-rescue.3.log
drwxr-xr-x 1 root  root      0 Mai 26  2012 proc
drwxr-xr-x 1 root  root    442 Jul  3 18:41 root
drwxr-xr-x 1 root  root   1546 Jun 29 21:24 sbin
-rw-r--r-- 1 root  root    307 Jun 29 21:24 signature.gpg
-rw-r--r-- 1 root  root   2456 Jul  3 13:36 signature.x509
drwxr-xr-x 1 root  root      0 Mai 26  2012 srv
drwxr-xr-x 1 root  root      0 Mai 26  2012 sys
drwxr-xr-x 1 root  root      0 Jun 29 21:17 syspart
drwxrwxrwt 1 root  root      0 Mai 26  2012 tmp
drwxr-xr-x 1 root  root     94 Mai 26  2012 usr
drwxr-xr-x 1 root  root    130 Mai 26  2012 var

traysh@japung ~/downloads/maemo/maemo6/malf/backups $ ls -l rootfs_20130704/
total 100
drwxr-xr-x 1 root  root    578 Jul  1 02:19 bin
drwxr-xr-x 1 root  root      0 Mai 26  2012 boot
drwxr-xr-x 1 root  root      0 Mai 26  2012 cdrom
drwxr-xr-x 1 root  root    550 Dez 31  1969 dev
drwxr-xr-x 1 root  root   2664 Jul  4 09:53 etc
drwxr-xr-x 1 root  root      0 Mai 26  2012 floppy
drwxr-xr-x 1 root  root      0 Jun 29 21:17 home
drwxr-xr-x 1 root  root      0 Mai 26  2012 initrd
drwxr-xr-x 1 root  root   3222 Jul  1 02:07 lib
drwx------ 1 root  root      0 Mai 26  2012 lost+found
drwxr-xr-x 1 root  root      0 Mai 26  2012 media
drwxr-xr-x 1 root  root      0 Mai 26  2012 mnt
drwxr-xr-x 1 root  root    342 Jul  4 08:54 opt
-rw-r--r-- 1 29999 29999  1926 Jul  3 15:27 pm-rescue.1.log
-rw-r--r-- 1 29999 29999 43091 Jul  1 03:47 pm-rescue.2.log
-rw-r--r-- 1 29999 29999 43982 Jul  1 03:47 pm-rescue.3.log
drwxr-xr-x 1 root  root      0 Mai 26  2012 proc
drwxr-xr-x 1 root  root    442 Jul  4 09:56 root
drwxr-xr-x 1 root  root   1546 Jun 29 21:24 sbin
-rw-r--r-- 1 root  root    307 Jun 29 21:24 signature.gpg
-rw-r--r-- 1 root  root   2456 Jul  3 13:36 signature.x509
drwxr-xr-x 1 root  root      0 Mai 26  2012 srv
drwxr-xr-x 1 root  root      0 Mai 26  2012 sys
drwxr-xr-x 1 root  root      0 Jun 29 21:17 syspart
drwxrwxrwt 1 root  root      0 Mai 26  2012 tmp
drwxr-xr-x 1 root  root     94 Mai 26  2012 usr
drwxr-xr-x 1 root  root    130 Mai 26  2012 var

traysh@japung ~/downloads/maemo/maemo6/malf/backups $ sudo btrfs subvolume list /
ID 478 gen 73465 top level 5 path home/traysh/.wine
ID 479 gen 72610 top level 5 path home/traysh/.wine_snapshot_20130629
ID 480 gen 73423 top level 5 path home/traysh/.wine_office2010
ID 481 gen 71520 top level 5 path home/traysh/.wine_office2010_snapshot_20130629
ID 482 gen 73406 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/rootfs_last
ID 483 gen 72962 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/rootfs_20130630
ID 484 gen 72962 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/rootfs_20130628_quebrado
ID 485 gen 72973 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/rootfs_limpo
ID 486 gen 72978 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/rootfs_20130703
ID 487 gen 73407 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/home_last
ID 488 gen 72990 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/home_20130630
ID 490 gen 73005 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/home_20130703
ID 491 gen 73405 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/MyDocs_last
ID 492 gen 73009 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/MyDocs_20130703
ID 497 gen 73272 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/MyDocs_20130704
ID 498 gen 73274 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/rootfs_20130704
ID 499 gen 73275 top level 5 path home/traysh/downloads/maemo/maemo6/malf/backups/home_20130704
In this example, the snapshots rootfs_20130703 and rootfs_20130704 have almost the same content (because one was taken in a day and the other in the next day). While I can list and access the files as if there were 2 copies of them, there is in fact only 1 copy of them on the disk, except for the differences.

One additional advantage is that the backups are fast because only the differences are transferred between the phone and the computer.

Example of the script output:
Code:
traysh@japung ~/downloads/maemo/maemo6/malf/backups $ sudo ./backup.sh 
This script creates incremental backups of the N9's partitions as btrfs snapshots. If your filesystem is not btrfs, this will fail.
In case of any error, please verify if any partitions were left mounted.
Press ENTER to continue.

sending incremental file list

sent 34433 bytes  received 349 bytes  9937.71 bytes/sec
total size is 693318158  speedup is 19933.25
Create a snapshot of 'MyDocs_last' in './MyDocs_20130704_2'
sending incremental file list

sent 1903873 bytes  received 3151 bytes  81149.96 bytes/sec
total size is 1927065669  speedup is 1010.51
Create a snapshot of 'rootfs_last' in './rootfs_20130704_2'
sending incremental file list

sent 546014 bytes  received 593 bytes  121468.22 bytes/sec
total size is 1086719262  speedup is 1988.12
Create a snapshot of 'home_last' in './home_20130704_2'

Backups created:
1) MyDocs_20130704_2
2) rootfs_20130704_2
3) home_20130704_2
And the script code (also attached to this post) to create the backup:
Code:
#!/bin/bash
#

# Script for creating backups of all the N9's default partitions.

# Author: Danilo Luvizotto
# Date: 20130704
# License: GPLv2
# No warranty! I'm not responsible for any damage. Please read the license for more information on that.

set -e

## Please adjust these variables

MyDocs_PART=/dev/disk/by-uuid/4F43-B7CB                               #MyDocs partition. For me, /dev/sdb1 would also work.
rootfs_PART=/dev/disk/by-uuid/1707b697-1350-4dd1-a111-56945171c7bd    #rootfs partition. For me, /dev/sdb2 would also work.
home_PART=/dev/disk/by-uuid/e035cb56-b983-477d-93f1-7da10896fd98      #home partition. For me, /dev/sdb3 would also work.
MOUNTPOINT=/mnt

##End of the adjusting section

#The script should be run as root
if [[ "$(whoami)" != "root" ]]
then
    echo "ERROR: This script must be run as root!"
    echo ""
    exit 1
fi

hash btrfs >/dev/null 2>&1 || { echo >&2 "This script requires your filesystem was formatted as btrfs. Exiting."; exit 1; }
hash rsync >/dev/null 2>&1 || { echo >&2 "This script requires rsync. Exiting."; exit 1; }

echo "This script creates incremental backups of the N9's partitions as btrfs snapshots. If your filesystem is not btrfs, this will fail."
echo "In case of any error, please verify if any partitions were left mounted."
echo "Press ENTER to continue."
read

function find_unused_filename {
i=1
OUTPUT="_$(date '+%Y%m%d')"
while [[ -a MyDocs"$OUTPUT" || -a rootfs"$OUTPUT" || -a home"$OUTPUT" ]]
do
    i=$(( $i + 1 ))
    OUTPUT="_$(date '+%Y%m%d')_$i"
done
}

mkdir -p $MOUNTPOINT
if [[ -n "$(mount | grep $MOUNTPOINT)" ]]
then
    echo "ERROR: something is already mounted in $MOUNTPOINT!"
    echo "Unmount it and try again."
    echo ""
    exit 1
fi

# 0 - Find an unused filename
find_unused_filename

# 1 - Backup MyDocs
OBJ=MyDocs
mount $MyDocs_PART $MOUNTPOINT
if [[ ! -a "$OBJ"_last ]]
then
    btrfs subvolume create "$OBJ"_last
fi
rsync -av --delete $MOUNTPOINT/ "$OBJ"_last/
btrfs subvolume snapshot "$OBJ"_last $OBJ$OUTPUT
umount $MOUNTPOINT

# 2 - Backup rootfs
OBJ=rootfs
mount $rootfs_PART $MOUNTPOINT
if [[ ! -a "$OBJ"_last ]]
then
    btrfs subvolume create "$OBJ"_last
fi
rsync -av --delete $MOUNTPOINT/ "$OBJ"_last/
btrfs subvolume snapshot "$OBJ"_last $OBJ$OUTPUT
umount $MOUNTPOINT

# 3 - Backup home
OBJ=home
mount $home_PART $MOUNTPOINT
if [[ ! -a "$OBJ"_last ]]
then
    btrfs subvolume create "$OBJ"_last
fi
rsync -av --delete $MOUNTPOINT/ "$OBJ"_last/
btrfs subvolume snapshot "$OBJ"_last $OBJ$OUTPUT
umount $MOUNTPOINT

echo ""
echo "Backups created:"
echo "1) MyDocs$OUTPUT"
echo "2) rootfs$OUTPUT"
echo "3) home$OUTPUT"
The script code to restore the backup to the device:
Code:
#!/bin/bash
#

# Script for restoring backups of all the N9's default partitions.

# Author: Danilo Luvizotto
# Date: 20130705
# License: GPLv2
# No warranty! I'm not responsible for any damage. Please read the license for more information on that.

set -e

## Please adjust these variables

MyDocs_PART=/dev/disk/by-uuid/4F43-B7CB                               #MyDocs partition. For me, /dev/sdb1 would also work.
rootfs_PART=/dev/disk/by-uuid/1707b697-1350-4dd1-a111-56945171c7bd    #rootfs partition. For me, /dev/sdb2 would also work.
home_PART=/dev/disk/by-uuid/e035cb56-b983-477d-93f1-7da10896fd98      #home partition. For me, /dev/sdb3 would also work.
MOUNTPOINT=/mnt

##End of the adjusting section

#The script should be run as root
if [[ "$(whoami)" != "root" ]]
then
    echo "ERROR: This script must be run as root!"
    echo ""
    exit 1
fi

hash rsync >/dev/null 2>&1 || { echo >&2 "This script requires rsync. Exiting."; exit 1; }

echo "This restores backups of the N9's partitions to the device."
echo "In case of any error, please verify if any partitions were left mounted."
echo "Enter 'I have just made a backup!' and press ENTER to continue."
read -r USERINPUT

if [[ "$USERINPUT" != "I have just made a backup!" ]]
then
    echo "You entered '$USERINPUT', thus choosed not to restore the backup".
    exit 0
else
    echo ""
    echo "...and I hope you really did..."
    echo ""
fi

if [[ -z "$1" ]]
then
    echo "Usage: $0 <backup date>[_#]"
    exit 1
fi

INPUT=_"$1"
##Verify if the required backup files exist
if [[ ! -a MyDocs"$INPUT" ]]
then
    echo "The file MyDocs$INPUT does not exist."
    echo "Can't restore backup, exiting!"
    echo ""
    exit 1
fi

if [[ ! -a rootfs"$INPUT" ]]
then
    echo "The file rootfs$INPUT does not exist."
    echo "Can't restore backup, exiting!"
    echo ""
    exit 1
fi

if [[ ! -a home"$INPUT" ]]
then
    echo "The file home$INPUT does not exist."
    echo "Can't restore backup, exiting!"
    echo ""
    exit 1
fi

mkdir -p $MOUNTPOINT
if [[ -n "$(mount | grep $MOUNTPOINT)" ]]
then
    echo "ERROR: something is already mounted in $MOUNTPOINT!"
    echo "Unmount it and try again."
    echo ""
    exit 1
fi

# 1 - Restore MyDocs
OBJ=MyDocs
mount $MyDocs_PART $MOUNTPOINT
rsync -av --delete "$OBJ$INPUT"/ $MOUNTPOINT/
umount $MOUNTPOINT

# 2 - Backup rootfs
OBJ=rootfs
mount $rootfs_PART $MOUNTPOINT
rsync -av --delete "$OBJ$INPUT"/ $MOUNTPOINT/
umount $MOUNTPOINT

# 3 - Backup home
OBJ=home
mount $home_PART $MOUNTPOINT
rsync -av --delete "$OBJ$INPUT"/ $MOUNTPOINT/
umount $MOUNTPOINT

echo ""
echo "Backups restored:"
echo "1) MyDocs$INPUT"
echo "2) rootfs$INPUT"
echo "3) home$INPUT"
This backup method requires:
1) The filesystem in the computer in which the backup will be made must be btrfs. No ext2, ext3, ext4 or anything else.
2) Rsync must be installed in the system
3) It is required familiarity with exposing the phone's partitions to a computer using this method.

I know this is too advanced to most of this forum users, but I hope it will be useful to someone.
Attached Files
File Type: txt backup.sh.txt (2.6 KB, 90 views)
File Type: txt restore.sh.txt (2.6 KB, 84 views)
Last edited by traysh; 2013-07-05 at 20:51. Reason: Added the restore script
 

The Following 6 Users Say Thank You to traysh For This Useful Post: