maemo.org - Talk - cloudsync.sh, replacement to DropBox and whatever you have

maemo.org - Talk (https://talk.maemo.org/index.php)

- Applications (https://talk.maemo.org/forumdisplay.php?f=41)

- - cloudsync.sh, replacement to DropBox and whatever you have (https://talk.maemo.org/showthread.php?t=89875)

cloudsync.sh, replacement to DropBox and whatever you have

Well, lately there were some threads about problems with Dropbox clients etc, and this makes me wonder...

I myself question the need to have an application and some external service just to sync my stuff. It seems a overkill, why do you need to do things the hard way when same (or better) functionality can be achieved with a simple shell script?

I have been syncing my data up&down with this kind of setup. This is easy to implement, portable, robust and beautiful.

1. You need to install few niceties first. This script uses rsync with a crontab trigger, so get those from rzr's Harmattan repository.
(I don't know about Frematle repositories but I belive cron and rsync are there too...)

if you don't have the repository there yet, put "deb http://repo.pub.meego.com/home:/rzr:...ttan_standard/ ./" into your sources list
apt-get update
apt-get install cron rsync

2. You need to create a pair of rsa keys and install the public key to the cloudservers users authorized_keys, just the standard stuff.
I recommend creating a dummy user just for the sync purposes...

3. Here is the cloudsync.sh script, put it to your "/root" for example:

Code:

#!/bin/sh



G_LOGTAG="cloudsync"

G_LOCKFILE="/var/lock/cloudsync.pid"

G_USERNAME="your_cloudserver_username_goes_here"

G_DNSNAME="dns.name.of.your.cloudserver"

G_LOCALNAME="server.local.ip.address"

G_OWNIP="ip.given.to.me"

G_UPLOAD_DIRS="/home/user/MyDocs/DCIM /home/user/MyDocs/.backups /home/user/MyDocs/AptBackup /home/user/MyDocs/meeTrainer_workouts /home/user/MyDocs/.CallRecorder /home/user/MyDocs/Pictures"



if [ -e "$G_LOCKFILE" ]; then

  logger -t "$G_LOGTAG" "cannot get lock $G_LOCKFILE, exiting"

  exit 1

fi



echo "$$" > "$G_LOCKFILE"



## get the wlan state to see if we're in home network

aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")

if [ "$?" -eq "0" ]; then

  G_CLOUDHOST=$G_LOCALNAME

  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"

else

  G_CLOUDHOST=$G_DNSNAME

  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"

fi



G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIRS $G_USERNAME@$G_CLOUDHOST:)

G_RET=$?



logger -t "$G_LOGTAG" "$G_SYNC"



if [ "$G_RET" -eq "0" ]; then

  logger -t "$G_LOGTAG" "Upload tranfer succeeded"

else

  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"

fi



G_SYNC=$(/usr/bin/rsync -rptv $G_USERNAME@$G_CLOUDHOST:upload/ /home/user/MyDocs/Downloads/)

G_RET=$?



logger -t "$G_LOGTAG" "$G_SYNC"



if [ "$G_RET" -eq "0" ]; then                 

  logger -t "$G_LOGTAG" "Download tranfer succeeded"

else

  logger -t "$G_LOGTAG" "Download transfer failed (error code $G_RET)"

fi



rm -f "$G_LOCKFILE"



return 0

4. Edit the variables in the script to your liking:

Code:

G_USERNAME="your_cloudserver_username_goes_here"

G_DNSNAME="dns.name.of.your.cloudserver"

G_LOCALNAME="server.local.ip.address"

G_OWNIP="ip.given.to.me"

5. put something like this in your crontab:

Code:

~ # 

~ # crontab -l



*/30  * * * * /root/cloudsync.sh > /dev/null 2>&1



~ #

DESCRIPTION:

So, for the people who bothered to read to the end of this posting :D

This script can be modified to sync any desired directories. This example syncs my:

DCIM
.backups
AptBackup
meeTrainer_workouts
.CallRecorder
Pictures

folders, but that's just my preferences. Anything goes.

The folder "upload" on the cloudserver is synced to "Downloads" on the users MyDocs, and same applies, anything goes.

The script detects if you are on home network and uses direct connection to the cloudserver if so. Of course that works this way for me because I have my servers on the same subnet as my WLAN. You might want to modify that setup based on your network configuration...

The crontab entry launches cloudsync every 30 minutes, and additionaly it can be launched at any given moment directly. The script creates a lockfile in /var/lock to prevent double invocation.

Security related thingies;

You will need the private rsa key in the roots .ssh/ on the device, which means that if you lose your device the key can be considered compromised. (but of course you have the root password to set to something else than "rootme", don't you?)
For this reason I recommend creating a separate user for backup purposes on the cloudserver. Additionally you might want to run that user on chroot jail, or forward the ssh connection to a virtual machine or separate HW.

Re: Arch Linux ARM on N900

Good job......juicy buddy

Re: cloudsync.sh, replacement to DropBox and whatever you have

This is cool. The one thing I like about dropbox, though is that it is 'event-based'. E.g, put a file there, and the kernel knows and does the sync then and there.

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Re: cloudsync.sh, replacement to DropBox and whatever you have

I had a such project to do this over webdav ...

Re: cloudsync.sh, replacement to DropBox and whatever you have

Quote:

Originally Posted by jackburton (Post 1338285)

Yes, that would easy to do really. The reason I don't have it is that I have just launched the sync up manually if I need something transferred immediately.

On the other hand, if there's something on the cloud end that needs to be synced immeiately to the phone it's a trickier situation. If that's needed then the phone would have to poll the remonte end at some shortish interval which is a sure battery killer.

Re: cloudsync.sh, replacement to DropBox and whatever you have

Quote:

Originally Posted by jackburton (Post 1338285)

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Well now as I tried it, I am thinking if it's worthwhile to push up an event about file change or if new file event is enough?

I guess normally dropbox reacts when you copy a file to the wtched directory, but what if a file is modified in the directory, should it be uploaded immediately?

Re: cloudsync.sh, replacement to DropBox and whatever you have

Last time I tried to implement my own sync solution with rsync I stumbled upon the inability of rsync to track deleted files in two way syncs. If you pass -delete it deletes missing files for both ends regardless if a file is new on one side. I resorted to using unison, which works like a charm four-way between my windows pc, ubuntu box, N900 and macbook. I suppose that compiling it for N9 should be trivial, if the debian binary doesn't work oob.

Re: cloudsync.sh, replacement to DropBox and whatever you have

Yes, it might get tricky if you want to handle more than just updates. That's the reason I just upsync stuff most of the time, and I have separate directory for things to download to the device from cloud.

Here is a quick&dirty solution for monitoring a directory (/home/user/mydocs/Upload/) and immediately syncing the content to cloud if a new file is dropped there:

Code:

#!/usr/bin/python

import os

import subprocess

import pyinotify



watchmanager = pyinotify.WatchManager()

watchmask =  pyinotify.IN_CREATE



class watchprocess(pyinotify.ProcessEvent):

   def process_IN_CREATE(self, event):

      subprocess.call("/root/upload2cloud.sh", shell=True)



inotifier = pyinotify.Notifier(watchmanager, watchprocess())

ret = watchmanager.add_watch('/home/user/MyDocs/Upload', watchmask, rec=True)



while True:

   try:

      inotifier.process_events()

      if inotifier.check_events():

         inotifier.read_events()

   except KeyboardInterrupt:

      inotifier.stop()

      break

You also need another script that is launched by the python inotify monitor script. I could have reused cloudsync.sh but I decided to strip it down a bit to "/root/upload2cloud.sh", like this:

Code:

G_LOGTAG="cloudsync"

G_LOCKFILE="/var/lock/update2cloud.pid"

G_USERNAME="your_cloudserver_username_goes_here"

G_DNSNAME="dns.name.of.your.cloudserver"

G_LOCALNAME="server.local.ip.address"

G_OWNIP="ip.given.to.me"

G_UPLOAD_DIR="/home/user/MyDocs/Upload/"



logger -t "$G_LOGTAG" "Triggered upload from filesystem"



if [ -e "$G_LOCKFILE" ]; then

  logger -t "$G_LOGTAG" "Cannot get lock $G_LOCKFILE, exiting"

  exit 1

fi



echo "$$" > "$G_LOCKFILE"



## get the wlan state to see if we're in home network

aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")

if [ "$?" -eq "0" ]; then

  G_CLOUDHOST=$G_LOCALNAME

  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"

else

  G_CLOUDHOST=$G_DNSNAME

  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"

fi



G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIR $G_USERNAME@$G_CLOUDHOST:download)

G_RET=$?



logger -t "$G_LOGTAG" "$G_SYNC"



if [ "$G_RET" -eq "0" ]; then

  logger -t "$G_LOGTAG" "Upload tranfer succeeded"

else

  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"

fi



rm -f "$G_LOCKFILE"



return 0

Re: cloudsync.sh, replacement to DropBox and whatever you have

While on topic, would anybody here want to try to evaluate sparkleshare for fremantle and harmattan?

Just a question:
When you update an existing file, do you upload/download a whole file or just the part of the file that changed?

Re: cloudsync.sh, replacement to DropBox and whatever you have

Quote:

Originally Posted by knobtviker (Post 1339392)

Just a question:
When you update an existing file, do you upload/download a whole file or just the part of the file that changed?

Well, that's up to the implementation of rsync on the platform.

As far as I know (and now checking from wikipedia confirms this) the linux rsync divides all files into fixed size blocks and only transmits the differing blocks based on checksumming algorithm. (source: http://en.wikipedia.org/wiki/Rsync, see the section "Determining which parts of a file have changed")

A short test of my own; I transferred a binary file of about 9MB with cloudsync.sh, then hexedited it so that I changed just one byte somewhere about middle of the file, and resent it.

Code:

Apr 29 08:25:50 (2013) cloudsync: Triggered upload from filesystem

Apr 29 08:25:50 (2013) cloudsync: Using WLAN connection to server

Apr 29 08:26:02 (2013) cloudsync: sending incremental file list ./ rsync-muutostesti  sent 9176267 bytes  received 34 bytes  734104.08 bytes/sec total size is 9175040  speedup is 1.00

Apr 29 08:26:02 (2013) cloudsync: Upload tranfer succeeded

Apr 29 08:26:54 (2013) cloudsync: Triggered upload from filesystem

Apr 29 08:26:54 (2013) cloudsync: Using WLAN connection to server

Apr 29 08:27:01 (2013) cloudsync: sending incremental file list ./ rsync-muutostesti  sent 15267 bytes  received 18244 bytes  5155.54 bytes/sec total size is 9175040  speedup is 273.79

Apr 29 08:27:01 (2013) cloudsync: Upload tranfer succeeded

From this simple test you can see that on the second serving, only 15kB needed to be sent to transfer the changes to the file. (and most of this is transferring the MD5 hashes of the file blocks I guess, plus the changed block when it is detected...)

The "speedup is 273.79" means that the filesize/(sent_bytes+received_bytes) = 273.79, so only 2.7% of the total file size needed to be sent on the second round.

Re: cloudsync.sh, replacement to DropBox and whatever you have

First of all, thanks for the script. After using it for some time I wanted to have easy access to the log, as well as a way of knowing when a sync is done. This is how I do it:

1) Redirect all output/logging to a log file (.txt extension)
2) After removing the lock file, open the txt-file via xdg-open (launches the document viewer)

Code:

G_LOGFILE="/home/user/logs/cloudsync.txt"



echo "logger text and output.." >> $G_LOGFILE



# after removing the lock file

xdg-open /home/user/logs/cloudsync.txt

Re: cloudsync.sh, replacement to DropBox and whatever you have

Would this still work? I'm thinking about dropping the owncloud cli-tool and use this. Since currently owncloud is holding me back from using newer vers of nextcloud on the desktop.

x

Re: cloudsync.sh, replacement to DropBox and whatever you have

Quote:

Originally Posted by xman (Post 1551163)

Would this still work? I'm thinking about dropping the owncloud cli-tool and use this. Since currently owncloud is holding me back from using newer vers of nextcloud on the desktop

Yes, as cloudsync uses basic shell tools it works pretty much on anything you could want to try it on.

Re: cloudsync.sh, replacement to DropBox and whatever you have

Okay I might need your assistance once I get around to trying this. Since I'm hoping to have it sync in both ways. which I know you didn't want for your use.

Would be amazing if I could get this going reliably. Since I then can still sync my caldav & cardiac with the newer nextcloud and then all is well again.

x