maemo.org - Talk - cloudsync.sh, replacement to DropBox and whatever you have

maemo.org - Talk (https://talk.maemo.org/index.php)

- Applications (https://talk.maemo.org/forumdisplay.php?f=41)

- - cloudsync.sh, replacement to DropBox and whatever you have (https://talk.maemo.org/showthread.php?t=89875)

cloudsync.sh, replacement to DropBox and whatever you have

Well, lately there were some threads about problems with Dropbox clients etc, and this makes me wonder...

I myself question the need to have an application and some external service just to sync my stuff. It seems a overkill, why do you need to do things the hard way when same (or better) functionality can be achieved with a simple shell script?

I have been syncing my data up&down with this kind of setup. This is easy to implement, portable, robust and beautiful.

1. You need to install few niceties first. This script uses rsync with a crontab trigger, so get those from rzr's Harmattan repository.
(I don't know about Frematle repositories but I belive cron and rsync are there too...)

if you don't have the repository there yet, put "deb http://repo.pub.meego.com/home:/rzr:...ttan_standard/ ./" into your sources list
apt-get update
apt-get install cron rsync

2. You need to create a pair of rsa keys and install the public key to the cloudservers users authorized_keys, just the standard stuff.
I recommend creating a dummy user just for the sync purposes...

3. Here is the cloudsync.sh script, put it to your "/root" for example:

Code:

#!/bin/sh



G_LOGTAG="cloudsync"

G_LOCKFILE="/var/lock/cloudsync.pid"

G_USERNAME="your_cloudserver_username_goes_here"

G_DNSNAME="dns.name.of.your.cloudserver"

G_LOCALNAME="server.local.ip.address"

G_OWNIP="ip.given.to.me"

G_UPLOAD_DIRS="/home/user/MyDocs/DCIM /home/user/MyDocs/.backups /home/user/MyDocs/AptBackup /home/user/MyDocs/meeTrainer_workouts /home/user/MyDocs/.CallRecorder /home/user/MyDocs/Pictures"



if [ -e "$G_LOCKFILE" ]; then

  logger -t "$G_LOGTAG" "cannot get lock $G_LOCKFILE, exiting"

  exit 1

fi



echo "$$" > "$G_LOCKFILE"



## get the wlan state to see if we're in home network

aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")

if [ "$?" -eq "0" ]; then

  G_CLOUDHOST=$G_LOCALNAME

  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"

else

  G_CLOUDHOST=$G_DNSNAME

  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"

fi



G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIRS $G_USERNAME@$G_CLOUDHOST:)

G_RET=$?



logger -t "$G_LOGTAG" "$G_SYNC"



if [ "$G_RET" -eq "0" ]; then

  logger -t "$G_LOGTAG" "Upload tranfer succeeded"

else

  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"

fi



G_SYNC=$(/usr/bin/rsync -rptv $G_USERNAME@$G_CLOUDHOST:upload/ /home/user/MyDocs/Downloads/)

G_RET=$?



logger -t "$G_LOGTAG" "$G_SYNC"



if [ "$G_RET" -eq "0" ]; then                 

  logger -t "$G_LOGTAG" "Download tranfer succeeded"

else

  logger -t "$G_LOGTAG" "Download transfer failed (error code $G_RET)"

fi



rm -f "$G_LOCKFILE"



return 0

4. Edit the variables in the script to your liking:

Code:

G_USERNAME="your_cloudserver_username_goes_here"

G_DNSNAME="dns.name.of.your.cloudserver"

G_LOCALNAME="server.local.ip.address"

G_OWNIP="ip.given.to.me"

5. put something like this in your crontab:

Code:

~ # 

~ # crontab -l



*/30  * * * * /root/cloudsync.sh > /dev/null 2>&1



~ #

DESCRIPTION:

So, for the people who bothered to read to the end of this posting :D

This script can be modified to sync any desired directories. This example syncs my:

DCIM
.backups
AptBackup
meeTrainer_workouts
.CallRecorder
Pictures

folders, but that's just my preferences. Anything goes.

The folder "upload" on the cloudserver is synced to "Downloads" on the users MyDocs, and same applies, anything goes.

The script detects if you are on home network and uses direct connection to the cloudserver if so. Of course that works this way for me because I have my servers on the same subnet as my WLAN. You might want to modify that setup based on your network configuration...

The crontab entry launches cloudsync every 30 minutes, and additionaly it can be launched at any given moment directly. The script creates a lockfile in /var/lock to prevent double invocation.

Security related thingies;

You will need the private rsa key in the roots .ssh/ on the device, which means that if you lose your device the key can be considered compromised. (but of course you have the root password to set to something else than "rootme", don't you?)
For this reason I recommend creating a separate user for backup purposes on the cloudserver. Additionally you might want to run that user on chroot jail, or forward the ssh connection to a virtual machine or separate HW.

Re: Arch Linux ARM on N900

Good job......juicy buddy

Re: cloudsync.sh, replacement to DropBox and whatever you have

This is cool. The one thing I like about dropbox, though is that it is 'event-based'. E.g, put a file there, and the kernel knows and does the sync then and there.

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Re: cloudsync.sh, replacement to DropBox and whatever you have

I had a such project to do this over webdav ...

Re: cloudsync.sh, replacement to DropBox and whatever you have

Quote:

Originally Posted by jackburton (Post 1338285)

Yes, that would easy to do really. The reason I don't have it is that I have just launched the sync up manually if I need something transferred immediately.

On the other hand, if there's something on the cloud end that needs to be synced immeiately to the phone it's a trickier situation. If that's needed then the phone would have to poll the remonte end at some shortish interval which is a sure battery killer.

Re: cloudsync.sh, replacement to DropBox and whatever you have

Quote:

Originally Posted by jackburton (Post 1338285)

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Well now as I tried it, I am thinking if it's worthwhile to push up an event about file change or if new file event is enough?

I guess normally dropbox reacts when you copy a file to the wtched directory, but what if a file is modified in the directory, should it be uploaded immediately?

Re: cloudsync.sh, replacement to DropBox and whatever you have

Last time I tried to implement my own sync solution with rsync I stumbled upon the inability of rsync to track deleted files in two way syncs. If you pass -delete it deletes missing files for both ends regardless if a file is new on one side. I resorted to using unison, which works like a charm four-way between my windows pc, ubuntu box, N900 and macbook. I suppose that compiling it for N9 should be trivial, if the debian binary doesn't work oob.

Re: cloudsync.sh, replacement to DropBox and whatever you have

Yes, it might get tricky if you want to handle more than just updates. That's the reason I just upsync stuff most of the time, and I have separate directory for things to download to the device from cloud.

Here is a quick&dirty solution for monitoring a directory (/home/user/mydocs/Upload/) and immediately syncing the content to cloud if a new file is dropped there:

Code:

#!/usr/bin/python

import os

import subprocess

import pyinotify



watchmanager = pyinotify.WatchManager()

watchmask =  pyinotify.IN_CREATE



class watchprocess(pyinotify.ProcessEvent):

   def process_IN_CREATE(self, event):

      subprocess.call("/root/upload2cloud.sh", shell=True)



inotifier = pyinotify.Notifier(watchmanager, watchprocess())

ret = watchmanager.add_watch('/home/user/MyDocs/Upload', watchmask, rec=True)



while True:

   try:

      inotifier.process_events()

      if inotifier.check_events():

         inotifier.read_events()

   except KeyboardInterrupt:

      inotifier.stop()

      break

You also need another script that is launched by the python inotify monitor script. I could have reused cloudsync.sh but I decided to strip it down a bit to "/root/upload2cloud.sh", like this:

Code:

G_LOGTAG="cloudsync"

G_LOCKFILE="/var/lock/update2cloud.pid"

G_USERNAME="your_cloudserver_username_goes_here"

G_DNSNAME="dns.name.of.your.cloudserver"

G_LOCALNAME="server.local.ip.address"

G_OWNIP="ip.given.to.me"

G_UPLOAD_DIR="/home/user/MyDocs/Upload/"



logger -t "$G_LOGTAG" "Triggered upload from filesystem"



if [ -e "$G_LOCKFILE" ]; then

  logger -t "$G_LOGTAG" "Cannot get lock $G_LOCKFILE, exiting"

  exit 1

fi



echo "$$" > "$G_LOCKFILE"



## get the wlan state to see if we're in home network

aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")

if [ "$?" -eq "0" ]; then

  G_CLOUDHOST=$G_LOCALNAME

  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"

else

  G_CLOUDHOST=$G_DNSNAME

  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"

fi



G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIR $G_USERNAME@$G_CLOUDHOST:download)

G_RET=$?



logger -t "$G_LOGTAG" "$G_SYNC"



if [ "$G_RET" -eq "0" ]; then

  logger -t "$G_LOGTAG" "Upload tranfer succeeded"

else

  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"

fi



rm -f "$G_LOCKFILE"



return 0

Re: cloudsync.sh, replacement to DropBox and whatever you have

While on topic, would anybody here want to try to evaluate sparkleshare for fremantle and harmattan?

Just a question:
When you update an existing file, do you upload/download a whole file or just the part of the file that changed?