cloudsync.sh, replacement to DropBox and whatever you have - maemo.org

Active Topics

Firefox with Leste (10)
to Maemo 7 / Leste by teroyk - 3 days, 6 hrs ago
more...

Page 1 of 2

Next >

Thread Tools

juiceme	2013-04-23 , 08:49
Community Council \| Posts: 4,920 \| Thanked: 12,867 times \| Joined on May 2012 @ Southerrn Finland	#1

Well, lately there were some threads about problems with Dropbox clients etc, and this makes me wonder...

I myself question the need to have an application and some external service just to sync my stuff. It seems a overkill, why do you need to do things the hard way when same (or better) functionality can be achieved with a simple shell script?

I have been syncing my data up&down with this kind of setup. This is easy to implement, portable, robust and beautiful.

1. You need to install few niceties first. This script uses rsync with a crontab trigger, so get those from rzr's Harmattan repository.
(I don't know about Frematle repositories but I belive cron and rsync are there too...)

if you don't have the repository there yet, put "deb http://repo.pub.meego.com/home:/rzr:...ttan_standard/ ./" into your sources list
apt-get update
apt-get install cron rsync

2. You need to create a pair of rsa keys and install the public key to the cloudservers users authorized_keys, just the standard stuff.
I recommend creating a dummy user just for the sync purposes...

3. Here is the cloudsync.sh script, put it to your "/root" for example:

Code:

#!/bin/sh

G_LOGTAG="cloudsync"
G_LOCKFILE="/var/lock/cloudsync.pid"
G_USERNAME="your_cloudserver_username_goes_here"
G_DNSNAME="dns.name.of.your.cloudserver"
G_LOCALNAME="server.local.ip.address"
G_OWNIP="ip.given.to.me"
G_UPLOAD_DIRS="/home/user/MyDocs/DCIM /home/user/MyDocs/.backups /home/user/MyDocs/AptBackup /home/user/MyDocs/meeTrainer_workouts /home/user/MyDocs/.CallRecorder /home/user/MyDocs/Pictures"

if [ -e "$G_LOCKFILE" ]; then
  logger -t "$G_LOGTAG" "cannot get lock $G_LOCKFILE, exiting"
  exit 1
fi

echo "$$" > "$G_LOCKFILE"

## get the wlan state to see if we're in home network
aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")
if [ "$?" -eq "0" ]; then
  G_CLOUDHOST=$G_LOCALNAME
  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"
else
  G_CLOUDHOST=$G_DNSNAME
  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"
fi

G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIRS $G_USERNAME@$G_CLOUDHOST:)
G_RET=$?

logger -t "$G_LOGTAG" "$G_SYNC"

if [ "$G_RET" -eq "0" ]; then
  logger -t "$G_LOGTAG" "Upload tranfer succeeded"
else
  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"
fi

G_SYNC=$(/usr/bin/rsync -rptv $G_USERNAME@$G_CLOUDHOST:upload/ /home/user/MyDocs/Downloads/)
G_RET=$?

logger -t "$G_LOGTAG" "$G_SYNC"

if [ "$G_RET" -eq "0" ]; then                 
  logger -t "$G_LOGTAG" "Download tranfer succeeded"
else
  logger -t "$G_LOGTAG" "Download transfer failed (error code $G_RET)"
fi

rm -f "$G_LOCKFILE"

return 0

4. Edit the variables in the script to your liking:

Code:

G_USERNAME="your_cloudserver_username_goes_here"
G_DNSNAME="dns.name.of.your.cloudserver"
G_LOCALNAME="server.local.ip.address"
G_OWNIP="ip.given.to.me"

5. put something like this in your crontab:

Code:

~ # 
~ # crontab -l

*/30  * * * * /root/cloudsync.sh > /dev/null 2>&1

~ #

DESCRIPTION:

So, for the people who bothered to read to the end of this posting

This script can be modified to sync any desired directories. This example syncs my:

DCIM
.backups
AptBackup
meeTrainer_workouts
.CallRecorder
Pictures

folders, but that's just my preferences. Anything goes.

The folder "upload" on the cloudserver is synced to "Downloads" on the users MyDocs, and same applies, anything goes.

The script detects if you are on home network and uses direct connection to the cloudserver if so. Of course that works this way for me because I have my servers on the same subnet as my WLAN. You might want to modify that setup based on your network configuration...

The crontab entry launches cloudsync every 30 minutes, and additionaly it can be launched at any given moment directly. The script creates a lockfile in /var/lock to prevent double invocation.

Security related thingies;

You will need the private rsa key in the roots .ssh/ on the device, which means that if you lose your device the key can be considered compromised. (but of course you have the root password to set to something else than "rootme", don't you?)
For this reason I recommend creating a separate user for backup purposes on the cloudserver. Additionally you might want to run that user on chroot jail, or forward the ssh connection to a virtual machine or separate HW.

Quote & Reply |

The Following 22 Users Say Thank You to juiceme For This Useful Post:
ajalkane, bockersjv, danested, durneyj, imaginaryenemy, localhost, MaddogG, maegon9y00, minimos, mrsellout, mr_pingu, nokiabot, P@t, peterleinchen, reinob, shanttu, Sohil876, thedead1440, Wikiwide, xman, zamorph

	nokiabot	2013-04-23 , 17:28
	Posts: 1,974 \| Thanked: 1,834 times \| Joined on Mar 2013 @ india	#2

Good job......juicy buddy

Quote & Reply |

The Following User Says Thank You to nokiabot For This Useful Post:
Wikiwide

jackburton	2013-04-23 , 17:38
Posts: 466 \| Thanked: 661 times \| Joined on Jan 2009	#3

This is cool. The one thing I like about dropbox, though is that it is 'event-based'. E.g, put a file there, and the kernel knows and does the sync then and there.

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Quote & Reply |

The Following 3 Users Say Thank You to jackburton For This Useful Post:
juiceme, nokiabot, Wikiwide

	www.rzr.online.fr	2013-04-23 , 18:18
	Posts: 1,348 \| Thanked: 1,863 times \| Joined on Jan 2009 @ fr/35/rennes	#4

I had a such project to do this over webdav ...

__________________
Current obsession:

https://purl.org/rzr/abandonware

Please help to list all maemo existing apps :

https://github.com/abandonware/aband...ment-578143760

https://wiki.maemo.org/Apps#

I am looking for " 4 inch TFT LCD display screen " for Nokia n950 HandSet

http://rzr.online.fr/q/lcd

Also, I need online storage to archive files :

http://db.tt/gn5Qffd6#

https://my.pcloud.com/#page=register...e=g8ikZmcfEJy#

Quote & Reply |

The Following User Says Thank You to www.rzr.online.fr For This Useful Post:
Wikiwide

juiceme	2013-04-23 , 18:34
Community Council \| Posts: 4,920 \| Thanked: 12,867 times \| Joined on May 2012 @ Southerrn Finland	#5

Originally Posted by jackburton

This is cool. The one thing I like about dropbox, though is that it is 'event-based'. E.g, put a file there, and the kernel knows and does the sync then and there.

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Yes, that would easy to do really. The reason I don't have it is that I have just launched the sync up manually if I need something transferred immediately.

On the other hand, if there's something on the cloud end that needs to be synced immeiately to the phone it's a trickier situation. If that's needed then the phone would have to poll the remonte end at some shortish interval which is a sure battery killer.

Quote & Reply |

The Following 2 Users Say Thank You to juiceme For This Useful Post:
maegon9y00, Wikiwide

juiceme	2013-04-23 , 20:09
Community Council \| Posts: 4,920 \| Thanked: 12,867 times \| Joined on May 2012 @ Southerrn Finland	#6

Originally Posted by jackburton

Perhaps you could use a python wrapper or something that uses inotify and then the same effect could be achieved without cron. Just an idea.

Well now as I tried it, I am thinking if it's worthwhile to push up an event about file change or if new file event is enough?

I guess normally dropbox reacts when you copy a file to the wtched directory, but what if a file is modified in the directory, should it be uploaded immediately?

Quote & Reply |

The Following User Says Thank You to juiceme For This Useful Post:
Wikiwide

	qwazix	2013-04-23 , 21:33
	Moderator \| Posts: 2,622 \| Thanked: 5,447 times \| Joined on Jan 2010	#7

Last time I tried to implement my own sync solution with rsync I stumbled upon the inability of rsync to track deleted files in two way syncs. If you pass -delete it deletes missing files for both ends regardless if a file is new on one side. I resorted to using unison, which works like a charm four-way between my windows pc, ubuntu box, N900 and macbook. I suppose that compiling it for N9 should be trivial, if the debian binary doesn't work oob.

__________________
Proud coding competition 2012 winner: ρcam
My other apps: speedcrunch N9 N900 Jolla –– contactlaunch –– timenow

Nemo UX blog: Grog
My website: qwazix.com
My job: oob

Quote & Reply |

The Following 3 Users Say Thank You to qwazix For This Useful Post:
juiceme, maegon9y00, Wikiwide

juiceme	2013-04-23 , 21:44
Community Council \| Posts: 4,920 \| Thanked: 12,867 times \| Joined on May 2012 @ Southerrn Finland	#8

Yes, it might get tricky if you want to handle more than just updates. That's the reason I just upsync stuff most of the time, and I have separate directory for things to download to the device from cloud.

Here is a quick&dirty solution for monitoring a directory (/home/user/mydocs/Upload/) and immediately syncing the content to cloud if a new file is dropped there:

Code:

#!/usr/bin/python
import os
import subprocess
import pyinotify

watchmanager = pyinotify.WatchManager()
watchmask =  pyinotify.IN_CREATE

class watchprocess(pyinotify.ProcessEvent):
   def process_IN_CREATE(self, event):
      subprocess.call("/root/upload2cloud.sh", shell=True)

inotifier = pyinotify.Notifier(watchmanager, watchprocess())
ret = watchmanager.add_watch('/home/user/MyDocs/Upload', watchmask, rec=True)

while True:
   try:
      inotifier.process_events()
      if inotifier.check_events():
         inotifier.read_events()
   except KeyboardInterrupt:
      inotifier.stop()
      break

You also need another script that is launched by the python inotify monitor script. I could have reused cloudsync.sh but I decided to strip it down a bit to "/root/upload2cloud.sh", like this:

Code:

G_LOGTAG="cloudsync"
G_LOCKFILE="/var/lock/update2cloud.pid"
G_USERNAME="your_cloudserver_username_goes_here"
G_DNSNAME="dns.name.of.your.cloudserver"
G_LOCALNAME="server.local.ip.address"
G_OWNIP="ip.given.to.me"
G_UPLOAD_DIR="/home/user/MyDocs/Upload/"

logger -t "$G_LOGTAG" "Triggered upload from filesystem"

if [ -e "$G_LOCKFILE" ]; then
  logger -t "$G_LOGTAG" "Cannot get lock $G_LOCKFILE, exiting"
  exit 1
fi

echo "$$" > "$G_LOCKFILE"

## get the wlan state to see if we're in home network
aa=$(/sbin/ifconfig wlan0 | grep "$G_OWNIP")
if [ "$?" -eq "0" ]; then
  G_CLOUDHOST=$G_LOCALNAME
  logger -t "$G_LOGTAG" "Using WLAN connection to server $G_CLOUDHOST"
else
  G_CLOUDHOST=$G_DNSNAME
  logger -t "$G_LOGTAG" "Using GPRS connection to server $G_CLOUDHOST"
fi

G_SYNC=$(/usr/bin/rsync -av $G_UPLOAD_DIR $G_USERNAME@$G_CLOUDHOST:download)
G_RET=$?

logger -t "$G_LOGTAG" "$G_SYNC"

if [ "$G_RET" -eq "0" ]; then
  logger -t "$G_LOGTAG" "Upload tranfer succeeded"
else
  logger -t "$G_LOGTAG" "Upload transfer failed (error code $G_RET)"
fi

rm -f "$G_LOCKFILE"

return 0

Quote & Reply |

The Following 7 Users Say Thank You to juiceme For This Useful Post:
jackburton, maegon9y00, peterleinchen, qwazix, reinob, thedead1440, Wikiwide

jackburton	2013-04-25 , 00:58
Posts: 466 \| Thanked: 661 times \| Joined on Jan 2009	#9

While on topic, would anybody here want to try to evaluate sparkleshare for fremantle and harmattan?

Quote & Reply |

The Following User Says Thank You to jackburton For This Useful Post:
Wikiwide

	knobtviker	2013-04-27 , 08:20
	Posts: 665 \| Thanked: 2,388 times \| Joined on Feb 2012 @ Zagreb, Croatia	#10

Just a question:
When you update an existing file, do you upload/download a whole file or just the part of the file that changed?

__________________
¬ Flattr me for Wazapp
¬ READ THE WAZAPP WIKI
¬ Support my work by buying Dropian - a Dropbox client
¬ Get Soundwave - SoundHound for N9

Quote & Reply |

The Following User Says Thank You to knobtviker For This Useful Post:
Wikiwide

Page 1 of 2

Next >

Active Topics

Firefox with Leste (10)

Notices