I have implemented a two way sync between my servers, process was not smooth and there was not enough documentation online for new version of softwares that I have used, so I had to tweak around until two way sync finally work as desired.
I have used OpenSSH, Unison 2.40 and lsyncd 2.1.4 to do this, there are other ways and other softwares each with their own pros and cons.
First Setup Password-less Key-Based SSH login for both servers
I have used following command to generate key.
ssh-keygen -t rsaIt will generate a ssh key for current server and save it in user’s /.ssh/ folder. If it is a user, it will go to /home/user/.ssh/id_rsa.pub , if it is root, it will go to /root/.ssh/id_rsa.pub, copy content of id_rsa.pub and add it to following file in your server 2. /home/user/.ssh/authorized_keys or /root/.ssh/authorized_keys ( file name might be authorized_keys2 but it won’t cause problem )
You don’t need to restart ssh or anything, now you can simply test your login to second server by using ssh command:
ssh -p 22 127.0.0.1where 127.0.0.1 is your server 2 IP address.
Now, repeat same thing on server 2 to enable login from server 2 to server 1 as well.
Install lsyncd and unison on both servers
You might need to add some rpm sources for it.
These are my favorite repositories.
rpm -Uvh http://ftp.jaist.ac.jp/pub/Linux/Fedora/epel/6/x86_64/epel-release-6-7.noarch.rpm (for CentOS 6)
rpm -Uvh http://rpms.famillecollet.com/enterprise/remi-release-6.rpmrpm -Uvh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
You can simply install lsyncd and unison by using our beloved Yum installer in linux, simply use following command:
yum install lsyncd unison
if things doesn’t go wrong and you have required repositories installed, there shouldn’t be a problem.
Now comes the hard part…
Configue lsyncd
I was struggling for hours to figure out how to establish a good two way sync. First I was using rsync and I noticed lsyncd will go into an endless loop, then I came up with unison and I have faced many different errors!
First of all, unison, need “HOME” in environment to be set, it will use it to hold last sync profile information and list of files with their timestamps.
You need to set it in service script of lsyncd.
Edit /etc/init.d/lsyncd:
nano /etc/init.d/lsyncd
and Add following at top of the file:
export HOME="/root/.unison/"
You can change folder to anything else if you want.
This will save you from annoying document-less error of:
Fatal error: exception Util.Fatal("Environment variable HOME not found")
Now, you need to configure lsyncd.
settings {
logfile = "/etc/.lsync/lsyncd.log",
statusFile = "/etc/.lsync/lsyncd.status",
insist = 1,
exitcodes = {[0] = "ok", [1] = "ok", [2] = "die"}
}
folder1 = {
onStartup = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder1 ssh://10.0.0.101//home/sanarena/public_html/folder1 -prefer /home/sanarena/public_html/folder1",
onCreate = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder1 ssh://[email protected]//home/sanarena/public_html/folder1 -prefer /home/sanarena/public_html/folder1",
onDelete = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder1 ssh://[email protected]//home/sanarena/public_html/folder1 -prefer /home/sanarena/public_html/folder1",
onModify = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder1 ssh://[email protected]//home/sanarena/public_html/folder1 -prefer /home/sanarena/public_html/folder1",
onMove = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder1 ssh://[email protected]//home/sanarena/public_html/folder1 -prefer /home/sanarena/public_html/folder1",
}
sync{folder1,maxProcesses = 1, source="/home/sanarena/public_html/folder1"}
folder2 = {
onStartup = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder2 ssh://[email protected]//home/sanarena/public_html/folder2 -prefer /home/sanarena/public_html/folder2",
onCreate = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder2 ssh://[email protected]//home/sanarena/public_html/folder2 -prefer /home/sanarena/public_html/folder2",
onDelete = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder2 ssh://[email protected]//home/sanarena/public_html/folder2 -prefer /home/sanarena/public_html/folder2",
onModify = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder2 ssh://[email protected]//home/sanarena/public_html/folder2 -prefer /home/sanarena/public_html/folder2",
onMove = "/usr/bin/unison -auto -batch -ignorearchives /home/sanarena/public_html/folder2 ssh://[email protected]//home/sanarena/public_html/folder2 -prefer /home/sanarena/public_html/folder2",
}
sync{folder2, maxProcesses = 1,source="/home/sanarena/public_html/folder2"}
As you can see, you need to set a folder for lsyncd log and status.
mkdir /etc/.lsync/
Unison return non-standard return codes and it will cause lsyncd to terminate, therefore, we override default exitcodes and make exit code 1 to be ok (default is die) which you can see in setting segment.
Then I have defined configuration for folder1 followed by command to run folder1 sync.
OnAttrib was giving me error, so I just removed it from the line, as it doesn’t really help me with what I was going do to. (and I didn’t need it)
unison’s auto and batch command make it to run silently in background without asking any additional question. it will sync folder1 to remote ssh folder in server2. prefer parameter will overwrite server2 file with server1 if there is conflict. I have set it because of an obvious reason. this commands will run only when there is a file change detected by lsyncd, so when lsyncd detect a file is modified, it means file is modified in current server, so it is safe to copy it to server2. ( for some weird reason, unison couldn’t keep track of copied file and it was reporting both server files as modified file, and halt the copy process, so I had to add “prefer” parameter )
Then I continue and define same things for second folder. you can add more folder to sync if you like to.
Repeat same thing with server2
just change IP address (and update folder paths if they are different)
Run and Test lsyncd
You can run lsyncd using following command, run this command on both servers:
service lsyncd restart
You can repeat same command two times, if second time, it is still showing “Failed” for shutting down service, it means service was automatically terminated and there is a error, otherwise, you are good to go!
If you experience error, you can monitor error using following command:
tail -f /etc/lsync/lsyncd.log
Boot load
We are good up until now, so lets lsyncd service to run in startup. run this command on both servers.
chkconfig --levels 235 lsyncd on
Enjoy!
I hope this article helped you to save time doing this on your servers, I did it on my CentOS 6 servers and it take me few hours as I was struggling with trial errors as I couldn’t find enough documentation online on how to setup this.