[technical] multi-master lsyncd replication and file locking

I have recently started to use lsyncd to support automated replication of data for web servers. Because of the particular requirements of one of my customers, I needed to set up bidirectional syncing between two servers on two different coasts. This presented a good deal of trouble initially because the lsyncd on one coast would react to files being written to it by writing those files back to the other coast. This was fine for writes that happened quickly, but for writes that happen slowly, like large files being FTP’d, it would lead to file truncation and file permission issues.

The solution I found was to update to the latest lsyncd (2.1.4) which permitted specifying the rsync binary, and then provide a “wrapper” binary written in perl that would verify that all of the files to be synced were not currently being written to (via lsof). Below is the perl script (placed in /usr/local/bin/rsync-wrapper):


#!/usr/bin/perl

$args = “”;

while($arg = shift) {
$args .= $arg . ” “;
push(@z, $arg);
}

slog(“args: $args”);

$dest = pop(@z);
$src = pop(@z);
slog(“dest: $dest src: $src”);

if($args =~ /–force/) {
# exec
slog(“delay list”);
$delay = 1;
}

$list = “”;

while($in = <>) {
$list .= $in;
if($delay) {
$file = $in;
chop($file);
$filepath = $src . $file;
if(-f $filepath) {
$count = 0;
while(0 == ($result = system(“lsof $filepath”))) {
slog(“$count – result: $result path: $filepath”);
$count++;
sleep(1);
}
}
}
}

slog(“Exec rsync”);
open(LIST, “|/usr/bin/rsync ” . $args);
print LIST $list;
close(LIST);

sub slog {
my $msg = shift;
open(LOG, “>>/var/log/rsync-wrapper.log”);
print LOG $msg . “\n”;
close(LOG);
}

and the config file for lsyncd:


settings = {
logfile = “/var/log/lsyncd.log”,
statusFile = “/tmp/lsyncd.status”,
maxProcesses = 5,
delay = 0,
}

sync {
default.rsync,
source=”/home”,
target=”www.east.omitted.net::home”,
excludeFrom=”/etc/lsyncd/exclude”,
rsync = {
binary = “/usr/local/bin/rsync-wrapper”,
owner = true,
group = true,
}
}

Note that this is only one of several solutions I identified for this problem. If you are having similar problems and this doesn’t help, feel free to contact me for a list of them.

One Response to “[technical] multi-master lsyncd replication and file locking”

  1. TheRealB Says:

    Hello,

    thanks for the great work that you did, but something is missing in your script.
    $in will never be set and so the script doesnt work.
    I guess $in should be a list of filenames…

Leave a Reply