rsync functionality in Ruby?
Paul van Delst
I wrote a ruby script to allow me to automatically update a repository working copy from
an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
add/delete and commit. Worked great on my test system (linux). However, because of
security issues, on the system where this script will be used there is no rsync utility
available.
So, I was wondering, is there an existing rsync-like ruby solution? I checked this
newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
functionality (lots of utilities that use/manage rsync though.)
Note that I don't really need the "r" portion of rsync (the sync occurs on the same
filesystem), just the ability to synchronise two hierarchies with "--exclude <files>"
capability.
I thought I'd ask before embarking on a wheel-reinvention project.
cheers,
paulv
p.s. I'm pushing my luck here, but guidance on an algorithm or recipe for sync
functionality would also be appreciated. :o)
dusty
I haven't found any either, but I do have a nagios plugin using rsync
to check the timestamps on remote files over rsync. New to posting
here, whats the best way to post code? Include it inline? I'd be
happy to share if you want it. I basically use open4 to call rsync
and then parse the results. It might be helpful in your wheel
project. :)
Dusty Doris
Paul van Delst
> On Aug 31, 12:49 pm, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:
>> Hello,
>>
>> I wrote a ruby script to allow me to automatically update a repository working copy from
>> an unversioned hierarchy. I use rsync to synchronise the directories prior to svn
>> add/delete and commit. Worked great on my test system (linux). However, because of
>> security issues, on the system where this script will be used there is no rsync utility
>> available.
>>
>> So, I was wondering, is there an existing rsync-like ruby solution? I checked this
>> newsgroup, rubyforge, raa, etc but couldn't find anything that replicated rsync
>> functionality (lots of utilities that use/manage rsync though.)
>>
> I haven't found any either, but I do have a nagios plugin using rsync
> to check the timestamps on remote files over rsync. New to posting
> here, whats the best way to post code? Include it inline? I'd be
> happy to share if you want it. I basically use open4 to call rsync
> and then parse the results. It might be helpful in your wheel
> project. :)
Hi,
Thanks for the reply. But, if I understand your reply, I can't use anything that uses unix
rsync under the covers, because there is no rsync available on the system in question.
cheers,
paulv
Corey Jewett
>> On Aug 31, 12:49 pm, Paul van Delst <Paul.vanDe...@noaa.gov> wrote:
>>> Hello,
>>>
>>> I wrote a ruby script to allow me to automatically update a
>>> repository working copy from
>>> an unversioned hierarchy. I use rsync to synchronise the
>>> directories prior to svn
>>> add/delete and commit. Worked great on my test system (linux).
>>> However, because of
>>> security issues, on the system where this script will be used
>>> there is no rsync utility
>>> available.
>>>
>>> So, I was wondering, is there an existing rsync-like ruby
>>> solution? I checked this
>>> newsgroup, rubyforge, raa, etc but couldn't find anything that
>>> replicated rsync
>>> functionality (lots of utilities that use/manage rsync though.)
You should look at FileUtils::cp_r with the :preserve option. It's
not the same as rsync by any stretch, but it should be fairly similar
assuming you didn't need --delete or --include/--exclude patterns. If
you do need those, then the source of FileUtils is probably a pretty
good place to start rolling your own since it properly does symlinks,
permissions, times, and other metadata.
Corey
Robert Klemme
Other options for unidirectional update on system with GNU utils are:
- use cp -au
- maybe you can also facilitate tar
You can build synchronization out of these by invoking then twice, i.e.
once for each direction.
Other than that it should not be too difficult to implement something on
your own using methods in class File.
Kind regards
robert
Bob Hutchison
On 31-Aug-07, at 4:15 PM, dusty wrote:
> I haven't found any either, but I do have a nagios plugin using rsync
> to check the timestamps on remote files over rsync. New to posting
> here, whats the best way to post code? Include it inline? I'd be
> happy to share if you want it. I basically use open4 to call rsync
> and then parse the results. It might be helpful in your wheel
> project. :)
Is this rsync+ssh? If so, what did you handle passwords when making
the connection (I wound up going though disgusting conniptions using
expect)? I know that you can avoid login by setting up .ssh/
known_hosts but that is frowned upon in some circles.
Cheers,
Bob
>
> Dusty Doris
----
Bob Hutchison -- tumblelog at http://
www.recursive.ca/so/
Recursive Design Inc. -- weblog at http://www.recursive.ca/
hutch
http://www.recursive.ca/ -- works on http://www.raconteur.info/
cms-for-static-content/home/
Dick Davies
> Is this rsync+ssh? If so, what did you handle passwords when making
> the connection (I wound up going though disgusting conniptions using
> expect)? I know that you can avoid login by setting up .ssh/
> known_hosts but that is frowned upon in some circles.
If you're dicking around with expect you have the
password in plain text somewhere along the line, so you'd be lucky to
avoid frowns even then :)
;Use RSA authentication - a blank passphrase on the key (for cronjobs), or
use ssh-agent (for everything else - unit testing etc).
--
Rasputin :: Jack of All Trades - Master of Nuns
http://number9.hellooperator.net/
Bil Kleb
> Hello,
Hi.
> I wrote a ruby script to allow me to automatically update a repository
> working copy from an unversioned hierarchy.
Is there anyway to use SVN's import command to get you 80% of
the way there? See, for example,
http://subversion.tigris.org/faq.html#in-place-import
Or, some combination of `cp` and `svn diff`? E.g.,
svn co working_copy
cp -r unversioned working_copy
svn diff > patch
[process the patch results]
or perhaps some sort of inverse, e.g.,
U_DIRS=`find unversioned -type d`
U_FILES=`find unversioned -type f`
svn co working_copy
cd working_copy
S_DIRS=`find . \( -name .svn -prune \) -o -type d -print`
S_FILES=`find . \( -name .svn -prune \) -o -type f -print`
svn rm --force [ S_DIRS - U_DIRS ]
svn rm --force [ S_FILES - U_FILES ]
cp -r unversioned/* .
svn ci -m'To sync with unversioned.'
Regards,
--
Bil Kleb
http://nasarb.rubyforge.org
Bob Hutchison
On 2-Sep-07, at 3:44 AM, Dick Davies wrote:
> On 01/09/07, Bob Hutchison <hu...@recursive.ca> wrote:
>
>> Is this rsync+ssh? If so, what did you handle passwords when making
>> the connection (I wound up going though disgusting conniptions using
>> expect)? I know that you can avoid login by setting up .ssh/
>> known_hosts but that is frowned upon in some circles.
>
> If you're dicking around with expect you have the
> password in plain text somewhere along the line, so you'd be lucky to
> avoid frowns even then :)
I know, funny isn't it :-)
>
> ;Use RSA authentication - a blank passphrase on the key (for
> cronjobs), or
> use ssh-agent (for everything else - unit testing etc).
I think I'm going to push a bit on the RSA authentication, see where
I get.
Cheers,
Bob
>
> --
> Rasputin :: Jack of All Trades - Master of Nuns
> http://number9.hellooperator.net/
>
----
dusty
Sorry I took so long to reply. Forgot about this thread. I am using
straight rsync. Running an rsync daemon on the servers I want. Then
I setup modules for the access. The reason I do this is to avoid
having ssh keys with blank passwords on my servers. Also, this is
only available on the internal network, so I'll also use things like
firewall rules and access-list in rsync to make sure only the machines
I want to get rsync access can.
Here's an example of a class that calls rsync to get a file listing in
a module.
----
require 'rubygems'
require 'open4'
class RsyncUtils
attr_reader :rsync
def initialize(path='')
paths = ['/bin',
'/usr/bin',
'/usr/local/bin',
'/opt/local/bin'
]
paths << path unless path.empty?
paths.each() do |path|
if(test(?x, "#{path}/rsync"))
@rsync = "#{path}/rsync"
end
end
raise "Cannot find rsync, please specify the path" unless @rsync
end
def list(hostname,path,recurs=false)
recursive = "--recursive" if recurs
r = run_command("#{@rsync} #{recursive} #{hostname}::#{path}")
if r.exitstatus == 0
r.results = RsyncFile.parse_list(r.stdout)
end
r
end
def list_recursive(hostname,path)
list(hostname,path,true)
end
private
def run_command(command)
pid, stdin, stdout, stderr = Open4.popen4(command)
ignored, status = Process::waitpid2 pid
RsyncResult.new(stdout.readlines,
stderr.readlines,
status.exitstatus)
end
end
Here is a class that hands back my results
----------
class RsyncResult
attr_reader :stdout, :stderr, :exitstatus
attr_accessor :results
def initialize(stdout,stderr,exitstatus)
@stdout = stdout
@stderr = stderr
@exitstatus = exitstatus
end
end
Here is the class that parses the filelisting to tell me about the
files
------
class RsyncFile
attr_reader :name, :type, :time, :size
FILE_TYPE = 0
DIRECTORY_TYPE = 1
UNKNOWN_TYPE = 2
def initialize(name,type,time,size)
@name = name
@time = time
@size = size
@type = type
end
def is_file?
@type == FILE_TYPE
end
def is_dir?
@type == DIRECTORY_TYPE
end
def self.parse_list(files)
results = []
files.each do |file|
parts = file.split
case parts[0]
when /^[rwx-]{10}$/
type = FILE_TYPE
when /^d[rwx-]{9}$/
type = DIRECTORY_TYPE
else
type = UNKNOWN_TYPE
end
time = Time.local(*ParseDate::parsedate("#{parts[2]}
#{parts[3]}"))
size = parts[1].to_i
name = parts[4]
results << RsyncFile.new(name,type,time,size)
end
results
end
end
Here is how I'd use it in a script.
rsync = RsyncUtils.new
check = rsync.list(@options["--hostname"],@options["--
path"],@options["--recursive"])
if check.exitstatus > 0
raise "#{check.stderr}"
end
#This would list all the files - the rsync output
if @options["--verbose"]
puts check.stdout
exit(0)
end
Now assume I have another class that is do a comparison of the results
for me. That is where I'm calling crit.compare. That's not as
important, as you can see I'm simply iterating through the results and
checking the Time on the current server vs. the time on the remove
file.
check.results.each do |r|
if r.type == RsyncFile::FILE_TYPE
diff = ((Time.now - r.time) - @options["--drift"].to_i).to_i
critical_files << r.name unless crit.compare(diff)
warning_files << r.name unless warn.compare(diff)
end
end
So, basically I would call the script
./rsync_file_age.rb --hostname somehost --path somedir/anotherdir/ --
warning 0:10 --critical 10:20
That would compile a command to RsyncUtils such as
rsync somehost::somedir/anotherdir/
It would then use open4 to run that command and would parse the
results.
Hope that is helpful.
BTW an rsyncd.conf file might look like this and would provide the
somedir module.
pid file = /var/run/rsyncd.pid
hosts allow = 10.0.0.10
uid = nobody
gid = nogroup
use chroot = no
max connections = 4
syslog facility = local5
[somedir]
path = /var/somedir
read only = true