File Replicating / Synchronization

[HKStrongside]

Howdy all. We are starting to grow our web farm significantly however
we are having a difficult time coming up with an effective and fast
solution for replicating files. What I would like to do is publish
from CMS to a server, then to have some 3rd party app or something
synchronize all of the web servers to this master. What have you all
used and if you could use anything what would it be?

Thanks!

HKStrongside

[Bill Dennen]

We're doing this with rsync. Works very well.

-Bill

[TheJonathan]

We use Repliweb (http://www.repliweb.com/). It's a great client-
server model for moving just the changed files across. It's very
flexible and has an API that you could use to trigger replication from
CMS Asynchronous tasks. We just use a 2 minute, changes-only job to
push files over to our production web-servers.

--Jonathan

[HKStrongside]

Thanks guys. How many servers receive/pull the files and how fast are
the transfers happening?
Currently we have 4 projects with about 600 active pages per project.
I publish to 3 servers via UNC out of CMS. We start publishing at
12:00am and our 2 biggest projects finish around 4-5am. However
depending on what was changed, during that time will have broken pages
and images all over the place. We are going to have around 15
servers. I would like to publish to to the CMS server from CMS then
have replication update the webservers from there. How does that
compare to what you guys are doing and the products you use to do it?
Thanks!
HKStrongside

[Rakefet Shohat, RepliWeb]

For the number of pages and servers that you mention, a 4 hours
process sounds terrible. RepliWeb can cut this down to minutes so if
you are thinking of scaling to 15 server you should take a look at it.
It supports both automated (scheduled) and continuous (real time) mode
and if this is IIS, it also takes care of all the metabase changes
replication. Works also extremely well on wide area networks.

[RedDotPM]

CMS publishing is the single most instensive process relating to
pretty much all aspect of the system, processor speed, disk speed and
network speeds are all heavily used. The thing to remember when
dealing with publishing is that, while you may only be setting up one
publishing job, if you are publishing to muliple servers, the job is
repeated each time for each server.

If you have multiple web servers and are using a load balancer, I
generally recommend that you set the CMS server up to publish to a
staging machine (an older, phased peice of hardware that is just
sitting around since so many company's seem to have them lately, and
then do exactly what everyone else has said and use something that
will monitor the folder and push the staging publish out to the web
servers, Cute FTP pro works really well for this and is only $69.00
(usd)

This actually serves two purposes. First, it cuts the publication time
down so that only one publishing job is involved and secondly, it
ensures uniformity throughout your live web servers.

I am currently running some benchmark tests relative to publishing and
should have that information in the next week or so.

Andrew Salik, Project Manager
RedDot Solutions, NA

On May 17, 6:02 am, "Rakefet Shohat, RepliWeb" <rake...@repliweb.com>
wrote:

[Nick Wesselman]

One aspect to keep in mind too when replicating from a staging server
as Andrew described is the type of pub target you use. Besides the
type of connection, the big difference between FTP and UNC publishing
is that RedDot will only transfer new and changed files when using FTP
publishing. A copy of the last publish is kept in RedDotTemp for this
purpose. UNC publishing will transfer every file, regardless of
changes. Unless your replication tool has a fast diff/checksum
process, FTP publishing may be a better option as then your
synchronization can be based on file date. As always you should
compare the two in your environment.

I'm curious to hear about your benchmarks Andrew as mine also seem to
indicate that as projects get larger, UNC loses what speed advantage
it has over FTP due to the diff process performed when using FTP
publishing.

In the realm of free replication options you might look at robocopy
(part of Windows Resource Kit) or rsync as Bill mentioned (which can
be installed on Windows from what I hear).

Nick

> > replication. Works also extremely well on wide area networks.- Hide quoted text -
>
> - Show quoted text -