Server Redundancy
Tim Schwartz
So I was going to give a little bit more two cents as to how to go
about building a redundant system. Two things to keep in mind as you
read this:
-I am not an expert at all but have dealt with replication and
redundancy with servers
-People in the technology sector have been dealing with disaster
preparedness for years, the bottom line is multiple backups and
failsafes in various locations, that is by far the best way to
minimize downtime and data loss
Basically what I would suggest is multiple VPSs on different hosts (at
least 3 hosts), making sure hosts are picked to span various networks
around the globe. Choosing the best and widest web-service to host
files and virtually mounting a directory to the webservers, currently
I think S3 would be a great choice and mounting with FUSE. This means
that the files among the VPS stay the same (of course each VPS should
backup those files to their disk every day or hour or something). Then
run a mysql replication among the 3, so each has the same data for the
wiki and wordpress installs. Finally something like Varnish would be
installed on all of the servers, but one would be chosen as the
master, but should that one go down, any of the VPS could pick up as
the master. The Varnish master would act as the load balancer
splitting requests among the VPSs. This could all be installed as an
image or one big script that goes onto a blank debain, unbuntu,
centos... image and updates the new install with all necessary
packages and sets up all of the syncing and configuration. That way at
any time a new VPS could be added on a new host.
In the end you design a system that is actively redundant among at
least 3 separate infrastructures. You gain capabilities in terms of
the load you can take because you are load balancing them with a
varnish round robin and this also serves as redundancy.
That said you will loose out on the DB replication among the 3
servers, this takes bandwidth and you would have to make sure you get
VPS with unlimited bandwidth. As well there is a dependency on Amazon
S3, but with each server backing up hourly the files (and each could
take over as a NFS server should S3 fail), you should be in good
hands.
It's just an idea. I have done the above under one hosting provider
with 5 servers and a development box to deploy changes from, and it
works great.
2cents,
Tim
deborah shaddon
I'll have to see if OSL has VPS, I'm not sure, we were not planning on
having the 'redundancy' discussions with them until we moved past
initial setup, but I do know we've spoken of the need for this
previously. That is part of the 'gap analysis' we are
doing...depending on this gap, we can solution around it, including a
secondary hosting backup if we have too (although they should be able
to do that). Regarding VPS for some dns level redundancy, that is a
good discussion too. A couple of OSL folks are members of this group,
but may respond here or through me, but realize we are also in the
process of establishing an SOW with them (so not everything will be on
the table with them) for current and future support, why we want to
manage the requirements will be managed centrally, but we can
certainly have the dialog....In general, I think we agree that we have
NFR's around redundancy (as a measure of our own disaster
prepardness), as well as some RTO/RPO requirements on data backups as
well to be refined/discussed. I'll get us going on a dialog about
this topic...
Now if I could actually get folks on this group to test the test-wiki
we setup at OSL, I could actually start the true migration from
dreamhost, but I won't fully move over until we kick-the-tires, and so
far, it's only been Spike and me....so please TEST folks!!: (smile)
http://cc-wiki.osuosl.org/wiki/Main_Page
deborah shaddon
Ok, here is the background: We have been working with OSL to setup a
VM environment, that is now their standard anyway (on a completely
opensource stack, using Ganeti). However, they are not at the level
of Amazon with the elacticity in the environment or the ability to
setup and manage VPS for the type of redundancy scenarios that were
mentioned, so this would be a 'gap' of a requirement, and then I would
think (per my notes the other day), that we would look to fill the gap
through some other mechanism not reliant on OSL, we can consider an
active-passive failover scenario (once we define the appropriate RTO/
RPO as I mentioned in other post, I outlined some requirements there),
at some time in the future at a different geographical site, however,
this isn't a huge priority at the moment, getting the server
environment stood up there and tested is, as well as working on the
SOW with OSL to outline support procedures and SLAs, so I'm kinda
focusing my energies on that and adding to the Infra wish-list, but
once we are stood up, if you want to help drive some solutions, would
love to hear those. This weekend, I'm also going to create the OSL
profile (now that we've had some experience this week with the wiki)
on the process, the technology stack, etc., so I can more broadly
share that with folks. (I've setup a skype chat for Saturday noon CT
to reconnect with folks, can't skype from work, and I've worked till
10pm every day this week)
Here is some stuff on ganeti if you are interested:
[1]
http://www.lancealbertson.com/2010/05/creating-a-scalable-virtualization-cluster-with-ganeti/
[2] http://www.lancealbertson.com/slides/ganeti-scale9x/#14
[3] http://code.google.com/p/ganeti/
Thanks!!
Deborah
On Mar 16, 7:46 pm, deborah shaddon <deborahshad...@gmail.com> wrote:
> Hey Tim,
>
> I'll have to see if OSL has VPS, I'm not sure, we were not planning on
> having the 'redundancy' discussions with them until we moved past
> initial setup, but I do know we've spoken of the need for this
> previously. That is part of the 'gap analysis' we are
> doing...depending on this gap, we can solution around it, including a
> secondary hosting backup if we have too (although they should be able
> to do that). Regarding VPS for some dns level redundancy, that is a
> good discussion too. A couple of OSL folks are members of this group,
> but may respond here or through me, but realize we are also in the
> process of establishing an SOW with them (so not everything will be on
> the table with them) for current and future support, why we want to
> manage the requirements will be managed centrally, but we can
> certainly have the dialog....In general, I think we agree that we have
> NFR's around redundancy (as a measure of our own disaster
> prepardness), as well as some RTO/RPO requirements on data backups as
> well to be refined/discussed. I'll get us going on a dialog about
> this topic...
>
> Now if I could actually get folks on this group to test the test-wiki
> we setup at OSL, I could actually start the true migration from
> dreamhost, but I won't fully move over until we kick-the-tires, and so
>
> - Show quoted text -