fault tolerant web servers on freebsd
Maciej Jan Broniarz
I am thinking about building a fault tolerant web servers running FreeBSD. The servers would be serving web sites using jails+apache+php+mysql.
Would anyone be so kind, as to give me some advice about building such a solution? I have read a lot about freebsd cluster, but it was focused on computing clusters, and not fault tolerant one.
If anyone has some experience with the issue, I'll be very grateful.
Best regards,
mjb
Mike Jakubik
http://www.freebsd.org/cgi/man.cgi?query=carp
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
>
Maciej Jan Broniarz
Thanks. Carp is fine, but I still have to figure out how to synchronize
data on disks or in the database. Usign just carp would leave me with
two separate databases containing only parts of users input. Is there a
soulution similar to drbd+heartbeat on Linux? Geom_gate looks nice, but
I have no idea if it has been used in a production enviroment.
All best,
mjb
Andriy Gapon
Perhaps you would find HAST useful:
http://wiki.freebsd.org/HAST
--
Andriy Gapon
Pete French
> I have no idea if it has been used in a production enviroment.
I use it for production - it works nicely, but you wont get automatic
failover that way.
-pete.
Maciej Jan Broniarz
> Maciej Jan Broniarz ha scritto:
>> W dniu 10-04-05 17:45, Mike Jakubik pisze:
>
> FreeBSD/Linux O.S.
> F.T. means outage can occur in every moment, but all current operation
> will be always completed by other equipments; so there will not be
> interrupted/lost operations.
Hmm. Thanks for the tip. Which *NIX os can be used to build an FT
solution then? Solaris? AIX? HP-UX?
> Carp is fine for HA is you have always both db and file storage always
> available.
> For DB you can use MySQL replica, cross-replicating local MySQL data
> among all servers. Each server will keep its own copy, always up to date
> with other servers.
> This will make all DB data always available, except data recorded in the
> last milliseconds.
>
> About file storage, you must have an external reliable NFS server.
So, to be redundant one should run 2 x web servers + 2 x storage +2 x
mysql. Sounds like fun, but also like a lot of work. So how do the web
hosting prividers manage to use FreeBSD for theirs solution?
I am curious do they have a cluster/reudndant solution for every part of
the system, or they just accept thet something will fail at some time.
Best regards,
mjb
Tonix (Antonio Nati)
Just to use exact words, fault toulerancy is not possible with any
FreeBSD/Linux O.S.
F.T. means outage can occur in every moment, but all current operation
will be always completed by other equipments; so there will not be
interrupted/lost operations.
The most you can have is HA (High Availability), that means operations
currently executed on servers which have outage are interrupted, but new
operations will be completed as they will be executed on other equipments.
Carp is fine for HA is you have always both db and file storage always
available.
For DB you can use MySQL replica, cross-replicating local MySQL data
among all servers. Each server will keep its own copy, always up to
date with other servers.
This will make all DB data always available, except data recorded in the
last milliseconds.
About file storage, you must have an external reliable NFS server.
Tonino
> All best,
> mjb
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
>
--
------------------------------------------------------------
Inter@zioni Interazioni di Antonio Nati
http://www.interazioni.it to...@interazioni.it
------------------------------------------------------------
Tonix (Antonio Nati)
> W dniu 10-04-05 22:08, Tonix (Antonio Nati) pisze:
>> Maciej Jan Broniarz ha scritto:
>>> W dniu 10-04-05 17:45, Mike Jakubik pisze:
> >
>> Just to use exact words, fault toulerancy is not possible with any
>> FreeBSD/Linux O.S.
>> F.T. means outage can occur in every moment, but all current operation
>> will be always completed by other equipments; so there will not be
>> interrupted/lost operations.
>
> Hmm. Thanks for the tip. Which *NIX os can be used to build an FT
> solution then? Solaris? AIX? HP-UX?
For a FT solution you need usually an hardware solution, very expensive.
Server must have all doubled, with custom chips for checking if parts
are working (and which part is broken, which may not be easy to understand).
VMware claims to have a software fault toulerant solution, syncronizing
two servers in real-time, but I don't know the efficiency.
The most ISP use HA solutions, which are ok for the most of WEB/e-mail
operations.
Operations are not available for a few milliseconds/seconds, depending
on the architecture you use.
Just to complete, drbd+heartbeat are NOT FT. They are HA. When the
master server goes down, backup server must acknowledge the new status,
then mount the replicated disk (think to disk check!), then start
services working on that disk. So this solution needs a lot of
seconds, probably minutes to work.
With a good carp architecture, you just need milliseconds.
Tonino
jfa...@goldsword.com
> W dniu 10-04-05 22:08, Tonix (Antonio Nati) pisze:
>> Maciej Jan Broniarz ha scritto:
>>> W dniu 10-04-05 17:45, Mike Jakubik pisze:
>>
>> Just to use exact words, fault toulerancy is not possible with any
>> FreeBSD/Linux O.S.
>> F.T. means outage can occur in every moment, but all current operation
>> will be always completed by other equipments; so there will not be
>> interrupted/lost operations.
>
> Hmm. Thanks for the tip. Which *NIX os can be used to build an FT
> solution then? Solaris? AIX? HP-UX?
You're asking a question that has not easy or compete answer. Fault
tolerant at what level? Do you want to guarantee that _every_ DB
operation competes? What about random file reads or writes? Do you
want to guarantee that two identical operations with the same data
will produce the same result? (That's _not_ the same as the previous
questions...)
So first you have to define your workload, then define what errors you
must avoid or allow, and then define how to deal with failures,
errors, etc.
Then you can start talking about High Availability vs. level of Fault
tolerance, vs. ....
John
-----------------------------------------------------------------
J. T. Farmer <jfa...@goldsword.com> <jfa...@orfencer.org>
GoldSword Systems, Knoxville TN Coach & Instructor
Consulting, Knoxville Academy of the Blade
Software Development, Maryville Fencing Club
Project Management
Maciej Jan Broniarz
> Quoting Maciej Jan Broniarz <gau...@gausus.net>:
>> W dniu 10-04-05 22:08, Tonix (Antonio Nati) pisze:
>>> Maciej Jan Broniarz ha scritto:
>>>> W dniu 10-04-05 17:45, Mike Jakubik pisze:
>>>
>
> must avoid or allow, and then define how to deal with failures, errors,
> etc.
> Then you can start talking about High Availability vs. level of Fault
> tolerance, vs. ....
Let's say i need to run a few php/sql based web sites and I would like
to maintain uptime of about 99,99% per month. No matter how good the
hardware - it will always fail at some time. My goal is to build a
system, that can maintain that uptime.
From what You say I need some level of HA system, to maintain the
required uptime.
So, as I've said earlier (correct me, if I'm wrong) - the setup could
look something like that:
- 2 web servers with carp
- 2 storage servers with on-line sync mechanism running
- 2 mysql servers with on-line database replication
(i'm skiping power and network issues at the moment).
Few people have told me about a setup with linux, drbd and heartbeat
which offers them some level of HA. Has anyone tried anything similar on
FreeBSD?
mjb
Adam Vande More
>
> Let's say i need to run a few php/sql based web sites and I would like to
> maintain uptime of about 99,99% per month. No matter how good the hardware -
> it will always fail at some time. My goal is to build a system, that can
> maintain that uptime.
>
> From what You say I need some level of HA system, to maintain the required
> uptime.
>
> So, as I've said earlier (correct me, if I'm wrong) - the setup could look
> something like that:
>
> - 2 web servers with carp
> - 2 storage servers with on-line sync mechanism running
> - 2 mysql servers with on-line database replication
>
> (i'm skiping power and network issues at the moment).
>
> Few people have told me about a setup with linux, drbd and heartbeat which
> offers them some level of HA. Has anyone tried anything similar on FreeBSD?
>
Each HA implementation is different, you still really haven't provided
enough details eg are servers running off same switch, or are you going to
have latency issues(DRBD and ggate both dislike latency). DB replication
doesn't necessarily need to be done via block level replication. Generally
speaking, you'll make your storage backend HA first, then build other
services off that like iscsi or nfs. There is no single answer to this
because there are so many needs and ways to address them. and yes,
sysutils/heartbeat works with *BSD and they are in use. There are also
other options like
http://lethargy.org/~jesus/writes/postgresql-warm-standby-on-zfs-crack
--
Adam Vande More
Mehmet Erol Sanliturk
There is a concept : Self-stabilization .
There will be a ring of computers working simultaneously . If any one of
them fails , others will continue to handle tasks requested from those
systems . Such a ring may utilize load-balancing to distribute work load to
currently working computers .
http://en.wikipedia.org/wiki/Self-stabilization
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.88.4096&rep=rep1&type=pdf
http://search.yahoo.com/search;_ylt=A03uoRZ7YbpLwjUB8W.bvZx4?p=self+stabilization&toggle=1&cop=mss&ei=UTF-8&fr=yfp-t-701
http://en.wikipedia.org/wiki/Load_balancing_%28computing%29
http://en.wikipedia.org/wiki/Category:Fault_tolerance
http://en.wikipedia.org/wiki/Category:Reliability_engineering
http://en.wikipedia.org/wiki/Category:Distributed_computing
Actually , FreeBSD itself is an operating system : It manages single
computers . You need other software to handle distributed computing . Such a
language among others is
www.mozart-oz.org
*Mozart is available as a FreeBSD port.
*Thank you very much .
Mehmet Erol Sanliturk
Chuck Swiger
> W dniu 10-04-05 22:43, jfa...@goldsword.com pisze:
>> Quoting Maciej Jan Broniarz <gau...@gausus.net>:
>> So first you have to define your workload, then define what errors you
>> must avoid or allow, and then define how to deal with failures, errors,
>> etc.
>> Then you can start talking about High Availability vs. level of Fault
>> tolerance, vs. ....
>
> Let's say i need to run a few php/sql based web sites and I would like to maintain uptime of about 99,99% per month. No matter how good the hardware - it will always fail at some time. My goal is to build a system, that can maintain that uptime.
You're attempting to move from ~1 hour of downtime per month to ~1 hour of downtime per year, or less than 5 minutes per month. To begin with, you must implement adequate monitoring to detect, notify, and track service outages (ie, Nagios, BigBrother, commercial test services like SiteScope, etc), and you need a 24/7/365 team available to immediately respond to pages/email/etc to minimize outage duration.
With 168 hours a week divided by nominal 40-hour workweek, that needs a team of 4.2 people, and also implies the cost of downtime per hour for the system should be higher than about 35K per hour to justify keeping such a team available. (Four people can do it with one working an extra 8-hour shift per week; however, at least local to me, California state law mandates that people on call for pager duty must be paid hourly overtime if they are expected to respond to issues.)
> From what You say I need some level of HA system, to maintain the required uptime.
>
> So, as I've said earlier (correct me, if I'm wrong) - the setup could look something like that:
>
> - 2 web servers with carp
> - 2 storage servers with on-line sync mechanism running
> - 2 mysql servers with on-line database replication
>
> (i'm skiping power and network issues at the moment).
Do you already know what your causes of downtime have been?
To my mind, you must consider all parts of the system, gather data, and resolve the problems which have the greatest downtime cost in a cost-effective fashion. The most common sources of machine failure are hard drives and PSUs; setting up RAID-1 mirrors for all machines and getting redundant power supplies on separate breakers should be a minimal starting point to avoid a likely single point of failure within a single machine.
Beyond that, the suggestion to have at least two of every component of the system is a right notion, but you need to include the glue which implements failover. That can be RFC-2391 style NAT to round-robin requests onto multiple webservers, but a hardware-based load-balancer (ServerIrons, Netscalers, etc) with aliveness or health checks will do a better job.
The better ones also support full redundancy, so you want a pair of those, or perhaps a pair of router/firewall/NAT boxes using VRRP or similar for the networking connectivity if you want to use software-based load-balancing.
Of course, all of this is assuming that the software is more reliable than the hardware it runs on. Good software can be, but it's more common for software failures, mistakes by admins, or the like to also contribute a lot to the system downtime.
Regards,
--
-Chuck
Aristedes Maniatis
We've recently set up a new colo facility with the following:
* dual ethernet links from our upstream
* dual HA pfSense (FreeBSD) boxes running haproxy to load balance incoming requests amongst live web servers
* dual switches
* 2 (or more) web (application) servers
* database
Until we get to 'database' everything is HA and quite easy to build and manage. Having a clustered database solution is expensive and beyond most smallish budgets. mysql and postgresql don't have anything available that is quite ready yet (IMO), so you'll need to be talking to the bigger (expensive) players about their clustered offerings.
You need redundancy within the database application across multiple machines. Possible, but not easy. You aren't going to be doing that completely within the operating system itself. DRDB sort of gets you there, but DRDB isn't synchronous with the database activity, so you might still lose data.
A cheaper option is to use master-slave replication (postgresql and mysql offer this) and CARP failover (just don't fail back!). But it hasn't been quite robust enough for my liking.
Ari Maniatis
--
-------------------------->
Aristedes Maniatis
ish
http://www.ish.com.au
Level 1, 30 Wilson Street Newtown 2042 Australia
phone +61 2 9550 5001 fax +61 2 9550 4001
GPG fingerprint CBFB 84B4 738D 4E87 5E5C 5EFA EF6A 7D2E 3E49 102A
Joshua Boyd
> on 07/04/2010 09:05 Aristedes Maniatis said the following:
> > Until we get to 'database' everything is HA and quite easy to build and
> > manage. Having a clustered database solution is expensive and beyond
> > most smallish budgets. mysql and postgresql don't have anything
> > available that is quite ready yet (IMO), so you'll need to be talking to
> > the bigger (expensive) players about their clustered offerings.
>
Master-master circular replication in mySQL probably fits the bill.
Master-slave requires a slave to promote itself to master, which can get
tricky.
>
> Out of curiosity: have you considered MySQL Cluster:
> http://en.wikipedia.org/wiki/MySQL_Cluster
> http://www.mysql.com/products/database/cluster/faq.html
>
> If yes, can you share your evaluation results?
> Thanks!
>
> --
> Andriy Gapon
> _______________________________________________
> freebsd...@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stabl...@freebsd.org"
>
--
Joshua Boyd
JBipNet
E-mail: boy...@jbip.net
Cell: (513) 375-0157
Andriy Gapon
> manage. Having a clustered database solution is expensive and beyond
> most smallish budgets. mysql and postgresql don't have anything
> available that is quite ready yet (IMO), so you'll need to be talking to
> the bigger (expensive) players about their clustered offerings.
Out of curiosity: have you considered MySQL Cluster:
Aristedes Maniatis
This is getting a bit offtopic to this list, but there are severe limitations with that product which make it unsuitable for my needs.
Matthew Seaman
Hash: SHA1
On 07/04/2010 08:15:16, Joshua Boyd wrote:
> On Wed, Apr 7, 2010 at 3:00 AM, Andriy Gapon <a...@icyb.net.ua> wrote:
>
>> on 07/04/2010 09:05 Aristedes Maniatis said the following:
>>> Until we get to 'database' everything is HA and quite easy to build and
>>> manage. Having a clustered database solution is expensive and beyond
>>> most smallish budgets. mysql and postgresql don't have anything
>>> available that is quite ready yet (IMO), so you'll need to be talking to
>>> the bigger (expensive) players about their clustered offerings.
>>
>
> Master-master circular replication in mySQL probably fits the bill.
> Master-slave requires a slave to promote itself to master, which can get
> tricky.
Although with master-master you need to be really careful to only use
one instance read-write at any one time. In theory you can design your
DB schema and SQL to work correctly with multiple masters; in practice
virtually no downloadable applications will work like this.
Also remember that MySQL replication runs in a single thread on a single
CPU core. It's quite easy for a busy DB master with plenty of CPU cores
to go so fast the replica can't actually keep up even if the replica
uses exactly the same hardware. Anyone who has a really good solution
to this problem is going to make themselves rich beyond the dreams of
avarice...
>> Out of curiosity: have you considered MySQL Cluster:
>> http://en.wikipedia.org/wiki/MySQL_Cluster
>> http://www.mysql.com/products/database/cluster/faq.html
>>
>> If yes, can you share your evaluation results?
>> Thanks!
You need at least three machines to build a MySQL cluster; preferably
more like 6 or 7. All of your data has to fit in RAM on those machines
and you need at least two copies of each item of data for resilience, so
don't bother trying this with anything other than a well populated 64bit
box. Also, if /all/ of your servers crash at the same time (power
problems tend to have this result) then your data has gone *poof* and
you'll be restoring from backup. You did remember to set up a regular
job to create snapshots of the clustered data didn't you?
Cluster tends to be slower than what you can achieve with straight MySQL
on the same hardware.
Cheers,
Matthew
- --
Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard
Flat 3
PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
Kent, CT11 9PW
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAku8NioACgkQ8Mjk52CukIxofQCePWmRADsM8r8+dLuA8MjDSL4U
3YEAoIxwGJrnQbzXABK9YU3Pc6+MVo8r
=tmTo
-----END PGP SIGNATURE-----
Paul Procacci
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 07/04/2010 08:15:16, Joshua Boyd wrote:
>
>> On Wed, Apr 7, 2010 at 3:00 AM, Andriy Gapon<a...@icyb.net.ua> wrote:
>>
>>
>>> on 07/04/2010 09:05 Aristedes Maniatis said the following:
>>>
>>>> Until we get to 'database' everything is HA and quite easy to build and
>>>> manage. Having a clustered database solution is expensive and beyond
>>>> most smallish budgets. mysql and postgresql don't have anything
>>>> available that is quite ready yet (IMO), so you'll need to be talking to
>>>> the bigger (expensive) players about their clustered offerings.
>>>>
>>>
>> Master-master circular replication in mySQL probably fits the bill.
>> Master-slave requires a slave to promote itself to master, which can get
>> tricky.
>>
> Although with master-master you need to be really careful to only use
> one instance read-write at any one time. In theory you can design your
> DB schema and SQL to work correctly with multiple masters; in practice
> virtually no downloadable applications will work like this.
>
> Also remember that MySQL replication runs in a single thread on a single
> CPU core. It's quite easy for a busy DB master with plenty of CPU cores
> to go so fast the replica can't actually keep up even if the replica
> uses exactly the same hardware. Anyone who has a really good solution
> to this problem is going to make themselves rich beyond the dreams of
> avarice...
>
I'd like to chime in and provide some ideas that we use here for some of
our clients which works quite well. In regards to master/slave, it's
quite trival to use a shared storage device like a 3par in combo' with
some heartbeat software. You have a floating ip that ping pongs back
and forth between two machines depending on the error conditions that
you set. As for the 3par itself, it has fiber to multiple switches so
losing that connection generally means a natural disaster taking it
offline. Multipath (teaming/bonding) configurations take care of switch
problems. I suppose this isn't really master/slave in the real sense,
rather two independant "masters" that can potentially read from the same
dataset, but does provide a very high level of redundancy without "data
duplication". As for shared storage devices, personally I'd only use
fault tolerant devices and stay away from an MSA for example.
In regards to master/master I agree with the above whole heartedly. We
have some clients that use it, but occasionally run into problems where
one of the "masters" falls so far behind another master, we're forced to
take it out of the pool. In our case, a quick and simple ipfw rule
blocking new connections tells our loadbalancer to quit sending new
connection here until it's caught up, but it's a royal PITA, but really
can't think of any other possible solution to this.
In addition, we evaluated commercial middleware appliances offered by a
couple of companies and while they all worked, they were all terribly slow.
Also in regards to MysqlCluster, I had tested its throughput about a
year ago, thinking that would be the way to go for future master/master
deployments. I don't remember the specific terminology so bear with me,
but as the number of "data stores" increased, the latency increased
along with it. On top of this, I was only able to get around 300 qps.
This is abyssmal compared to a standalone machine. Naturally this was
all on 8 core machines, tons of ram, and GigE interfaces. Needless to
say, we scratched that idea real quick. The cost to perfmance ratio
just wasn't worth it. Maybe this has changed since I've last visited
this, though I'm uncertain.
Anyways, I hope this provides some useful insight.
~Paul
This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/about-us-legal-email-disclaimer.htm for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.
Alban Hertroys
> Until we get to 'database' everything is HA and quite easy to build and manage. Having a clustered database solution is expensive and beyond most smallish budgets. mysql and postgresql don't have anything available that is quite ready yet (IMO), so you'll need to be talking to the bigger (expensive) players about their clustered offerings.
There are quite a few replication projects for postgres, aren't you writing them off a bit too easily? I know that for example the guys from Skype wrote and maintain one of them, I imagine they'd be quite concerned about HA. And there are a number more (see: http://wiki.postgresql.org/wiki/Replication,_Clustering,_and_Connection_Pooling).
I suggest that you poll their mailing list with your problem/requirements if you didn't already. They're usually rather helpful and will certainly confirm if there's no solution to your situation (yet).
Alban Hertroys
--
If you can't see the forest for the trees,
cut the trees and you'll see there is no forest.
!DSPAM:74,4bbc600010411126416115!
Marek 'Buki' Kozlovský
[snip]
> >> Out of curiosity: have you considered MySQL Cluster:
> >> http://en.wikipedia.org/wiki/MySQL_Cluster
> >> http://www.mysql.com/products/database/cluster/faq.html
> >>
> >> If yes, can you share your evaluation results?
> >> Thanks!
>
> You need at least three machines to build a MySQL cluster; preferably
> more like 6 or 7. All of your data has to fit in RAM on those machines
> and you need at least two copies of each item of data for resilience, so
> don't bother trying this with anything other than a well populated 64bit
> box. Also, if /all/ of your servers crash at the same time (power
> problems tend to have this result) then your data has gone *poof* and
> you'll be restoring from backup. You did remember to set up a regular
> job to create snapshots of the clustered data didn't you?
two machines will suffice. Of course, prefferably more :)
Data no longer need to fit into memory, IIRC. Only indexes must.
We were running MySQL cluster in production few years back (with 4.x MySQL -
when the data had to fit in mem) and it was quite usable.
> Cluster tends to be slower than what you can achieve with straight MySQL
> on the same hardware.
unfortunately, I didn't perform any test.
> Cheers,
>
> Matthew
>
> --
> Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard
> Flat 3
> PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate
> Kent, CT11 9PW
more on the topic at http://www.mysql.com/products/database/cluster/
Regards,
Buki
--
PGP public key: http://dev.null.cz/buki.asc
/"\
\ / ASCII Ribbon Campaign
X Against HTML & Outlook Mail
/ \ http://www.thebackrow.net
Robert Joosten
> Let's say i need to run a few php/sql based web sites and I would like
> to maintain uptime of about 99,99% per month.
> From what You say I need some level of HA system, to maintain the
> required uptime.
Okay
> (i'm skiping (..) network issues at the moment).
Then you won't succeed. Four nines availabillity without regarding network
outages... sjeesh.
> Few people have told me about a setup with linux, drbd and heartbeat
> which offers them some level of HA. Has anyone tried anything similar on
> FreeBSD?
Did you actually *look* around for possible solutions ? I mean, I searched
with google and found geom + ggated solutions... Go figure.
Depending on your budget, I would build two 'systems', spread over two
coloc's. Use two squid/memcached caches, two backend servers with
http-daemons and php on it, two sql servers that replicate in realtime,
all (maybe exclude the sql servers from that.. depending on writes
figures) connected to a NAS. Having caches at the frontend, make sure you
fully understand http://www.mnot.net/cache_docs/ (for example, may more
resources are available).
Then, have that NAS replicate to the other coloc site (don't forget the
encryption heh). Pay attention to the switched LAN behind it. Maybe
redundant switches would be a very clever investment.
At last, have 2 dns A/AAAA pointers to both sites with a reasonable
but short TTL that you can change right away. These autoritive dns servers
must be as redundant as well.
Now we're talking about resiliency. And money, so skip out all what proves
to be too expensive. I'm sure you already estimated how many dollars
unavailabillity costs you, so invest wisely. If this is a commercial
hosting exercise, buy stuff for the upcoming 36 months. After that,
redesign may be wise.
You wanna learn how other folks are doing this ? Have a look at wikipedia
! And also try to learn from their power outage few weeks ago :-/ You may
poke around at http://wikitech.wikimedia.org/view/Main_Page and get some
idea's.
Goodluck.
Robert