ripco status for now (8/5/08)
Bruce Esquibel
Looks like we were hit by lightning besides the power outage.
The shell box is toast, not sure what can be done about that yet.
The phone system is out, any calls to the office is just going to voicemail
and those have to picked up.
All the managed (snmp) ethernet switches were blown.
The dialup is not working, right now I don't have a clue why.
I'm too tired to fuck around with this anymore tonight, will figure out what
to do with the shell box and get the dialup repaired in the morning.
Most of the backup stuff I had around for "things like this" is in service
now, so we bees out of backup things. Another storm like the one we had last
night and you can find me and Melissa in Brazil.
No joke, the rest of the building didn't fair well either, their voip phone
service is out, their router/switch for the local network and at two
machines were all DOA when the power came back up.
Most of our stuff uses replaceable power modules but there wasn't enough to
go around, thus no shell box. Will see about getting some fed-ex'ed out
tomorrow.
-bruce
b...@ripco.com
Bruce Esquibel
> The dialup is not working, right now I don't have a clue why.
I figured out what was wrong with the dialup around 4:30am so that is
working at least for ppp.
Shell box is next but this might take a while.
-bruce
b...@ripco.com
Bruce Esquibel
> Well, at least something's working: I just got my urgent "Subscription
> expires 8/15/2008" notice.
Yeah that always get fixed first.
I got one idea for the shell box to get it online later this morning but is
just a theory right now. If it doesn't work, depending if I hear from the
guy in time, a trip to milwaukee is in order to pick up the stuff I need.
Or if anyone has any (2, perfer 3) Sun 300-1260 power modules laying around,
let me know.
-bruce
b...@ripco.com
jamie
Bruce Esquibel <b...@e4500.ripco.com> wrote:
>Bruce Esquibel <b...@e4500.ripco.com> wrote:
>
>> The dialup is not working, right now I don't have a clue why.
>
>
>I figured out what was wrong with the dialup around 4:30am so that is
>working at least for ppp.
Thanks, Bruce. I hope the 4:30 am doesn't mean you were up all night.
I was shocked and awed when I idlely tried to dial up this while
munching breakfast and it actually connected. (It took about 5 minutes
before I realized I hadn't gotten an error message.)
Jamie
Bruce Esquibel
> Thanks, Bruce. I hope the 4:30 am doesn't mean you were up all night.
> I was shocked and awed when I idlely tried to dial up this while
> munching breakfast and it actually connected. (It took about 5 minutes
> before I realized I hadn't gotten an error message.)
No, but woke up around 4 because when things like this happen with the power
and lightning, gets me agitated and can't sleep much. I really do take the
systems performance personally but it seems like all we learned from past
events like this one, ripco still could of done a better job restoring
service.
I really hate the idea of moving into a data or colo facility but it's
coming down to that, there only are a few local providers left around here
so exchanging services (over exchanging cash) is thin.
As far as Scotts comment about the generator backup, yeah we should have it
but don't, with reason.
About 4 years ago we had an extended outage, no reason, power just went out
and stayed out for 12-14 hours. After that I contacted some outfit in the
burbs to come in and do a survey/estimate about having our own power system
installed.
I don't remember all the specific numbers but based on the hardware and
cooling system, the natural gas generators wouldn't work with the feed we
have (i dunno, 1/2" pipe?). The largest they had for that delivery flow
would run either but not both. Since this type of outage always seems to
occur in the summer, no point just keeping the hardware up. When we've had
problems with the a/c, this room gets into the upper 90's in no time. After
the air temp reaches around 105, the machines will start shutting down by
themselves.
Anyway, this left the deisel ones and that was a kick in the pants. Even
though there is like a 15 pump gas station out the front door from us, being
the above ground tank would have to go in back, we're too close to a
residential building to put anything in larger than 20 gallons. That
wouldn't last very long (think it was 6 or 7 gallons an hour). Since the
station across the street doesn't sell it anyway, would be a headache to
transport in 5 gallon containers every few hours.
All of that left me a choice of either moving to more of an industrial area
or paying the gas company to install a larger feed. Either would of been a
small fortune, much more than the generators/transfer switches would of
cost. Plus with the gas around here going for $1.50 a cubit foot, I bet the
outage this week would of cost me thousands.
So we live off batteries, normal day-to-day stuff it's fine, really does
cover the normal spot brown/blackouts that happen so it's just practical
without costing a fortune.
I know we're going to lose some customers over this but being we lose them
when everything goes right, it's just the cost of doing business. I do like
the first ones on the phone are the ones who getting them to pay is like
pulling teeth.
What I mean about the cost of doing business is "shit happens". There was an
article a couple years ago in one of the paper printed web hosting magazines
(yeah they actually print them) about either iPlanet or Hostgator, some big
hosting company in Houston.
Short version of the story, two years ago they spend a bundle putting in
this 3 megawatt backup system, enough to keep 3 large buildings running with
ten's of thousands of machines (and cooling).
Guess the karma train was rolling by and they lost power from their "grid"
(fed by two utility companies) and the backup system kicked on full power.
Seems like someone didn't do their homework on the difference between 240v
and 120v, blew out one building totally and half the machines in another.
I have to give them credit for getting "most" of their customers back up 3
days later but after that kind of investment going up in smoke literally, I
just don't think there are any perfect plans, just practical ones.
So I'll stick with my cheesy battery backup and just do without when it
happens.
-bruce
b...@ripco.com
Bruce Esquibel
> The temporary shell seems to work, but I can't port forward (I can't
> port forward to the news servers). Will port forwarding be available on
> this temporary box?
It should be working now but if you are already connected, you have to
disconnect and establish a new ssh session for the changes to take place.
Was just an oversight in the config file.
-bruce
b...@ripco.com
KK
Bruce Esquibel <b...@e4500.ripco.com> wrote:
well it's working now
Cydrome Leader
> jamie <ja...@ftupet.com> wrote:
>
>> Thanks, Bruce. I hope the 4:30 am doesn't mean you were up all night.
>> I was shocked and awed when I idlely tried to dial up this while
>> munching breakfast and it actually connected. (It took about 5 minutes
>> before I realized I hadn't gotten an error message.)
>
>
> No, but woke up around 4 because when things like this happen with the power
> and lightning, gets me agitated and can't sleep much. I really do take the
> systems performance personally but it seems like all we learned from past
> events like this one, ripco still could of done a better job restoring
> service.
>
> I really hate the idea of moving into a data or colo facility but it's
> coming down to that, there only are a few local providers left around here
> so exchanging services (over exchanging cash) is thin.
coloing is awesome, if you pick a good place.
power and cooling are no longer your problem. Neither is where you get
upstream connections from. You don't need to rent an office anymore
either, run out of a mailbox or something.
The downside would be, from looking at the part numbers of those sun power
supplies is you have some huge ass equipment. Toss that shit out and get
and get machines that don't come on casters and you can then fit stuff in
a cabinet somewhere. I'm not sure how you terminate calls, but I'm
guessing that stuff will waste the most space even if you modeernize a
bit.
I clinged to a shitty "datecenter" in an office building way to long. Once
you leave and go somewhere real, there's no going back.
Bruce Esquibel
> I clinged to a shitty "datecenter" in an office building way to long. Once
> you leave and go somewhere real, there's no going back.
Yeah but keep in mind you were probably still a gleam in your daddy's eye
when we started.
I have no doubt anymore doing what you are doing is the way to go but just
from experience it's not a perfect solution either. If anything with the way
Ripco is ran now, it's fully under our control. This week was bad, probably
the worst we had to deal with but once the initial panic went away we were
able to restore most of the operations.
These days there are so many hosting facilities (the term ISP seems old hat
now) ran by teenagers from their moms basement using a $50 a month leased
server, tossing like CentOS and cPanel on it, then offering $3.95 services.
When those guys go down, the majority of the time they just disappear or put
the blame anywhere else except on them. What do they have to lose, even if
they get behind the $50 a month server lease, they just go somewhere else,
open up another hosting service with a different name and start over again.
Don't get me wrong, a lot of the customers don't give a shit either, they
sort of figure for $3.95 a month it's not going to be the most reliable
service and when things go wrong, they just change to another $3.95
provider.
Anyway I'm not trying to start anything, just my opinion of how a business
should be ran. It's obvious after 15 years of ripco.com we still don't have
our shit together yet either but this week was a good lesson to change
things, suck up to the fact that things were not handled as well as they
could of been but move on to improve things.
Right now I don't know what that means. Maybe we'll lease all new hardware,
maybe we'll lease a rack and move our stuff into it, maybe we'll just find a
new office where we can install a proper backup power system and call it
quits on Clybourn.
I'm just saying if there are 101 ways to skin a cat, 99 of them are boarding
on being irresponsible with the way the "ISP" business is being done these
days.
-bruce
b...@ripco.com
Cydrome Leader
> Cydrome Leader <pres...@mungepanix.com> wrote:
>
>> I clinged to a shitty "datecenter" in an office building way to long. Once
>> you leave and go somewhere real, there's no going back.
>
>
> Yeah but keep in mind you were probably still a gleam in your daddy's eye
> when we started.
I do recall being amused that you were running AT&T SV with a kernel from
jun or jul 1992 before you went solaris.
> I have no doubt anymore doing what you are doing is the way to go but just
> from experience it's not a perfect solution either. If anything with the way
> Ripco is ran now, it's fully under our control. This week was bad, probably
> the worst we had to deal with but once the initial panic went away we were
> able to restore most of the operations.
>
> These days there are so many hosting facilities (the term ISP seems old hat
> now) ran by teenagers from their moms basement using a $50 a month leased
> server, tossing like CentOS and cPanel on it, then offering $3.95 services.
That cpanel shit on some white box crap doesn't count to me at all either.
more than 56% uptime is sort of a requirement these days.
The only thing I still can't pull off is tranferring my busines license to
the physical address of where I keep the equipment.
That would save a few bucks on having a "lease" at a commercial address to
keep the city happy for the time being.
Still though, power, cooling and security aren't things I have to think
about anymore. All that stuff was payed for many times over by the bigger
customers.
Backups are still a nightmare though.
Chicago Paddling-Fishing
Hey, there is a building in Schaumburg which might still be for sale (it's
empty now). It's got 3 * 15,000 gallon underground diesel tanks, four V12
caterpiller generators, two wells to keep water cooled CPU's happy and a
room stuffed with shelves loaded with car batteries... two different Edison
inputs from 2 different stations... power is converted into 12 volts and run
into the battery room and then power comes out from the battery room so you
don't get a brownout when the cats fire up (the cats take a while to warm up
but fire up automagically upon power failure)... I don't know how old most of
the hardware is but the building was retrofitted with the additional fuel
tank, generator and well in 1999 to ensure it could operate a minumum of 3
months without outside assistance during y2k.
Only bad thing is the building is really poorly constructed, many of the
office doors don't shut tight because of angle issues and when the diesels
are running, the scent of diesel permeates the building...
Might be cheap with real estate going the way it is...
--
John Nelson
-----------------------------------------------------------------------------
Chicago Area Paddling/Fishing Page
http://www.chicagopaddling.org http://www.chicagofishing.org
(A Non-Commercial Web Site: No Sponsors, No Paid Ads and Nothing to Sell)
Todd H.
>
> Hey, there is a building in Schaumburg which might still be for sale (it's
> empty now). It's got 3 * 15,000 gallon underground diesel tanks, four V12
> caterpiller generators, two wells to keep water cooled CPU's happy and a
> room stuffed with shelves loaded with car batteries... two different Edison
> inputs from 2 different stations... power is converted into 12 volts and run
> into the battery room and then power comes out from the battery room so you
> don't get a brownout when the cats fire up (the cats take a while to warm up
> but fire up automagically upon power failure)... I don't know how old most of
> the hardware is but the building was retrofitted with the additional fuel
> tank, generator and well in 1999 to ensure it could operate a minumum of 3
> months without outside assistance during y2k.
>
> Only bad thing is the building is really poorly constructed, many of the
> office doors don't shut tight because of angle issues and when the diesels
> are running, the scent of diesel permeates the building...
>
> Might be cheap with real estate going the way it is...
What road is this building on? Martingale?
--
Todd H.
http://toddh.net/
Chicago Paddling-Fishing
<snip>
>What road is this building on? Martingale?
Yep, and Corporate Crossing... big ugly building...
Todd H.
> Todd H. <T...@toddh.net> wrote:
> <snip>
>>What road is this building on? Martingale?
>
> Yep, and Corporate Crossing... big ugly building...
That's the one!
I used to work as a contractor at that building and the gensets and
battery room are impressive as hell. Redundant data and power
connectivity from two ordinal directions as well.
I never recall any diesel smell when the gens were running their usual
load tests. I do however remember when the hvac unit caught fire on
the roof in the winter of 03. I was glad to be coming in later that
day before having to evacuate. That was one hell of a ball of smoke:
High-rise evacuation planning pays off 900 safely leave
Schaumburg building as fire burns on roof
Date: February 15, 2003
Publication: Daily Herald
Something in their co-workers' voices Friday told employees at
Schaumburg's IBM building they weren't having just another fire drill.
As a black cloud of smoke billowed across the sky at 9 a.m., a blaze
atop the high-rise at 231 N. Martingale Road forced a mass evacuation
that sent up to 900 workers out into the cold.
An electrical short in a rooftop heating unit sparked the fire,
officials said. No one was injured, thanks largely to a Schaumburg...
Cydrome Leader
I wonder if the Winstar space in the loop is still for lease. It's been
empty for years, and last time I looked, came with a phone switch full of
dead cards from around the US.
Chicago Paddling-Fishing
>> Todd H. <T...@toddh.net> wrote:
>> <snip>
>>>What road is this building on? Martingale?
>>
>> Yep, and Corporate Crossing... big ugly building...
>That's the one!
>I used to work as a contractor at that building and the gensets and
>battery room are impressive as hell. Redundant data and power
>connectivity from two ordinal directions as well.
Yep, plus the well for CPU cooling... depending on when you were there, they
added an additional 15k gallon fuel tank and 1 more generator just before
y2k (ripped up part of the parking lot for weeks)... if you drive past it
now, someone has painted it to almost look nice... (almost). The generators
were very cool... I got facilities to take my scouts on a tour through there
and the data center floors... those V12's are monsterous... they turn on
automagically and he said if they turned on while we were in there to just
run for the door, because it was deafing...
That vent in the flowerbox had a single garage door like attachment to it
that he said would open automatically to vent the room to the street when they
were on... from looking at the outside, I thought those grates were hooked
up directly to intakes, but they just vented the entire room... awesome power
source in there...
I don't think the scouts were too impressed though, for them the big topic
was the tape library robots... they had one with a glass end and the kids
watched it for a while and kept asking "What if the robot gets out and kills
everyone"... if I recall the guy doing the tour said the thing was doing
about 30mph on that rail in there... dark track, single headlight, and a
bunch of folks issuing mount commands just to keep it hopping while we were
on the tour...
>I never recall any diesel smell when the gens were running their usual
>load tests. I do however remember when the hvac unit caught fire on
>the roof in the winter of 03. I was glad to be coming in later that
>day before having to evacuate. That was one hell of a ball of smoke:
It was a splendid sight indeed... up on the upper floors of the north
side we'd get the smell from the generators... you always knew when every
other Wednesday came around...
Course, even with all the cool stuff in there bad things happened sometimes...
Todd H.
> Todd H. <T...@toddh.net> wrote:
>>Chicago Paddling-Fishing <j...@shell3.ripco.com> writes:
>
>>> Todd H. <T...@toddh.net> wrote:
>>> <snip>
>>>>What road is this building on? Martingale?
>>>
>>load tests. I do however remember when the hvac unit caught fire on
>>the roof in the winter of 03. I was glad to be coming in later that
>>day before having to evacuate. That was one hell of a ball of smoke:
>
> It was a splendid sight indeed... up on the upper floors of the north
> side we'd get the smell from the generators... you always knew when every
> other Wednesday came around...
Interesting. I worked on teh top floor on the north side in
2002-2004ish and never experienced this. Lucky I guess!
But yeah, that place does have some physical plant. Lack of parking
garage in the winter sure sucked though. Almost as much as the
cafeteria.
Chicago Paddling-Fishing
>> Todd H. <T...@toddh.net> wrote:
>>>Chicago Paddling-Fishing <j...@shell3.ripco.com> writes:
>>
>>>> Todd H. <T...@toddh.net> wrote:
>>>> <snip>
>>>>>What road is this building on? Martingale?
>>>>
>>>load tests. I do however remember when the hvac unit caught fire on
>>>the roof in the winter of 03. I was glad to be coming in later that
>>>day before having to evacuate. That was one hell of a ball of smoke:
>>
>> It was a splendid sight indeed... up on the upper floors of the north
>> side we'd get the smell from the generators... you always knew when every
>> other Wednesday came around...
>Interesting. I worked on teh top floor on the north side in
>2002-2004ish and never experienced this. Lucky I guess!
Top floor? Training room or the lab? That sofa up there was in high demand
often times in the winter when it was snowing...
We were on 7 and it got the smell pretty good.
>But yeah, that place does have some physical plant. Lack of parking
>garage in the winter sure sucked though. Almost as much as the
>cafeteria.
Yep, during the space crunch in the 90's we were across the street in the
425 building for a while and it was quite nice with the garage and the
bridge, but that had it's own problems with women being attacked in the
garage and cars being stolen...
Similar issues in the old AT&T building in Oak Brook at 22nd and spring rd...
Folks who are up to no good tend to favor parking garages over wide open
parking lots...
Plus, with an open lot, those of us with kayaks or bikes on our roofs have
no worries about hitting the ceiling...
One thing I always liked in the cafeteria was the soup... course it might have
just been campbells, but it was always pretty good...
Cydrome Leader
> Todd H. <T...@toddh.net> wrote:
>>Chicago Paddling-Fishing <j...@shell3.ripco.com> writes:
>
>>> Todd H. <T...@toddh.net> wrote:
>>> <snip>
>>>>What road is this building on? Martingale?
>>>
>>> Yep, and Corporate Crossing... big ugly building...
>
>>That's the one!
>
>>I used to work as a contractor at that building and the gensets and
>>battery room are impressive as hell. Redundant data and power
>>connectivity from two ordinal directions as well.
>
> Yep, plus the well for CPU cooling... depending on when you were there, they
The well is probably for the sprinkler system. In places with real fire
codes, you have to leave the bulding if you lose water the sprinklers.
There are some big deal telco buildings in the loop area with their own
wells so nobody has to evacuate if they lose city water pressure.
In the city, tap water cooled AC units were banned long ago, although some
places still have water cooled stuff like that and freezers etc.
Chicago Paddling-Fishing
>Chicago Paddling-Fishing <j...@shell3.ripco.com> wrote:
>> Todd H. <T...@toddh.net> wrote:
>>>Chicago Paddling-Fishing <j...@shell3.ripco.com> writes:
>>
>>>> Todd H. <T...@toddh.net> wrote:
>>>> <snip>
>>>>>What road is this building on? Martingale?
>>>>
>>>> Yep, and Corporate Crossing... big ugly building...
>>
>>>That's the one!
>>
>>>I used to work as a contractor at that building and the gensets and
>>>battery room are impressive as hell. Redundant data and power
>>>connectivity from two ordinal directions as well.
>>
>> Yep, plus the well for CPU cooling... depending on when you were there, they
>The well is probably for the sprinkler system. In places with real fire
>codes, you have to leave the bulding if you lose water the sprinklers.
The well is there as a general supply. One of the primary efforts around y2k
was to ensure that the building had enough supplies on hand for 3 months of
"interruption" (that's why they dug up the parking lot and put in the
additional 15k gallon tank, so they'd have enough fuel for 3 months on site).
A lot of businesses made a killing on y2k... the 2001 recession should have
been no surprise due to the level of spending that preceeded it...
>There are some big deal telco buildings in the loop area with their own
>wells so nobody has to evacuate if they lose city water pressure.
At one time that building was home to the largest data network in the world,
but not anymore (course, at one time SNA was hip ;-)... the control room
looked straight out of the war games movie set with the light up wall maps.
>In the city, tap water cooled AC units were banned long ago, although some
>places still have water cooled stuff like that and freezers etc.
When I was assigned to American National Bank in the early 80's the first
thing that caught your eye walking into the data center was a green garden
hose that ran from the water fountain over to the 3032 (or was it a 3031)
CPU, hopefully they have done some upgrades as those were old then...
They never did get that whole water cooled CPU business figured out during
the years I was assigned there... the hose was always in view...