here's a gun. let's jump it.

[hymie!]

Friday 12:30pm -- Let's make a major update to replace one of our systems.
Monday 9:30am -- Well, nobody's complained. I guess the change was a
huge success. Let's start turning off the old system.

--hymie! http://lactose.homelinux.net/~hymie hy...@lactose.homelinux.net
-------------------------------------------------------------------------------

[LP]

On 2013-04-15, hymie! <hy...@lactose.homelinux.net> wrote:
> Friday 12:30pm -- Let's make a major update to replace one of our systems.
> Monday 9:30am -- Well, nobody's complained. I guess the change was a
> huge success. Let's start turning off the old system.

Fools!

Everone knows that major architectural changes to production systems are best
done at 16:30 on the Friday before you go on holiday for a fortnight, without
telling any of your colleagues that you've done anything.

Wait. I mean the opposite of that.

</comparatively recent experience>

-Paul
--
http://paulseward.com

[Lawns 'R' Us]

On 2013-04-17, Satya <sat...@satyaonline.cjb.net> wrote:
> Be bar pbhyq qb n aba-cebqhpgvba qrcybl ba Ghrfqnl ng 10nz ybpny naq
> lrg raq hc univat na bcrengbe va n qvssrerag gvzrmbar trg cntrq njnl
> sebz uvf yhapu ol sng-svatrevat, naq gura xvyyvat, gur qrcybl pbzznaq.

Erzvaqf zr bs na nyregvat flfgrz gung V unq gb qrny jvgu jura V jnf ng
Tvtnagvp Terra. Vg jnf pbasvtherq gb envfr n uvtu cevbevgl gvpxrg
jurarire cnegvphyne zrffntrf jrer ybttrq ol gur onpxhc flfgrz. Bar bs
gubfr zrffntrf jbhyq or ybttrq vs V dhrevrq gur flfgrz sbe n cebprff
gung unq nyernql svavfurq.

Shssvpr gb fnl gung bapr V ernyvfrq jung jnf tbvat ba, V dhvpxyl
erzbirq gung cnegvphyne zrffntr sebz gur yvfg. Tvira gur ohernhpengvp
cebprff vaibyirq va trggvat sbezny nccebiny gb qb fb, V whfg jrag
nurnq naq qvq vg, svthevat gung V pbhyq whfgvsl vg vs V tbg tevyyrq
nobhg vg yngre ba.

I arire qvq. Gur ahzore bs fchevbhf cubar pnyyf qebccrq fvtavsvpnagyl
nf n pbafrdhrapr. (Naq lrf, gurl jrer fchevbhf - gur qrfpevorq
pbaqvgvba jnf, naq vf, gur _bayl_ pbaqvgvba haqre juvpu gung zrffntr
jbhyq or ybttrq. Qrsvavgryl abg jbegul bs n uvtu cevbevgl gvpxrg.)

Yrf, vg'f gehr. V unir nofbyhgryl ab dhnyzf nobhg olcnffvat gur
cebprff jura vg'f oybbql shpxvat fghcvq - rira xabjvat gung vs V trg
vg jebat, vg'f zl wbo. V fnir gur pvephziragvba sbe jura V xabj V
jba'g trg vg jebat.

[Simon Smith]

In message <slrnkmspir...@invalid.hostname.does.not.exist.666.au>

Lawns 'R' Us <nob...@nowhere.example.com> wrote:

> On 2013-04-17, Satya <sat...@satyaonline.cjb.net> wrote:
> > Be bar pbhyq qb n aba-cebqhpgvba qrcybl ba Ghrfqnl ng 10nz ybpny naq
> > lrg raq hc univat na bcrengbe va n qvssrerag gvzrmbar trg cntrq njnl
> > sebz uvf yhapu ol sng-svatrevat, naq gura xvyyvat, gur qrcybl pbzznaq.
>

> Erzvaqf zr bs na nyregvat fl<snip>
>
> Shssvpr gb fnl gung ba<snip>

> I arire qvq. Gur<snip>

> Yrf, vg'f gehr. <snip>


'Suffice...'

'I...'

'Yes...'

Ooh, not drop-caps, but rot-caps. I quite like that.


--
Simon Smith

When emailing me, please use my preferred email address, which is on my web
site at http://www.simon-smith.org

[Lawns 'R' Us]

On 2013-04-22, Simon Smith <simon_sm...@zen.co.uk> wrote:
> In message <slrnkmspir...@invalid.hostname.does.not.exist.666.au>
> Lawns 'R' Us <nob...@nowhere.example.com> wrote:
>> Erzvaqf zr bs na nyregvat fl<snip>
>>
>> Shssvpr gb fnl gung ba<snip>
>
>> I arire qvq. Gur<snip>
>
>> Yrf, vg'f gehr. <snip>
>
>
> 'Suffice...'
>
> 'I...'
>
> 'Yes...'
>
> Ooh, not drop-caps, but rot-caps. I quite like that.

I'm going to blame it on lack of sleep. Or maybe it's just that the
machine uprising has begun. Yeah. That's it. Well, at least it means
that I don't have to worry about cleaning up this mess that others
have dumped into my lap ...

[apeiron]

On 2013-04-17, Lawns 'R' Us scribbled these curious markings:

> Erzvaqf zr bs na nyregvat flfgrz gung V unq gb qrny jvgu jura V jnf ng
> Tvtnagvp Terra. Vg jnf pbasvtherq gb envfr n uvtu cevbevgl gvpxrg
> jurarire cnegvphyne zrffntrf jrer ybttrq ol gur onpxhc flfgrz. Bar bs
> gubfr zrffntrf jbhyq or ybttrq vs V dhrevrq gur flfgrz sbe n cebprff
> gung unq nyernql svavfurq.
>
> Shssvpr gb fnl gung bapr V ernyvfrq jung jnf tbvat ba, V dhvpxyl
> erzbirq gung cnegvphyne zrffntr sebz gur yvfg. Tvira gur ohernhpengvp
> cebprff vaibyirq va trggvat sbezny nccebiny gb qb fb, V whfg jrag
> nurnq naq qvq vg, svthevat gung V pbhyq whfgvsl vg vs V tbg tevyyrq
> nobhg vg yngre ba.

At $WORK we make $MONITORINGTOOL--actually, several tools. They work
reasonably well together, and I'm honestly impressed by how not-suck
the "as a service" bit is compared to other, similar tools. The details
of the tools aren't really important[0], but the context of what we do
is. It should not surprise anyone that, in addition to dev/admin
conslutting, we try to get clients to use our monitoring tool.

What the developers don't take into account is that the consumers of
their software are tools (and I don't mean software). I did some $UI to
a client service this weekend. Usual procedure: pause the alerts, etc.,
etc. Just as I was finishing up, a coworker texts me and asks if I'm
breaking the fooservice. I didn't think I was, as I was able to reach it
just fine. He showed me a separate alert for fooservice. Turns out there
were two separate alerts for foo (which my earlier search, and a search
I did just then, didn't find[1]...).

Of course, this is the client where it's uncommon for an employee to be
unable to add checks and alerts--that page other people. This, combined
with their recent propensity of drastically shrinking their technical
staff without knowledge transfer, means the admins are presently in the
unenviable position of being oncall for services no one knows anything
about. Thankfully, their shrinking tech staff means we'll be picking up
more of their dev work. Evidently their code is crap, and dev lead has
said he wants to rewrite it all. That works for me; it means I can call
our devs at 3AM instead of trying to reach the increasingly clueless
client. We're fortunate enough to have a culture where devs expect to be
called when things break.

Is this kind of dev accountability rare? It certainly seems to be. Most
devs I talk to say they'd refuse to take a job if it required them to be
on-call. To me, that just means they don't want to be held accountable
for their shit code.

[0] Except for one little detail; see below.

[1] Search being schizophrenic is a known issue. I just set a reminder to
cattleprod the cats responsible. The two leads are either on vacation or
jetsetting, though, so I'm not sure how much progress will be made.

--
apeiron

[Joe Zeff]

On Sun, 28 Apr 2013 16:30:41 +0000, apeiron wrote:

> Is this kind of dev accountability rare? It certainly seems to be. Most
> devs I talk to say they'd refuse to take a job if it required them to be
> on-call. To me, that just means they don't want to be held accountable
> for their shit code.

Just because they're not willing to be called out at 3 AM when something
breaks doesn't mean they're not being held accountable as long as they
check their messages as soon as they get to work in the morning and deal
with whatever came up during the night in a prompt and timely manner.
(Now, if the contract with the client says that they'll be on-call,
that's a different story.)

--
Joe Zeff -- The Guy With The Sideburns:
http://www.zeff.us http://www.lasfs.info
I just have a very broad definition of "normal."

[Alan J Rosenthal]

apeiron <ape...@pobox.com> writes:
>Most devs I talk to say they'd refuse to take a job if it required them to
>be on-call. To me, that just means they don't want to be held accountable
>for their shit code.

To me, that just means that they don't want to be on-call.
Being on call sucks.

[Steve VanDevender]

Satya <sat...@satyaonline.cjb.net> writes:

> Gur ynfg, naq bayl, gvzr V tbg pnyyrq ng 3nz (yvgrenyyl) jnf orpnhfr n qngn
> frg gung V chfurq gb cebq ng 10cz[1] fgnegrq pnhfvat gur jro fvgr gb guebj
> uvture-guna-hfhny[0] 500f. Cebq rat cntrq ba-pnyy thl ng 3 jura vg orpnzr
> nccnerag gung gur reebe-ercbegvat jnfa'g orvat bireyl-cvpxl.

. . .

> [0] We prefer "usual" to equal zero, TYVM.
> [1] Maintenance window.

Ahem. Why would you rot that post (which has nothing that needs
rotting) and not rot its footnotes?

--
Steve VanDevender "I ride the big iron" http://hexadecimal.uoregon.edu/
ste...@hexadecimal.uoregon.edu PGP keyprint 4AD7AF61F0B9DE87 522902969C0A7EE8
Little things break, circuitry burns / Time flies while my little world turns
Every day comes, every day goes / 100 years and nobody shows -- Happy Rhodes

[apeiron]

On 2013-04-29, Alan J Rosenthal scribbled these curious markings:

Only if your stuff falls over more than it doesn't. As described
elsethread, achieving that is easier said than done. While there is no
silver bullet, I think regular ones would do just fine. Users may be
clueless, but I've not yet seen any clamoring for brains. Whether this
would be an improvement remains to be seen.

At least $CLIENT got rid of their PHB. My response to their lead SA when he
informed me of this may or may not have been "\o/". This was a man who was in
charge of operations, yet when the site was down, would send emails saying "the
site is down, please investigate" to our oncall address, and provide no further
details. I inquired for more details once during business hours when he
made a similar comment, and he responded "that should be enough". It
was more the way it was said than anything else that really got to me.

--
apeiron

[Peter H. Coffin]

Being on call sucks. Being on call for no additional compensation
sucks worse. Being on call for no compensation and no comprehension
that it's actually work, such that one's normal kinds of relaxation
are circumscribed, sucks even worse. Can't drink heavily, commit to
an expensive dinner out or evening of theater or Monsters of Rock
concert, pile onto the motorbike and head off into the wilds, go sailing
offshore, fly airplanes, etc. Why? Because the damned phone might go off
and you're supposed to answer it. It's a weekend in name only, or for
those who's lives are mild enough that their plans involve little more
than reading, sleeping, and maybe a little puttering around the garden.
Tyranny of the boring.

--
39. If I absolutely must ride into battle, I will certainly not ride at
the forefront of my Legions of Terror, nor will I seek out my
opposite number among his army.
--Peter Anspach's list of things to do as an Evil Overlord

[Peter Corlett]

apeiron <ape...@pobox.com> wrote:
[...]

> Is this kind of dev accountability rare? It certainly seems to be. Most devs
> I talk to say they'd refuse to take a job if it required them to be on-call.
> To me, that just means they don't want to be held accountable for their shit
> code.

It is also quite possible that they've been burned by being on-call before, and
it's actually a problem with management. For example, it could be one or more
of these:

* On-call is unpaid, but the person on-call is not allowed to drink or do stuff
that makes them uncontactable or unable to respond quickly, so it's
effectively a pay cut.

* Certain self-important people (i.e. management) treat the on-call number as a
general support number because they don't want to wait until the morning to
get their password reset or whatever.

* Management expect the on-call person to be on-call 24/7, and still turn up at
8am and do a solid twelve hours of work despite having been called out at
stupid o'clock.

* The code *is* shit, but that's due to management diktat, demanding quick
fixes and not allowing time to be spent on testing, insisting that CUC and
ZlFDY are used, buying substandard hardware to use as servers, and so on.

[Shmuel Metz]

In <kljirh$5kb$1...@dont-email.me>, on 04/28/2013

at 04:30 PM, apeiron <ape...@pobox.com> said:

>Is this kind of dev accountability rare?

In the m/f community it varies wildly by shop. In some shops not only
don't they require the developers to be on call, they prohibit the
developers from touching the production version. It's not my dog.

Personally, I'd rather be called at 0 dark hundred than have to clean
up after someone else's incorrect fix.

--
Shmuel (Seymour J.) Metz <http://patriot.net/~shmuel> ISO position
Reply to domain Patriot dot net user shmuel+bspfh to contact me.
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

[Shmuel Metz]

In <kll0g7$36q$1...@dont-email.me>, on 04/29/2013

at 05:29 AM, apeiron <ape...@pobox.com> said:

>On 2013-04-29, Alan J Rosenthal scribbled these curious markings: >
>apeiron <ape...@pobox.com> writes:
>>> Most devs I talk to say they'd refuse to take a job if it required them to
>>> be on-call. To me, that just means they don't want to be held accountable
>>> for their shit code.
>>
>> To me, that just means that they don't want to be on-call.
>> Being on call sucks.

>Only if your stuff falls over more than it doesn't.

I wish that I lived in your universe, where you only get paged if

1. There really is a bug

2. It really is in your code

3. You are the one on call.

It sounds like a lovely universe.

[John Burnham]

On Mon, 29 Apr 2013 13:24:15 -0400, Jim wrote:

>
> I've spent the last 10+ years trying to get our devs to understand that
> a 500 error means they did something wrong [1]. If you handle errors in
> the application, nothing should result in a 500 error that the user
> sees.
>
> [1] Haven't had a lot of luck. Getting tired of programmers who think
> error handling is for someone else to implement.

Yeah, I know the feeling. We had a "wunderkind" contractor that I thought
was a waste of matter. Despite supposedly being a Java expert, he seems
not to have heard of exception handling. A webservice he wrote spews 500
errors like no-one's business - to the point that I stuck a script in
that emails people every time it does in a vain attempt to point out how
fucked this is and how someone should bloody well fix it. There is a
happy end to this tale. After I exploded at the Director here about this
guy, his contract is not being renewed and they're hiring proper staff
who have been interviewed by people whose technical opinions I trust.

J

[David Taylor]

On 2013-04-29, Shmuel Metz <spam...@library.lspace.org.invalid> wrote:
>
> I wish that I lived in your universe, where you only get paged if
>
> 1. There really is a bug
>
> 2. It really is in your code
>
> 3. You are the one on call.
>
> It sounds like a lovely universe.

Well, points two and three seem somewhat contradictory.

--
David Taylor

[Joe Zeff]

On Tue, 30 Apr 2013 09:16:28 +0100, John Burnham wrote:

> After I exploded at the Director here about this guy, his contract is
> not being renewed and they're hiring proper staff who have been
> interviewed by people whose technical opinions I trust.

I think it's safe to say that the former skriptkiddie was hired by
somebody whose technical opinions you could trust. Trust to be wrong,
that is.

--
Joe Zeff -- The Guy With The Sideburns:
http://www.zeff.us http://www.lasfs.info

Hah! If they were going to do that, they'd be just as likely to buy some
extra rope to keep the pigs from getting out of their rooftop aviary.

[Wojciech Derechowski]

On Tue, 30 Apr 2013 17:42:31 +0000, Joe Zeff wrote:
> On Tue, 30 Apr 2013 09:16:28 +0100, John Burnham wrote:
>
>> After I exploded at the Director here about this guy, his contract is
>> not being renewed and they're hiring proper staff who have been
>> interviewed by people whose technical opinions I trust.
>
> I think it's safe to say that the former skriptkiddie was hired by
> somebody whose technical opinions you could trust. Trust to be wrong,
> that is.
>

And consistent. Imagine what would happen if they developed an intermittent
fault and were sometimes right. Then you couldn't trust yourself or at the
very least you would ask too many questions. Then *you* make a mistake and
suddenly everybody and their dog is trying to bite your head off.

WD
--
Who is Entscheidungs and what is his problem?

[Shmuel Metz]

In <slrnknv69r.aq0...@webber.yadt.co.uk>, on 04/30/2013

Not even close. The operators are not supposed to know who wrote what;
they're supposed to know what the duty roster[1] is for the shift. If
you're on call then it's up to you to know how to handle problems.
That might include calling the author[2], but usually it means writing
a temporary fix and updating the trouble report so that a permanent
fix can be developed, documented and tested. BTDT,GTTS.

[1] The people listed normally have enough familiarity with the
code to determine whether there really is a problem, to
decide whether it can wait untill the next day, to know whether
it is prudent to attempt repair and to know whom to contact if
it is beyond their expertise.

[2] The actual author, not some random programmer who wrote some
unrelated routines.