Deploying patches to running web apps
Jonathon McKitrick
that should not be stopped and restarted?
Petter Gustad
> What are some proven approaches to deploying patches to applications
> that should not be stopped and restarted?
In Allegroserve you can just compile the new funcion in the REPL or
simply typing C-C C-C inside the function in Emacs if you're using
SLIME. Also see news:87mzhlwd...@parish.home.gustad.com
Petter
--
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing on usenet and in e-mail?
Christophe Rhodes
> "Jonathon McKitrick" <j_mck...@bigfoot.com> writes:
>
>> What are some proven approaches to deploying patches to applications
>> that should not be stopped and restarted?
>
> In Allegroserve you can just compile the new funcion in the REPL or
> simply typing C-C C-C inside the function in Emacs if you're using
> SLIME. Also see news:87mzhlwd...@parish.home.gustad.com
This is the traditional answer (not just in Allegroserve, but in most
lisp applications), but sometimes I wonder. Consider
(defun foo (a)
(bar (frob a)))
Now suppose that you need to change both bar and frob: say you need to
return more information to bar frob (an extra element in a list, say).
How do you do this safely, without stopping the application? I don't
think it's as simple as typing C-c C-c twice, or compiling "the new
function" in the repl, because doing so introduces inconsistencies in
code which might be running -- it would be bad if an application
thread is running in the old FROB when you compile the new BAR.
Christophe
Kenny Tilton
> Petter Gustad <newsma...@gustad.com> writes:
>
>
>>"Jonathon McKitrick" <j_mck...@bigfoot.com> writes:
>>
>>
>>>What are some proven approaches to deploying patches to applications
>>>that should not be stopped and restarted?
>>
>>In Allegroserve you can just compile the new funcion in the REPL or
>>simply typing C-C C-C inside the function in Emacs if you're using
>>SLIME. Also see news:87mzhlwd...@parish.home.gustad.com
>
>
> This is the traditional answer (not just in Allegroserve, but in most
> lisp applications), but sometimes I wonder. Consider
>
> (defun foo (a)
> (bar (frob a)))
>
> Now suppose that you need to change both bar and frob: say you need to
> return more information to bar frob (an extra element in a list, say).
I sense a missing preposition, perhaps in "to bar frob".
Anyway, if you are that worried about it, first introduce a bar upwards
compatible with the change you plan for frob. then change frob.
kenny
Petter Gustad
> This is the traditional answer (not just in Allegroserve, but in most
> lisp applications), but sometimes I wonder. Consider
>
> (defun foo (a)
> (bar (frob a)))
>
> Now suppose that you need to change both bar and frob: say you need to
> return more information to bar frob (an extra element in a list, say).
> How do you do this safely, without stopping the application? I don't
> think it's as simple as typing C-c C-c twice, or compiling "the new
> function" in the repl, because doing so introduces inconsistencies in
> code which might be running -- it would be bad if an application
> thread is running in the old FROB when you compile the new BAR.
Either there must be support for atomic compilation in the
implementation (i.e. in compile-file) or you have to handle this
yourself.
You could:
1) compile the old bar function as old-bar (or some unique name)
2) make a new bar function which will check the length of the argument
list and dispatch old-bar in the shorter case, compile the new bar
3) compile the new frob
4) remove the dispatch of old-bar from bar and recompile
5) remove the old-bar symbol
You could also use conditions to detect the call to old-bar. Of course
there are zillions of variants of this problem, not only the length of
the argument list, but different types, new slots in classes, etc. This
can get complicated as the number of functions and changes increase.
On a critical application you could test the patch on a different
system before you deploy the patch in the real world.
There probably exists lots of papers on this and similar issues.
People with extensive experience from systems like these can give a
much better and general answer than mine.
Christophe Rhodes
> Christophe Rhodes wrote:
>> Now suppose that you need to change both bar and frob: say you need
>> to
>> return more information to bar frob (an extra element in a list, say).
>
> I sense a missing preposition, perhaps in "to bar frob".
Quite right -- "to bar from frob". Metasyntactic variable overload
:-)
> Anyway, if you are that worried about it, first introduce a bar
> upwards compatible with the change you plan for frob. then change frob.
Right. I suppose the original solution (C-c C-c on changed stuff, in
some order) is proven in the sense that that's what most of us
probably do (i.e. at least it's tested); on the other hand, it's not
"proven" in any kind of formal sense (no proof of correctness exists,
because it's fairly easy to see that it can't possibly work in all
cases, and more care is needed). It depends on the needs of the OP
which version of "proven" solutions he wants...
Christophe
John Thingstad
Not to mention update of macroes..
ASDF..
--
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/
Thomas A. Russ
This is certainly true, and to some extent you can't really totally
avoid the problem unless you can always generate backward-compatible
versions of the updated functions.
You can perhaps limit the potential for some of these interactions by
loading a compiled patch file inside a WITHOUT-INTERRUPTS form, so that
you don't have another thread using only part of the update. But there
is still the problem of having the function definition change while you
are in the middle of executing one of the old functions, and before it
has done the name lookup to get the function code which now changes.
That is an unfortunate consequence of having full-time dynamic linking
of functions as part of the language. Unless there were a way of
forcing a static link during compilation, I don't see a way around it,
though.
--
Thomas A. Russ, USC/Information Sciences Institute
Paul Wallich
If you designed for this from the beginning, wouldn't you have something
like a versioning setup, probably with appropriate macrology or
packaging (depending on how you generate/call the code for a particular
session), so that stuff currently executing would see the old
definitions, and any new page/session/whatever that got spawned would
see the new ones? Sure, there would be some indirection overhead, at the
very least until all the old-definition sessions have finished
executing, but that's unlikely to be a bottleneck.
There would probably be a limit to just how radically you could change
the structure without needing a brief no-new-sessions interval, but
that's probably acceptable.
paul
Christophe Rhodes
> You can perhaps limit the potential for some of these interactions by
> loading a compiled patch file inside a WITHOUT-INTERRUPTS form, so that
> you don't have another thread using only part of the update.
Ahaha -- are you one of these people who thinks that (a)
WITHOUT-INTERRUPTS means WITHOUT-SCHEDULING and (b) that
WITHOUT-SCHEDULING is implementable? :-)
> But there is still the problem of having the function definition
> change while you are in the middle of executing one of the old
> functions, and before it has done the name lookup to get the
> function code which now changes.
>
> That is an unfortunate consequence of having full-time dynamic
> linking of functions as part of the language. Unless there were a
> way of forcing a static link during compilation, I don't see a way
> around it, though.
For what it's worth, I might be tempted to use something like CMUCL's
block compilation at the file level, which has the potential to make
the granularity of updates sufficiently large. (This "block
compilation" is essentially a static link at some scope, usually file,
which rewrites all internal calls as local calls). Not that I have
such a high-availability application in the wild, in any case.
Christophe
senator
> (bar (frob a)))
Here another thought (well, not my thought): keyword arguments. To
quote Paul Graham:
"Rtml even depended heavily on keyword parameters, which up to that
time I had always considered one of the more dubious features of
Common Lisp. Because of the way Web-based software gets released,
you have to design the software so that it's easy to change. "
>From http://lib.store.yahoo.net/lib/paulgraham/bbnexcerpts.txt
Cheers.
Thomas F. Burdick
The missing nuance in the traditional answer is that you should be
doing this on your testing server, not directly in production. Once
you have your patch worked out, I bet most people just load the fasl
into the running server image, and hope that no one makes a request at
just the wrong moment.
On the other hand, CMUCL developers are quite used to bootstrapping
changes into their running system, so maybe they tend to do this
right. If the bootstrapping gets tricky, I personally punt and just
hot-swap the server itself.
--
/|_ .-----------------------.
,' .\ / | Free Mumia Abu-Jamal! |
,--' _,' | Abolish the racist |
/ / | death penalty! |
( -. | `-----------------------'
| ) |
(`-. '--.)
`. )----'
Jonathon McKitrick
Thomas F. Burdick wrote:
> The missing nuance in the traditional answer is that you should be
> doing this on your testing server, not directly in production. Once
> you have your patch worked out, I bet most people just load the fasl
> into the running server image, and hope that no one makes a request at
> just the wrong moment.
Is there a huge difference between loading fasls versus fetching new
source from cvs and loading that?
Thomas F. Burdick
I'd expect loading a fasl to be faster, but in principle there's no
difference. The traditional answer I was referring to above was
compiling functions one at a time, interactively. *That* is likely to
leave your server in an inconsistent state for significant periods of
time if you're not careful.
Tim X
> "Jonathon McKitrick" <j_mck...@bigfoot.com> writes:
>
> > Thomas F. Burdick wrote:
> > > The missing nuance in the traditional answer is that you should be
> > > doing this on your testing server, not directly in production. Once
> > > you have your patch worked out, I bet most people just load the fasl
> > > into the running server image, and hope that no one makes a request at
> > > just the wrong moment.
> >
> > Is there a huge difference between loading fasls versus fetching new
> > source from cvs and loading that?
>
> I'd expect loading a fasl to be faster, but in principle there's no
> difference. The traditional answer I was referring to above was
> compiling functions one at a time, interactively. *That* is likely to
> leave your server in an inconsistent state for significant periods of
> time if you're not careful.
>
I agree. I also wonder how critical an advantage it really is to be
able to change a production system 'on the fly', especially in this
day of growing use of clustered and load balanced systems. It strikes
me as just being one of those "wow look what I can do with CL" things
- impressive on one level, but not really that beneficial on another.
Tim
--
Tim Cross
The e-mail address on this message is FALSE (obviously!). My real e-mail is
to a company in Australia called rapttech and my login is tcross - if you
really need to send mail, you should be able to work it out!
Thomas F. Burdick
> t...@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
>
> > "Jonathon McKitrick" <j_mck...@bigfoot.com> writes:
> >
> > > Thomas F. Burdick wrote:
> > > > The missing nuance in the traditional answer is that you should be
> > > > doing this on your testing server, not directly in production. Once
> > > > you have your patch worked out, I bet most people just load the fasl
> > > > into the running server image, and hope that no one makes a request at
> > > > just the wrong moment.
> > >
> > > Is there a huge difference between loading fasls versus fetching new
> > > source from cvs and loading that?
> >
> > I'd expect loading a fasl to be faster, but in principle there's no
> > difference. The traditional answer I was referring to above was
> > compiling functions one at a time, interactively. *That* is likely to
> > leave your server in an inconsistent state for significant periods of
> > time if you're not careful.
> >
>
> I agree. I also wonder how critical an advantage it really is to be
> able to change a production system 'on the fly', especially in this
> day of growing use of clustered and load balanced systems. It strikes
> me as just being one of those "wow look what I can do with CL" things
> - impressive on one level, but not really that beneficial on another.
It's certainly helpful for development, which you should be doing on a
test server anyhow.
Espen Vestre
> I agree. I also wonder how critical an advantage it really is to be
> able to change a production system 'on the fly', especially in this
> day of growing use of clustered and load balanced systems.
Well, even in highly clustered and load balanced system, you usually
fix connections to one machine if the protocol involves a lot of
statekeeping on the server side. Of course it's /possible/ to
implement hot standby solutions even in such cases (the hot standby
protocol for OpenBSDs pf firewall comes to mind), but that's rather
an exception, I think.
So it is indeed extremely convenient to be able to patch servers while
they're operating, and I do it all the time, which has ensured
continues uptimes of up to half a year in one case (by that time the
patches had gotten a little too complex and anyway the lispworks
runtime was getting a bit out of fashion ;-)).
(And then I'm not even talking about test servers, which I usually
run directly off the development environment)
--
(espen)
Thomas F. Burdick
You patch your servers live, but I assume you do that by loading a
fasl, not by piece-wise developing the patch on the production machine
-- you do that on the test machines, right?
Espen Vestre
> You patch your servers live, but I assume you do that by loading a
> fasl, not by piece-wise developing the patch on the production machine
Right. Either by loading a fasl or by loading a .lisp (for the simpler
cases), or, for the quick-and-dirty small fixes, simply by typing into
a listener. Typically I do the latter programmatically, I have a shell
script that will connect to listeners on each server (about 20 of
them, in PrimeTrader's case) and evaluate the same form in each of
them.
> -- you do that on the test machines, right?
Yes.
--
(espen)
Tim X
> Tim X <ti...@spamto.devnul.com> writes:
>
> > t...@conquest.OCF.Berkeley.EDU (Thomas F. Burdick) writes:
> >
> > > "Jonathon McKitrick" <j_mck...@bigfoot.com> writes:
> > >
> > > > Thomas F. Burdick wrote:
> > > > > The missing nuance in the traditional answer is that you should be
> > > > > doing this on your testing server, not directly in production. Once
> > > > > you have your patch worked out, I bet most people just load the fasl
> > > > > into the running server image, and hope that no one makes a request at
> > > > > just the wrong moment.
> > > >
> > > > Is there a huge difference between loading fasls versus fetching new
> > > > source from cvs and loading that?
> > >
> > > I'd expect loading a fasl to be faster, but in principle there's no
> > > difference. The traditional answer I was referring to above was
> > > compiling functions one at a time, interactively. *That* is likely to
> > > leave your server in an inconsistent state for significant periods of
> > > time if you're not careful.
> > >
> >
> > I agree. I also wonder how critical an advantage it really is to be
> > able to change a production system 'on the fly', especially in this
> > day of growing use of clustered and load balanced systems. It strikes
> > me as just being one of those "wow look what I can do with CL" things
> > - impressive on one level, but not really that beneficial on another.
>
> It's certainly helpful for development, which you should be doing on a
> test server anyhow.
>
Having worked with C, C++ and Java web frameworks, I agree 100%. CL
gives a really nice interactive development cycle in
comparison. However, introducing temporary 'bugs' into a development
system isn't really a big problem - if things get into an inconsistent
state, you can just restart with a fresh environment. However IIRC,
this thread started with a question regarding one of lisps supposed
selling points as a web framework and the ability to change the
application on the fly by redefining functions without restarting the
system. I just question how much of a real benefit that is in a
production system, especially in this growing environment of clustered
systems in which restarting services only impacts performance and
doesn't result in service interruption.
Tim X
> Espen Vestre <es...@vestre.net> writes:
>
> > Tim X <ti...@spamto.devnul.com> writes:
> >
> > > I agree. I also wonder how critical an advantage it really is to be
> > > able to change a production system 'on the fly', especially in this
> > > day of growing use of clustered and load balanced systems.
> >
> > Well, even in highly clustered and load balanced system, you usually
> > fix connections to one machine if the protocol involves a lot of
> > statekeeping on the server side. Of course it's /possible/ to
> > implement hot standby solutions even in such cases (the hot standby
> > protocol for OpenBSDs pf firewall comes to mind), but that's rather
> > an exception, I think.
> >
> > So it is indeed extremely convenient to be able to patch servers while
> > they're operating, and I do it all the time, which has ensured
> > continues uptimes of up to half a year in one case (by that time the
> > patches had gotten a little too complex and anyway the lispworks
> > runtime was getting a bit out of fashion ;-)).
> >
> > (And then I'm not even talking about test servers, which I usually
> > run directly off the development environment)
>
> You patch your servers live, but I assume you do that by loading a
> fasl, not by piece-wise developing the patch on the production machine
> -- you do that on the test machines, right?
Shiver - now your scaring me - you mean people would develop and test
patches on a production system. Eeeeeek!
Actually, I should not sound shocked at this. Only two weeks ago, I
was going through the upgrade plans one of our staff was writing up
for a significant application upgrade. The app has over 25,000 users,
with around 20,000 logging in every day. I listened to the plan and at
the end asked "What is your backup plan if the upgrade fails? How will
you roll back to a known good version of the system?". I was stunned
and disapointed to see nothing abut a blank stare with the beginnings
of a dawning awareness of the importance to plan for failure as well
as success - we all want success, but sometimes failures occur and we
need to have a plan on how to deal with them.
Also, doesn't really matter what sort of sophisticated service/options
your framework has - it all still comes down to thought and careful
planning.
Paolo Amoroso
> Shiver - now your scaring me - you mean people would develop and test
> patches on a production system. Eeeeeek!
Well, didn't Paul Graham do that at Viaweb? :)
Paolo
--
Why Lisp? http://wiki.alu.org/RtL%20Highlight%20Film
The Common Lisp Directory: http://www.cl-user.net