Reloading environment on each request
Michael Mahemoff
This is a wishlist request for a script to refresh the environment
each time, so I don't have to start the server whenever I change any
code. Probably needless to say that it would be very beneficial for
rapid development.
If there's no ticket for this, I'll add one.
Chris Dent
On Jul 15, 2009, at 3:27 PM, Michael Mahemoff wrote:
> This is a wishlist request for a script to refresh the environment
> each time, so I don't have to start the server whenever I change any
> code. Probably needless to say that it would be very beneficial for
> rapid development.
You may be asking two things at once, so let me see if I understand
things correctly.
One thing you could be saying is that you'd like the
tiddlywebconfig.py data to be refreshed in RAM as needed. The other
thing you could be saying is that you'd like the TiddlyWeb code (and
included modules) to be refreshed in RAM as needed.
While it would be possible to watch the tiddlywebconfig file at the
start of each request, and re-read it in if it has changed, doing so
would not work well as the vast majority of information in
tiddlywebconfig refers to modules that contain code that support
functionality of the system (e.g. serializers, stores, plugins,
renderers, filters etc). As with the scenario below, those modules
would need to be reimported, and their imports as well. There are ways
to do such things[1] but its the sort of complexity that seems
unwarranted when there's ctrl-c, kill, command history and shell
scripts at your disposal right their on your very machine (and the
other solutions below). That is: there are ways to automate the
process in your own environment that works best for you.
For refreshing the TiddlyWeb code, this is the job of the server that
hosts the code, not the code itself. What could happen is that the
server sits and watches some or all of the directories of code
involved and when a change happens, respawns itself so that code is re-
interpreted. A new process is required, otherwise all the modules
involved are not recompiled and some wild stuff can happen.
So this is not a problem that is specific to TiddlyWeb, and will _not_
be solved in the core of TiddlyWeb as TiddlyWeb does not (and should
not) do its own process management: the server hosting it does.
Possible solutions:
* Run using CGI. This will be slow, because your code is being
recompiled with each request, but because of that, you've always got
the latest code.
* Run under mod_wsgi, which will under some circumstances do process
reloads for you. See: http://code.google.com/p/modwsgi/wiki/ReloadingSourceCode
and other documentation in the vicinity.
* If you are using CherryPy (the server that runs when you run
twanager server) you can try to get http://www.cherrypy.org/wiki/AutoReload
to work. I've not tried it. It looks like it may be somewhat CPU
intensive and may require some fiddling to get to work in the
TiddlyWeb env.
* Run using spawning, a whiz bang high performance web server for
Python stuff that is supposed to have built module watching: http://tiddlyweb.peermore.com/wiki/#%5B%5BUsing%20spawning%5D%5D
If you come up with additional solutions do please post or patch as
seems relevant.
Michael Mahemoff
relatively complex framework; as a developer naieve to the underlying
environment, I just want any changes to work, but ultimately there
must be compromises to the amount of processing that goes on.
I am probably typical in that I could deal with restarting the server
upon each config change if I must, but I would not want to have to
start it after each code change, since my coding style is one where I
tend to test the app after just about every new line.
The typical developer would be using "twanager server", so CherryPi is
the critical component here. However, I don't have enough background
on how it fits with twanager to perform the changes described in that
AutoReload web page.
Chris Dent
On Jul 16, 2009, at 12:07 AM, Michael Mahemoff wrote:
> Right - it's two different things. It's the usual situation with any
> relatively complex framework; as a developer naieve to the underlying
> environment, I just want any changes to work, but ultimately there
> must be compromises to the amount of processing that goes on.
Heh. As you may recall I never really wanted TiddlyWeb to be a
framework, which is why the project (and I) am experiencing some
growing pains as people want and expect TiddlyWeb to be a framework.
My own biases in the design of the system are showing through rather
strongly. To be more explicit about some of them I'll exaggerate some
of them in this response. I understand that everybody works
differently and I'm happy about that, just making these responses so
the perspectives are clear. I'll be very happy if TiddlyWeb gets
better at being able to support lots of different styles of use, I
just don't want to be on the hook myself for making it happen. I've
got different areas of expertise.
First off, I don't like the notion of a developer being naive to the
underlying environment. I know that's kind of the point of a
framework, but since TiddlyWeb was designed as what amounts to a
pedagogical tool its connection to its underlying environment is
intentionally transparent and explicit.
> I am probably typical in that I could deal with restarting the server
> upon each config change if I must, but I would not want to have to
> start it after each code change, since my coding style is one where I
> tend to test the app after just about every new line.
My reaction to that is it sounds like you should be writing automated
tests. TiddlyWeb itself has a large body of tests, including ones that
interact with the provided web services without the use of a web
server. This is a very quick and easy to repeat way of testing code
changes without needing to wait around for the web server to do its
thing, or for me to switch to my browser and reload a page.
> The typical developer would be using "twanager server", so CherryPi is
> the critical component here. However, I don't have enough background
> on how it fits with twanager to perform the changes described in that
> AutoReload web page.
Sounds like a great opportunity to learn something about the
"underlying environment" :)
Anybody out there want to volunteer the effort to explore how to get
AutoReload working?
FND
Indeed - I've realized this myself only recently.
While writing tests is of course a good idea in general, it's even more
useful/efficient when the browser comes into play.
(Using py.test, the overhead is almost zero - it's essentially just a
bunch of assert statements.)
> Anybody out there want to volunteer the effort to explore how to get
> AutoReload working?
One could also imagine a brute-force approach, running a simple script
which periodically kills and restarts the server, perhaps using some
file's timestamp as trigger.
-- F.
simon mcmanus
So I agree with Mike here.
I understand your point about writing tests, but the fact is that people don't always write tests.
Sometimes you just want to figure out how an environment works, and in my experience the best way to do that is get in there and start changing stuff and see what happens.
In the PHP world people expect to be able to change code and then see the results simply by refreshing the browser. It has taken some adjustments in my working style to restart the server on each change. I don't know if this is standard practice with python code but I can imagine many PHP folk just would not bother.
So I think it would make the path to learning and building stuff with TiddlyWeb a great deal easier if we didn't have to restart the server on each .py change.
I will also continue to use TiddlyWeb despite this, but suspect it could be an unnecessary barrier to entry for many.
ps: As Mike points out, CherryPi will be most peoples first experience of TiddlyWeb so is really the critical component.
Thanks
Simon
2009/7/16 FND <FN...@gmx.net>
--
Simon McManus
blog : http://simonmcmanus.com
FND
> don't always write tests.
Yes, sadly. (This includes myself, though I'm getting better at it.)
> So I think it would make the path to learning and building stuff with
> TiddlyWeb a great deal easier if we didn't have to restart the server on
> each .py change.
That's why I suggested writing a simple script to do that automatically.
We can then add that to TPC as best-practice documentation.
-- F.
Chris Dent
On Jul 20, 2009, at 11:34 AM, simon mcmanus wrote:
> So I agree with Mike here.
That's great, that means there's at least two of you who can work on
coming up with a solution. :)
> In the PHP world people expect to be able to change code and then
> see the results simply by refreshing the browser. It has taken some
> adjustments in my working style to restart the server on each
> change. I don't know if this is standard practice with python code
> but I can imagine many PHP folk just would not bother.
In the PHP world you have to have an external server that knows how to
reload itself host your PHP. Different burdens in different situations.
> So I think it would make the path to learning and building stuff
> with TiddlyWeb a great deal easier if we didn't have to restart the
> server on each .py change.
Then fix it! You have the power. The TiddlyWeb code and community is
sitting there eagerly awaiting your contributions. It's the open
source way and all that, yeah?
To help you get started I've written a tiddlyweb plugin called
reloader.py:
http://github.com/tiddlyweb/tiddlyweb-plugins/tree/master/reloader
As far as I know it does all that you've asked for, but while it works
on Linux, it does not work on the Mac because of some interesting
limitations with calling exec from a multi-threaded application. I
suspect the fix is:
* to fork() from the watcher thread, and then in the forked process
call exec.
* in the leftover watcher thread call wait on the forked process to
exit and then call os._exit() (to fully kill off the old tiddlyweb)
The challenge here may be compensating for any conflicts with the old
process still listening on whatever port the new process now wants to
occupy. A time.sleep(1) somewhere will probably handle it, but the
where is a bit of a question.
Michael Mahemoff
> reloader.py:
>
> http://github.com/tiddlyweb/tiddlyweb-plugins/tree/master/reloader
>
> As far as I know it does all that you've asked for, but while it works
> on Linux, it does not work on the Mac because of some interesting
> limitations with calling exec from a multi-threaded application. I
> suspect the fix is:
>
> * to fork() from the watcher thread, and then in the forked process
> call exec.
> * in the leftover watcher thread call wait on the forked process to
> exit and then call os._exit() (to fully kill off the old tiddlyweb)
>
think it should be default behaviour - make it easy to get up and
running in development mode and using TiddlyWeb as intuitively as
possible. Optimising for production can happen later, as and when it
needs to happen.
Chris Dent
On Jul 20, 2009, at 4:39 PM, Chris Dent wrote:
> As far as I know it does all that you've asked for, but while it
> works on Linux, it does not work on the Mac because of some
> interesting limitations with calling exec from a multi-threaded
> application. I suspect the fix is:
>
> * to fork() from the watcher thread, and then in the forked process
> call exec.
> * in the leftover watcher thread call wait on the forked process to
> exit and then call os._exit() (to fully kill off the old tiddlyweb)
>
> The challenge here may be compensating for any conflicts with the
> old process still listening on whatever port the new process now
> wants to occupy. A time.sleep(1) somewhere will probably handle it,
> but the where is a bit of a question.
It was worse than all that. A tale too torrid to tell.
In the end, to get something working on the mac I had to go with a
different approach, so in here
http://github.com/tiddlyweb/tiddlyweb-plugins/tree/master/reloader
alongside the reloader system_plugin (which works using threads on
stuff besides OS X) is a wserver twanager_plugin which adds a command
which does the code watching, but in this case by using multiple
processes. I've tested it on OS X but suspect it will work on Linux as
well. I have no immediate access to ways to testing on Windows.
'twanager wserver' appears to be not quite as reliable as the reloader
option. It will sometimes restart the server twice when it should do
it only once. Also if you add a new module to the code base, the
parent process will never detect this, so you'll need to start from
scratch at that point.
Both of these processes work by paying attention to the data that are
in sys.modules. If the watcher process is happenning through multiple
processes (rather than threads) and only the child process actually
run all the tiddlyweb "stuff" then sys.modules can be left not up to
date with things that are loaded dynamically like available
serializations and challengers. If this turns out to be a problem
there are two ways to deal with it:
* Add the filenames to the list at config['reloader_extra_files'] and
they will be attended to directly. (wserver uses reloaders config
directives too)
* Have the serializations and challengers be loaded via plugins, not
just config. Then their containing modules will be in sys.modules. For
this to work most effectively 'wserver' ought to be the twanager
plugin listed _last_ in configuration.
wserver makes sure that it gets any system_plugins and all the built
in web handlers by calling tiddlyweb.web.server.load_app(). This
imports all handlers and system_plugins. If the above issues become a
big issue then code for automagically importing serializations and
challengers could be added.
Chris Dent
On Jul 20, 2009, at 6:37 PM, Michael Mahemoff wrote:
> Chris, this is a great help, thanks for making the plugin. FWIW I
> think it should be default behaviour - make it easy to get up and
> running in development mode and using TiddlyWeb as intuitively as
> possible. Optimising for production can happen later, as and when it
> needs to happen.
This is worth considering if a) there' s a general clamoring, b)
either of the reloader plugins prove they are stable and robust enough
for anyone/everyone to use. At the moment I'm not convinced of b:
there's all kinds of jiggery pokery going on in there that needs to
play out.
If you and Simon have success with either of the plugins for a while,
then we can see where things stand.
simon mcmanus
Thanks for the efforts here. It looks good. I will let you know when I get some time to try it.
Thanks
Simon
If you and Simon have success with either of the plugins for a while,
then we can see where things stand.