More cool stuff from the dark corners of setuptools

[Adam Jones]

Ok, maybe that is a little melodramatic, but there are a lot of cool
things setuptools can do for you. One of the really nice ones is
automatically generating platform-specific console scripts and
installing them in the appropriate console directory. You have already
seen this done, since it is how the tg-admin command is created. Here's
a quick run-through on wrapping up your start-<projectname>.py file for
easy access.

First, copy your start-<projectname>.py file into the project itself in
a new file named "commands.py". Then reformat it so it looks something
like this:

import pkg_resources
pkg_resources.require("TurboGears")

from os.path import *
import os
import sys

def start():
from turbogears import update_config, start_server
import cherrypy
cherrypy.lowercase_api = True

# first look on the command line for a desired config file,
# if it's not on the command line, then
# look for setup.py in this directory. If it's not there, this
script is
# probably installed
if len(sys.argv) > 1:
update_config(configfile=sys.argv[1],
modulename="sample.config")
elif exists(join(os.getcwd(), "setup.py")):

update_config(configfile="dev.cfg",modulename="sample.config")
else:

update_config(configfile="prod.cfg",modulename="sample.config")

from sample.controllers import Root

start_server(Root())

The big change that we made was capturing most of the startup script's
logic in a new start function that takes no arguments. Other than that
we had to modify the elif statement that is looking for your dev.cfg
file so it uses the current directory instead of looking for it in the
same directory as the calling file.

Now all you have to do is add an entry point that tells setuptools you
are implementing a console script. This is done in your setup.py file.
Here is a sample line:

setup(
name="sample",
version=version,
...
entry_points = """
[console_scripts]
sample-start = sample.commands:start
"""

And that is pretty much it. When your project is installed, unless the
user specifically requests that console scripts are not installed, a
"sample-start" script is created and placed in the normal system
directory.

-Adam

[Jorge Vargas]

nice
I like the idea

anyone else thinks something like this should be the default?

> -Adam
>
>
> >
>

[Adam Jones]

I don't know about default, but "optional and trivial" sounds good to
me. There is not much to stop us from switching to putting the startup
command in a "command.py" file in the project folder and
importing/running from start-<projectname>.py. From there setup.py can
have a console script line provided but commented out. (like the
turbogears.* keywords are now)

Pros:
* Very easy to migrate to console-script based startup
* Provides a convenient place by default for command extensions
* Startup procedures migrate with the project, convenient place for
initialization
* Able to retain original user interaction model

Cons:
* Adds another file to the project directory (and another letter to
controllers.py file name completion!)
* Not many other uses for console scripts in TG (due to scheduler,
tg-admin extensions, etc)
* Few people know about and/or use console scripts or command
extensions.

Overall I think it is a +1, but then again I've been looking for an
easier way to start TG apps for a while now.

-Adam

[Christopher Arndt]

Adam Jones schrieb:

> I don't know about default, but "optional and trivial" sounds good to
> me. There is not much to stop us from switching to putting the startup
> command in a "command.py" file in the project folder and
> importing/running from start-<projectname>.py. From there setup.py can
> have a console script line provided but commented out. (like the
> turbogears.* keywords are now)

+1 (but I would call it 'startup.py' or something similar)

I very often have a few modifications in my start scripts, and it would be nice
to have the code in my project package, so that different versions can be
installed at the same time.

> Cons:
> * Adds another file to the project directory (and another letter to
> controllers.py file name completion!)

I don't get this. Which file completion do you mean?

> * Not many other uses for console scripts in TG (due to scheduler,
> tg-admin extensions, etc)

Oh, I could think of a few. Off-line database maintenance scripts for example.

> * Few people know about and/or use console scripts or command
> extensions.

That's no real argument. I, for one, didn't know much about how to build eggs
at all, before I started using TG...

Chris

[Adam Jones]

Christopher Arndt wrote:
> Adam Jones schrieb:
> > I don't know about default, but "optional and trivial" sounds good to
> > me. There is not much to stop us from switching to putting the startup
> > command in a "command.py" file in the project folder and
> > importing/running from start-<projectname>.py. From there setup.py can
> > have a console script line provided but commented out. (like the
> > turbogears.* keywords are now)
>
> +1 (but I would call it 'startup.py' or something similar)
>
> I very often have a few modifications in my start scripts, and it would be nice
> to have the code in my project package, so that different versions can be
> installed at the same time.
>

The reason I like command.py is that it seems appropriate to place
anything that ends up on the command line there, so your startup
function, command-line scripts, tg-admin extension(s) ... they can all
go in one place. Sort of like how controllers.py is a decent collection
place even though "dump it all in one file" doesn't work after a while.

> > Cons:
> > * Adds another file to the project directory (and another letter to
> > controllers.py file name completion!)
>
> I don't get this. Which file completion do you mean?
>
> > * Not many other uses for console scripts in TG (due to scheduler,
> > tg-admin extensions, etc)
>
> Oh, I could think of a few. Off-line database maintenance scripts for example.
>

Off-line as in "run when the server is down", or off-line as in "run
outside of a thread"? If its run outside of a thread the scheduler is a
better way to go unless you really really want to use cron or
something.

> > * Few people know about and/or use console scripts or command
> > extensions.
>
> That's no real argument. I, for one, didn't know much about how to build eggs
> at all, before I started using TG...

Yes, and that is part of the point. Eggs can be kind of confusing to
people who don't know how to use it. One thing that should be avoided
is having a bewildering selection of files and folders when a project
is created. Right now I don't know if the extra flexability in handling
project startup is worth the confusion it will cause.

I guess I can follow the lazy dev solution to it, file a patch and let
Kevin decide.

-Adam

[Christopher Arndt]

Adam Jones schrieb:

> The reason I like command.py is that it seems appropriate to place
> anything that ends up on the command line there, so your startup
> function, command-line scripts, tg-admin extension(s) ... they can all
> go in one place. Sort of like how controllers.py is a decent collection
> place even though "dump it all in one file" doesn't work after a while.

Point taken. But wouldn't the plural 'commands.py' be better then? (Though
there is a stdlib module of the same name, which is superseded by the
subprocess module.)

> Off-line as in "run when the server is down", or off-line as in "run
> outside of a thread"?

Offline as in when the site is down.

> If its run outside of a thread the scheduler is a
> better way to go unless you really really want to use cron or
> something.

What about database bootstrapping, i.e. pre-filling it with groups,
permissions, etc.?

> Yes, and that is part of the point. Eggs can be kind of confusing to
> people who don't know how to use it. One thing that should be avoided
> is having a bewildering selection of files and folders when a project
> is created. Right now I don't know if the extra flexability in handling
> project startup is worth the confusion it will cause.

I know what you mean, but it wouldn't be a problem to add some explanatory
comments in the Paste template.

Something like:

# The function in this module are used by the 'myproject-start.py' script
# in your main project directory.
#
# You can add you own functions used by command line scripts here.
# See the 'myproject-start.py' script for an example on how to structure
# your command line script.
#
# You should NOT remove this file unless you provide your own custom
# start-up script.

(BTW: I wished that the 'json.py' file had a similar comment that tells you
that it is save to remove it, if you don't use it's functions)

Chris

[Adam Jones]

Christopher Arndt wrote:
> Adam Jones schrieb:
> > The reason I like command.py is that it seems appropriate to place
> > anything that ends up on the command line there, so your startup
> > function, command-line scripts, tg-admin extension(s) ... they can all
> > go in one place. Sort of like how controllers.py is a decent collection
> > place even though "dump it all in one file" doesn't work after a while.
>
> Point taken. But wouldn't the plural 'commands.py' be better then? (Though
> there is a stdlib module of the same name, which is superseded by the
> subprocess module.)

I'd like to say that I thought of the namespace issue and chose
command.py instead for that reason, but in reality it was just luck.

>
> > Off-line as in "run when the server is down", or off-line as in "run
> > outside of a thread"?
>
> Offline as in when the site is down.
>
> > If its run outside of a thread the scheduler is a
> > better way to go unless you really really want to use cron or
> > something.
>
> What about database bootstrapping, i.e. pre-filling it with groups,
> permissions, etc.?
>

Yeah, I thought of that too. In fact, that exact issue is why I started
TurboSetup[1] and why I was looking into this to begin with. I'd like
to be able to have people go from zero to a running server without ever
editing a config file or working with the shell. Both of those are good
tools, but they were never intended for end user interaction.
Development on that kind of stopped when I ran into this issue with
starting an application, now that I have that resolved I can get back
to work on making TG installations drop dead simple.

> > Yes, and that is part of the point. Eggs can be kind of confusing to
> > people who don't know how to use it. One thing that should be avoided
> > is having a bewildering selection of files and folders when a project
> > is created. Right now I don't know if the extra flexability in handling
> > project startup is worth the confusion it will cause.
>
> I know what you mean, but it wouldn't be a problem to add some explanatory
> comments in the Paste template.
>
> Something like:
>
> # The function in this module are used by the 'myproject-start.py' script
> # in your main project directory.
> #
> # You can add you own functions used by command line scripts here.
> # See the 'myproject-start.py' script for an example on how to structure
> # your command line script.
> #
> # You should NOT remove this file unless you provide your own custom
> # start-up script.

That would probably work. Some people might delete the file without
reading, but it should be pretty easy to fix that. IMO anyone who
deletes files without knowing what they are needs that kind of learning
experience anyways.

-Adam

[1] http://cheeseshop.python.org/pypi/TurboSetup

[Christopher Arndt]

Adam Jones wrote:
> IMO anyone who
> deletes files without knowing what they are needs that kind of learning
> experience anyways.

I agree :-)

I've seen that you put this recipe in the official docs. Here's a few
remarks and questions:

- The setup.py snippet in the docs is missing the end quotation marks.

- If you add the "entry_points" argument to setup() you'll probably want
to remove the "scripts" argument, don't you?

- For development you'll still want a sample-start.py script that just has:

#!/usr/bin/env python
from sample.commands import start
start()

or is there a way to use the entry point when the egg is not installed?

- minor niggling: the comment in the start function should say "look in
the current directory" instead of "look in this directory".

Chris

[Adam Jones]

Christopher Arndt wrote:
> Adam Jones wrote:
> > IMO anyone who
> > deletes files without knowing what they are needs that kind of learning
> > experience anyways.
>
> I agree :-)
>
>
> I've seen that you put this recipe in the official docs. Here's a few
> remarks and questions:
>
> - The setup.py snippet in the docs is missing the end quotation marks.

It isn't a full quote on the setup function call anyways, hence the
ellipses. That said I cleaned it up a bit.

>
> - If you add the "entry_points" argument to setup() you'll probably want
> to remove the "scripts" argument, don't you?

Yes, you probably do. Really the only difference between the scripts
argument and what we are doing here is that it moves the whole thing
into the package and references that with setuptools. It makes working
with it a little bit cleaner and opens things up for reuse (the
start-sample command could also be tg-admin sample start with no code
duplication) but it isn't exactly revolutionary. I'd like to know how
the gui_scripts entry_point is supposed to work, as that would be
really cool too.

>
> - For development you'll still want a sample-start.py script that just has:
>
> #!/usr/bin/env python
> from sample.commands import start
> start()
>
> or is there a way to use the entry point when the egg is not installed?
>

No, but there is a way to cheat on installation. Run "setup.py develop"
and it will install your egg by telling setuptools to look in
setup.py's current directory. That way any change you make to the
project are automatically available without having to reinstall the egg
each time.

> - minor niggling: the comment in the start function should say "look in
> the current directory" instead of "look in this directory".

Fixed, thanks for the help.

-Adam

>
>
> Chris

[Christopher Arndt]

Adam Jones schrieb:

> Christopher Arndt wrote:
>> - The setup.py snippet in the docs is missing the end quotation marks.
>
> It isn't a full quote on the setup function call anyways, hence the
> ellipses. That said I cleaned it up a bit.

There are still unmatched triple quotes:

setup(
name="sample",
version=version,
...
entry_points = """
[console_scripts]
sample-start = sample.commands:start

...
)

should be

setup(
name="sample",
version=version,
...
entry_points = """\
[console_scripts]
sample-start = sample.commands:start

""",
...
)

when I fist saw the snippet as it is now, I was a bit confused, because it was
not immediately obvious that the value of entry_points is a string, containing
some form of an INI-style config.

Chris

[Ian Wilson]

Hello hello. So I'm trying to do this from the docs where it is
mentioned(ie. making a command line script using setup.py) but the
created script can't find the prod.cfg file. So my question is where
should that file be so that the start-sample script can find it?

-Ian

[Adam Jones]

That file should be in your current directory. You can also pass the
full path to the current directory as a command line argument.

Please note that TG actually does this by default, just in a slightly
different way. I would recommend sticking with their version unless you
are just testing this as a jumpstart to an entirely different command
line script.

-Adam

[Ian Wilson]

Well I was just going to put it into a gentoo startup script. So what
is the point of installing start-project or start-project.py in a
/usr/bin if you have to run it in a directory containing prod.cfg ?

[Christopher Arndt]

Ian Wilson schrieb:

>
> Well I was just going to put it into a gentoo startup script. So what
> is the point of installing start-project or start-project.py in a
> /usr/bin if you have to run it in a directory containing prod.cfg ?

No, you got that wrong. start-project.py looks for (in that order)

1. a cfg (full path) file provided on the command line

2. if "setup.py" exists in the current directory: "dev.cfg" in the current
directory

3. else: "prod.cfg" in the current directory

Putting the script in the egg as per the recipe you used, does not change that.

Chris

[Ian Wilson]

Thanks I think I am finally getting it now. So for my installation
the prod.cfg should go here:
/usr/lib/python2.4/site-packages/projectname-1.0-py2.4.egg/

I was unclear that i needed to manually copy the file to that location
to get the start script to find it. I thought setup.py would be
picking it up or something.

I think that is why this has the note: "needs work, how to configure"
http://docs.turbogears.org/1.0/DeployWithAnEgg
because the prod.cfg file is mentioned as required but it never tells
you really where to put it.

-Ian

[Christopher Arndt]

Ian Wilson schrieb:

> Thanks I think I am finally getting it now. So for my installation
> the prod.cfg should go here:
> /usr/lib/python2.4/site-packages/projectname-1.0-py2.4.egg/

I'm afraid not.

> I think that is why this has the note: "needs work, how to configure"
> http://docs.turbogears.org/1.0/DeployWithAnEgg
> because the prod.cfg file is mentioned as required but it never tells
> you really where to put it.

Because you can put it wherever you like, it just has to be either in the
directory "from where you start* start-project.py (i.e the current directory),
or you have to specify its location on the command line for start-project.py.

The current directory of the start-project.py script is *not* the the
egg-directory nor the directory, where the start-project.py script is installed
(usually /usr/bin or /usr/local/bin), but the directory from where you call it.

But you're right in that egg deployment currently does not address this
problem, and that's why you have to copy the config file by hand. You could put
a default configuration file in the egg (by including it somewhere below your
project package directory) and then locate it in your start script with the
pkg_resource module, but I haven't seen a ready-made recipe for that yet.

HTH, Chris

[Ian Wilson]

Ha awesome. All I need is a start/stop script that runs my production
server. So I'm just going to put my prod.cfg in
/usr/local/project/conf/prod.cfg and just hard code it in the shell
script. That was my orignal plan but I thought there might be a
cleaner or best practices, way of doing it. Thanks.

-Ian