Taming management.py, the 1730-line behemoth
Adrian Holovaty
by how large django/core/management.py has gotten. It's 1730 lines
long -- and messy.
I'd like to split this into several files, turning
django.core.management into a package. The purest way of splitting the
functionality would be to give each manage.py action ("runserver",
"syncdb", etc.) its own Python module, like this:
django/
core/
management/
__init__.py
backends/
__init__.py
runserver.py
sqlall.py
syncdb.py
This would be the extent of my refactoring for now. Any strong
thoughts either way before I go ahead and implement it?
Adrian
--
Adrian Holovaty
holovaty.com | djangoproject.com
Adrian Holovaty
> backends/
> __init__.py
> runserver.py
> sqlall.py
> syncdb.py
I wrote "backends" there, but I meant "commands", which would be a
more meaningful package name.
Deryck Hodge
> django.core.management into a package. The purest way of splitting the
> functionality would be to give each manage.py action ("runserver",
> "syncdb", etc.) its own Python module, like this:
>
> django/
> core/
> management/
> __init__.py
> backends/
> __init__.py
> runserver.py
> sqlall.py
> syncdb.py
>
Hi, Adrian.
A nitpick, I know. :-) But "backends" seems a weird wording to me.
Why not "commands" or something similar for the name of the inner
directory?
Cheers,
deryck
Deryck Hodge
Forgive the noise. Apparantely I hit send a minute or two after you
answered this. I guess the "Update conversation" link for Gmail
doesn't work in Safari. :-)
Cheers,
deryck
Jacob Kaplan-Moss
> django.core.management into a package. The purest way of splitting the
> functionality would be to give each manage.py action ("runserver",
> "syncdb", etc.) its own Python module, like this:
Sounds like a really good idea to me -- looking forward to it!
I'd like to suggest a (somewhat controversial) extension:
Let any (installed) app provide its own manage.py actions in a similar
way -- something like::
schema_evolution/
management/
__init__.py
commands/
__init__.py
evolve.py
This would let projects like S-E -- there's a reason I used it for the
example -- provide custom management actions; installing said app
would make those actions "appear" in manage.py. This would go a long
way towards letting optional bits "feel" more built-in.
Jacob
Adrian Holovaty
> Sounds like a really good idea to me -- looking forward to it!
Cool -- I'm on it.
> I'd like to suggest a (somewhat controversial) extension:
>
> Let any (installed) app provide its own manage.py actions in a similar
> way -- something like::
I like this idea a lot. This will be made a lot easier with the refactoring!
George Vilches
>
> Let any (installed) app provide its own manage.py actions in a similar
> way -- something like::
>
> schema_evolution/
> management/
> __init__.py
> commands/
> __init__.py
> evolve.py
>
> This would let projects like S-E -- there's a reason I used it for the
> example -- provide custom management actions; installing said app
> would make those actions "appear" in manage.py. This would go a long
> way towards letting optional bits "feel" more built-in.
There aren't enough +1s in the world for me to vote as strongly as I
would like on both Adrian's initial proposal and this extension.
Looking forward to seeing this.
gav
Jacob Kaplan-Moss
> There aren't enough +1s in the world for me to vote as strongly as I
> would like on both Adrian's initial proposal and this extension.
That's just because I've been hoarding them. Once the bottom drops out
of the subprime market I think the next big thing is gonna be +1s. I'm
telling you, it's the next big investment.
Jacob
SmileyChris
> This would be the extent of my refactoring for now. Any strong
> thoughts either way before I go ahead and implement it?
Sounds great! One other thing that would be nice is if the sql
generation methods were refactored too so they could be reused easily
(for example, in schema evolution), but that can come after the
initial refactor I guess.
Brantley Harris
> way -- something like:
+1. As I was reading Adrian's post, I was thinking the very same thing.
Marty Alchin
> I'd like to suggest a (somewhat controversial) extension:
>
> Let any (installed) app provide its own manage.py actions in a similar
> way -- something like::
>
> schema_evolution/
> management/
> __init__.py
> commands/
> __init__.py
> evolve.py
>
> This would let projects like S-E -- there's a reason I used it for the
> example -- provide custom management actions; installing said app
> would make those actions "appear" in manage.py. This would go a long
> way towards letting optional bits "feel" more built-in.
I wholeheartedly agree that this functionality would be useful, but
I'm not sure I like the idea of specifying a structure for it. I think
I'd rather see a registration utility like the template library uses,
so apps could use whatever structure they like, and just register them
in management.py, since that's already used to look for syncdb
handlers anyway.
That's just my opinion, but it seems like it could avoid a lot of
bikeshed discussions where people debate the best file structure to
use.
-Gul
Adrian Holovaty
> That's just my opinion, but it seems like it could avoid a lot of
> bikeshed discussions where people debate the best file structure to
> use.
I think it would merely delay the bikeshed discussions. If we don't
have a required file layout, people will spend long mailing-list
threads five months from now talking about how *they* do things, and
how there should be an established convention. We can't win. :-)
(For the record, I don't feel strongly either way. What I do feel
strongly about is getting this initial refactoring done...)
Russell Keith-Magee
>
> On 8/15/07, Adrian Holovaty <holo...@gmail.com> wrote:
> > I'd like to split this into several files, turning
> > django.core.management into a package. The purest way of splitting the
> > functionality would be to give each manage.py action ("runserver",
> > "syncdb", etc.) its own Python module, like this:
+1 to the general idea of refactoring. The management.py beast needs
to be put on a leash :-)
> Let any (installed) app provide its own manage.py actions in a similar
> way -- something like::
+1 to this specific suggestion.
Yours,
Russ Magee %-)
Marty Alchin
> have a required file layout, people will spend long mailing-list
> threads five months from now talking about how *they* do things, and
> how there should be an established convention. We can't win. :-)
Understandable.
> (For the record, I don't feel strongly either way. What I do feel
> strongly about is getting this initial refactoring done...)
Agreed.
-Gul
Jannis Leidel
> I'd like to split this into several files, turning
> django.core.management into a package. The purest way of splitting the
> functionality would be to give each manage.py action ("runserver",
> "syncdb", etc.) its own Python module, like this:
Adrian,
+1 Yeah!
Dealing with management.py was a major pain in creating something
useful for my SoC project. A change to a more "pluggable" interface
makes the implementation of my release management [1] much easier.
Speaking of SoC, I also hate to have a crowded site-packages directory
but also think that the only chance for a vivid app repository is to
use the PyPI - which leads us right to the setuptools, upcoming
standard in Python 2.6. As a user I really like the idea of egg files
as much as dependency tracking, the development mode, installation via
easy_install and automatic unittesting.
That is why my current patch contains the setuptools bootstrap file to
build reusable application packages. A new setup.py file for Django
[2] is there too.
Regarding my patch for management.py: I changed "startapp" and created
"editapp", which are both calling a new metadata assistent. "--no-
skeleton" is for the old project-centered behaviour.
django.utils.package contains a ReleaseWrapper class for accessing
release metadata, an example release assistent "standalone_app", the
skeleton directory and ez_setup.py. The resulting app looks like this:
MyApp/
|-- INSTALL.txt
|-- MANIFEST.in
|-- README.txt
|-- docs/
|-- myapp/
| |-- __init__.py
| |-- models.py
| |-- templates
| | `-- myapp
| `-- views.py
|-- release.py
|-- setup.py
`-- tests/
I would like to encourage users to setup an account with the PyPI and
upload their apps to it. The Django application repository app which
I'm currently building will scan the PyPI for a hardcoded keyword
(e.g. "django.app") and save the resulting information for later
review by (to be decided) editorial functions on the website (e.g.
django-voting).
Since you are refactoring the whole thing anyway, what do you think
about a generic command like "app", "package" or something else which
combines "startproject", "startapp" and my changes?
I hope this wasn't too much off-topic..
Best,
Jannis
1: https://websushi.org/trac/browser/django-package/django-package.diff
2: https://websushi.org/trac/pastebin/2
Gary Wilson
> On 8/15/07, Jacob Kaplan-Moss <jacob.ka...@gmail.com> wrote:
>> This would let projects like S-E -- there's a reason I used it for the
>> example -- provide custom management actions; installing said app
>> would make those actions "appear" in manage.py. This would go a long
>> way towards letting optional bits "feel" more built-in.
>
> I wholeheartedly agree that this functionality would be useful, but
> I'm not sure I like the idea of specifying a structure for it. I think
> I'd rather see a registration utility like the template library uses,
> so apps could use whatever structure they like, and just register them
> in management.py, since that's already used to look for syncdb
> handlers anyway.
Yes, I think registration would be a good way to go. Where you could
register new commands or replace/extend existing ones. In fact, just
the other day I was taking a look at how Bazaar does this with their
plugins. If interested, take a look at bzrlib/commands.py
http://bazaar-vcs.org/bzr/bzr.dev/bzrlib/commands.py
Gary
Adrian Holovaty
> by how large django/core/management.py has gotten. It's 1730 lines
> long -- and messy.
>
> I'd like to split this into several files, turning
> django.core.management into a package.
As of changeset [5898], I've checked in this refactoring. Here's a
quick overview of the new design:
* django/core/management is now a package.
* django/core/management/commands is a package in which each module is
a django-admin.py command -- "syncdb.py", "runserver.py", etc. To add
or remove commands, just add/delete the modules in there; there's
nothing else to do (with one exception, which I'll bring up in a bit).
* Each command is a subclass of
django.core.management.base.BaseCommand. Each subclass is required to
implement a handle() method, which is passed the command-line
arguments as *args and the command-line options as **kwargs. These
subclasses also define "help" and "args" attributes, which are used in
the "django-admin.py --help" usage text.
* Some of the commands use a different parent class, AppCommand, which
takes care of some model-related functionality. In this case, the
subclasses are required to implement handle_app(), not handle().
* It would be nice if each Command knew which options it took -- e.g.,
the "shell" command takes the optional "--plain" command. I originally
implemented this but ran into the problem that the optparse module
doesn't allow the registration of multiple options with the same name,
which means we'd have to keep track of which options had already been
registered, and what do we do in the case of conflicts, etc. I just
decided to skip over this for now, just to get the refactoring done.
So if/when we add new django-admin commands that require new options,
we'll have to remember to add them to ManagementUtility.execute() in
django/core/management/__init__.py.
* There's a new API for calling commands directly: use
django.core.management.call_command(). Examples:
call_command('syncdb')
call_command('shell', plain=True)
call_command('sqlall', 'myapp', pythonpath='/foo/')
* This is backwards incompatible for any code that relied on the
various django.core.management functions. One notable exception to
this (i.e., a backwards *compatibility*) is that
django.core.management.execute_manager still exists -- because every
manage.py file out there relies on this function.
* I have tried to iron out all the bugs I could find (and there were a
lot, as this was quite a big refactoring!), but I'm sure I missed
some. Please poke around and speak up if there are any errors. Two
outstanding issues are that the "syncdb" command displays far more
verbose data than it did before, and a few of the unit tests are
failing (due to an output-formatting issue). I'm off to bed but will
be back at this tomorrow in case of problems. Django committers Down
Under, feel free to chip in, if you'd like.
limodou
>
> On 8/15/07, Jacob Kaplan-Moss <jacob.ka...@gmail.com> wrote:
> > Sounds like a really good idea to me -- looking forward to it!
>
> Cool -- I'm on it.
>
> > I'd like to suggest a (somewhat controversial) extension:
> >
> > Let any (installed) app provide its own manage.py actions in a similar
> > way -- something like::
>
> I like this idea a lot. This will be made a lot easier with the refactoring!
>
--
I like python!
UliPad <<The Python Editor>>: http://code.google.com/p/ulipad/
My Blog: http://www.donews.net/limodou
limodou
> > In implementing the "manage.py testserver" command, I've been struck
> > by how large django/core/management.py has gotten. It's 1730 lines
> > long -- and messy.
> >
> > I'd like to split this into several files, turning
> > django.core.management into a package.
>
> As of changeset [5898], I've checked in this refactoring. Here's a
> quick overview of the new design:
>
> * django/core/management is now a package.
>
> * django/core/management/commands is a package in which each module is
> a django-admin.py command -- "syncdb.py", "runserver.py", etc. To add
> or remove commands, just add/delete the modules in there; there's
> nothing else to do (with one exception, which I'll bring up in a bit).
>
there is a `commands` folder in user's project, so user can put his
own customized command script file in `commands` folder, then
management.py can find them. And this could be treated as global
command extension. And for per installed app, there maybe also a
`commands` folder, and management.py can find all command script file
according to all installed apps' `commands` folder. Just a suggestion.
Michael Elsdoerfer
> I'd rather see a registration utility like the template library uses,
> so apps could use whatever structure they like,
I'd prefer this too. At there very least, provide the option to manually
register commands as an alternative.
Michael
Vsevolod Solovyov
> As of changeset [5898], I've checked in this refactoring.
manage.py startapp doesn't work after update. I've filled ticket
http://code.djangoproject.com/ticket/5179
--
Vsevolod Solovyov
Russell Keith-Magee
> Under, feel free to chip in, if you'd like.
Have done; a few smallish changes leading up to [5903].
There were a few small bugs with missing imports; I've also added two
new base command types to improve error checking:
* NoArgCommand, for commands that don't take arguments. This causes
'manage.py syncdb foo' to raise a nice error, rather than a Python
stacktrace of "TypeError: handle() takes at most 2 non-keyword
arguments (3 given)"
* LabelCommand, for commands that take an argument that isn't an
application (createcachetable, startapp and startproject).
I also removed CopyFilesCommand. The argument checking that was
required for startapp and startproject was common with
createcachetable (and conceptually, any other command that requries a
single non-applabel argument).
I considered making CopyFilesCommand an extension of LabelCommand, but
it struck me that the workhorse copy_helper function was more useful
as a generic resource than as something that was tied to a command
that took a single non-application argument.
Yours,
Russ Magee %-)
Adrian Holovaty
> Have done; a few smallish changes leading up to [5903].
>
> There were a few small bugs with missing imports; I've also added two
> new base command types to improve error checking:
Excellent -- thanks for making the good changes.
Russell Keith-Magee
>
> I'd like to suggest a (somewhat controversial) extension:
>
> Let any (installed) app provide its own manage.py actions in a similar
> way -- something like::
FYI - I've committed this feature in [5923].
Yours,
Russ Magee %-)
Russell Keith-Magee
... and rolled it back out again in [5929] - it broke call_command in
a way I didn't consider. Back to the drawing board...
Russ %-)