Accessible Command Output using self.stdout & self.stderr
maxweld
modified to enable command output to be directed to self.stdout and
self.stderr so that the command output can be captured and made
accessible to an application.
I use a virtual web hosting environment where no shell access is
permitted. I would like to set up an administrative web interface
which enables commands to be executed and the command output
redisplayed to the user. However, as commands currently send output
directly to stdout or stderr, the comand output is not readily
accessible (Django 1.2.1).
Discussions with Russ McGee on Django Users suggested that some
commands, for testing system purposes, are now sending to self.stdout
and self.stderr, allowing the target to be redefined and the output to
be captured. Only dumpdata and loaddata commands have currently been
converted in trunk. I am proposing extending this to all commands in
core.management, and suggest that this should be promoted as the norm
for custom developed commands also.
If agreed, the work required to implement would be:
a) convert existing commands
b) document the feature
c) improve documentation of the existing feature allowing applications
to construct their own commands
d) improve documentation of how to execute commands programatically
(as opposed to in a shell)
If there is agreement, I am happy to undertake the above although
would appreciate a mentor being available for advice from time to time
as this would be my first contribution.
Longer term, if an administrative web interface to commands is to be
provided, we would need to devise an interface that adapted well into
the existing administrative interface. However, I see this as a second
stage in the process. First we need the command output to be
available.
Does the above proposal meet with agreement? What say you all?
Russell Keith-Magee
> I would like to propose that all commands in core.management be
> modified to enable command output to be directed to self.stdout and
> self.stderr so that the command output can be captured and made
> accessible to an application.
>
> I use a virtual web hosting environment where no shell access is
> permitted. I would like to set up an administrative web interface
> which enables commands to be executed and the command output
> redisplayed to the user. However, as commands currently send output
> directly to stdout or stderr, the comand output is not readily
> accessible (Django 1.2.1).
>
> Discussions with Russ McGee on Django Users suggested that some
Magee. I'm not Scottish :-)
> commands, for testing system purposes, are now sending to self.stdout
> and self.stderr, allowing the target to be redefined and the output to
> be captured. Only dumpdata and loaddata commands have currently been
> converted in trunk. I am proposing extending this to all commands in
> core.management, and suggest that this should be promoted as the norm
> for custom developed commands also.
>
> If agreed, the work required to implement would be:
> a) convert existing commands
> b) document the feature
> c) improve documentation of the existing feature allowing applications
> to construct their own commands
> d) improve documentation of how to execute commands programatically
> (as opposed to in a shell)
Seconded, and carried. I've already said I'll accept any patch that
implements any (or preferably all) of these points.
> If there is agreement, I am happy to undertake the above although
> would appreciate a mentor being available for advice from time to time
> as this would be my first contribution.
I'm happy to give you any feedback you need; if you post to django-dev
when you need assistance or feedback, I'll try to respond; if I don't,
someone else hopefully will.
> Longer term, if an administrative web interface to commands is to be
> provided, we would need to devise an interface that adapted well into
> the existing administrative interface. However, I see this as a second
> stage in the process. First we need the command output to be
> available.
There may well be a place for these commands to be exposed via a web
interface (especially if you're building a shared hosting type
environment), but I'm not convinced that this is something that is
appropriate to be baked into Django's core.
Firstly, it's a bit of an edge case; the biggest use case for this
sort of facility is to support a virtual server for which you don't
have shell access. I don't have any firm numbers, but I suspect that
this is an edge case of Django deployments.
Secondly, some management commands aren't really well suited to
execute on the same server that is serving them. It may be fine to
have one server executing management commands on another, though
(especially if you're setting up some sort of push-button hosting
environment).
Thirdly, some management commands can be time consuming or have the
potential for introducing database locks; as such, they aren't well
suited to execution in the request/response cycle. This means you
should be looking at implementing this using a job queue, and Django
is agnostic when it comes to such issues.
That said, if such an administrative interface were to be introduced,
it would almost certainly be as a contrib application. The usual path
for adding new contrib applications is for them to start as a
standalone project, and then be integrated when an argument can be
made that they are a "defacto standard implementation of a common
pattern". If this is something you're enthused about, I heartily
encourage you to start it as a standalone project.
Yours,
Russ Magee %-)
Gregor Müllegger
It is ofcourse possible to have wrapper in the Command class for stdout and
stderr output that redirect all data to system's stdout in most cases. But
this might not work for all output since we don't have control about all
management commands of thirdparty apps. A more "secure" approach of capturing
output is to overwrite `sys.stdout` and `sys.stderr` with your own file like
objects. A short example:
import sys
class MyStdout(object):
def __init__(self):
self.data = ''
def write(self, s):
self.data += s
myout = MyStdout()
sys.stdout = myout
print "Hello, World!"
assert myout.data == "Hello, World!\n"
This is not as clean as the self.stdout approach but will capture *all* output
instead of only the ouput of commands that will support the new API - which
might be what you want if you provide a webinterface.
I'm aware that Russ is not a big fan of Monkeypatching :) but I think this
would be a legal case - since there is no other way of intercepting the real
stdout output (I'm not sure but this might be even documented in python docs,
that its ok to overwrite sys.stdout and sys.stderr with your own objects).
Just want to point out that its possible - not that its the only way to go :)
Gregor
2010/7/13 Russell Keith-Magee <rus...@keith-magee.com>:
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
Russell Keith-Magee
> Related to the issue with stdout/stderr:
>
True :-)
> but I think this
> would be a legal case - since there is no other way of intercepting the real
> stdout output (I'm not sure but this might be even documented in python docs,
> that its ok to overwrite sys.stdout and sys.stderr with your own objects).
If you want to get really technical, there *is* another way it can be
done -- at least, if you're on a *NIX; you can redirect the file
descriptors that represent stdout/stderr. It's low-level plumbing, but
it works.
Yours,
Russ Magee %-)
maxweld
Ok - so I have a Git repository with Django trunk in it, and have
proceeded to change the commands. Following the changes there are no
failures in the regression tests. I am now trying to add new tests to
prove that my changes actually work.
Here is where I run into a few problems, no doubt at least in part as
I am new to this.
Q1. Should I create a separate, new module with my tests in it, or
just add tests to an existing module? If an existing one, how do I
determine which one?
Q2. In attempting my first test one change, it seems as though the
changes had had no effect. Have I missed some vital understanding...
I changed the last line of management/commands/flush.py thus
- print "Flush cancelled."
+ self.stdout.write("Flush cancelled.\n")
and created a basic test thus:
def test_flush(self):
new_io = StringIO.StringIO()
management.call_command('flush', verbosity=0,
interactive=True)
command_output = new_io.getvalue().strip()
self.assertEqual(command_output, "Flush cancelled.")
I can see the test run, as it asks for a response. But it writes the
command output to stdout (i.e. in the shell window) and not to new_io.
This was copied from the example you suggested. Have I missed
something basic?
Regards
David
Eric Holscher
Q1. Should I create a separate, new module with my tests in it, or
just add tests to an existing module? If an existing one, how do I
determine which one?
Q2. In attempting my first test one change, it seems as though the
changes had had no effect. Have I missed some vital understanding...
I changed the last line of management/commands/flush.py thus
- print "Flush cancelled."
+ self.stdout.write("Flush cancelled.\n")
and created a basic test thus:
def test_flush(self):
new_io = StringIO.StringIO()
management.call_command('flush', verbosity=0,
interactive=True)
command_output = new_io.getvalue().strip()
self.assertEqual(command_output, "Flush cancelled.")
maxweld
On Aug 1, 6:47 pm, Eric Holscher <e...@ericholscher.com> wrote:
> You need to pass in your custom stdout into the call_command function. You
> can see[1] where it takes those inputs and assigns them to your provided
> values.
Is there any way to supply a response in a test case that would
normally be run interactively? The test can only runs in interactive
mode, but if the test is left active in the test suite then users will
need to respond at each invocation, which is not good.
David