Displaying single-line progress while a management command runs

[Phil Gyford]

I have a custom management command which calls a method in another class, which fetches lots of data from a third-party API. Fetching the data could take a few seconds or it could take over an hour, depending on the quantity.

I'd like to display progress on the command line, so the user knows the command's still going but am struggling with a good way to do this. Criteria:

a) A single line that updates as the task progresses.

b) That line doesn't display while running unit tests (or is very easy to disable for tests).

c) That line doesn't display while running the command automatically (via cron or whatever).

Things I've tried so far:

1) Using print(), e.g.:

    print('Fetched %d of %d' % (n, total), end='\r')

In a loop, this nicely shows a single line that constantly updates with progress. But print() is nasty and when I run my unit tests, this output is displayed among the testing output. I assume it'll also be a pain to have that output when running the commands scheduled with cron (or whatever).

2) Using Django logging. This is "better" than print(), and doesn't mess up test output, but as far as I can tell there's no way to display a single, constantly updated, line showing progress. It's only going to show one line after another:

    Fetched 1 of 3000

    Fetched 2 of 3000

    Fetched 3 of 3000

3) Using tqdm <https://pypi.python.org/pypi/tqdm>. Makes it easy to display a command line progress bar but, again, I end up with loads of progress bars displaying in my test output, and I assume it'll do the same when scheduling the task to run.

Have I missed a standard way to do this? Or is there a way I haven't found to easily suppress tqdm's output during tests and when running the task scheduled? I can't be the first to have wanted this...

Thanks,

Phil

--

http://www.gyford.com/

[Erik Cederstrand]

> Den 9. maj 2016 kl. 14.23 skrev Phil Gyford <gyf...@gmail.com>:
>
> I have a custom management command which calls a method in another class, which fetches lots of data from a third-party API. Fetching the data could take a few seconds or it could take over an hour, depending on the quantity.
>

> [...]

> Things I've tried so far:
>
> 1) Using print(), e.g.:
>
> print('Fetched %d of %d' % (n, total), end='\r')
>
> In a loop, this nicely shows a single line that constantly updates with progress. But print() is nasty and when I run my unit tests, this output is displayed among the testing output. I assume it'll also be a pain to have that output when running the commands scheduled with cron (or whatever).

I do this kind of progress reporting a lot. Usually, I get around the test/cron output pollution by adding a 'silent' argument to the management command which determines if the commend should print progress reports or not. See below.

> 2) Using Django logging. This is "better" than print(), and doesn't mess up test output, but as far as I can tell there's no way to display a single, constantly updated, line showing progress. It's only going to show one line after another:
>
> Fetched 1 of 3000
> Fetched 2 of 3000
> Fetched 3 of 3000

It's actually quite simple. You need to create a custom handler like so:

import logging
import time
from django.core.management.base import BaseCommand

class OverwriteHandler(logging.StreamHandler):
# The extra spaces wipe previous output in case your messages are wariable-width
terminator = ' '*80 + '\r'

log = logging.getLogger('')
h = OverwriteHandler()
log.addHandler(h)

class Command(BaseCommand):
def handle(self, silent=False, **options):
log.setLevel(logging.DEBUG if silent else logging.INFO)
log.info('1 of 2')
time.sleep(1)
log.info('2 of 2')
time.sleep(1)

If you want to mix normal and progress logging in your management command, you need to use two loggers with different handlers.

Erik

[Phil Gyford]

Belated thanks for this Erik - that does work nicely. It gets complicated/annoying trying to untangle other kinds of logging too, including logging from third-party modules, but that's a separate problem :)

Erik

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/463A7786-888C-4CB0-9C68-43F855401924%40cederstrand.dk.
For more options, visit https://groups.google.com/d/optout.

--

http://www.gyford.com/

[Anton Samarchyan]

> Makes it easy to display a command line progress bar but, again, I end up with loads of progress bars displaying in my test output, and I assume it'll do the same when scheduling the task to run 

You can try a module I made to avoid this particular issue in django management command - https://pypi.python.org/pypi/django-tqdm