Define tasks as classes

[Thomas Kluyver]

I really like the ideas behind doit - simple, flexible task definitions. The only bit that seems out of place is stuffing everything into a dictionary to define a task. I can see the advantages for generating a series of tasks, but it seems to make the simple case of writing a single task more fiddly than it needs to be. I looked at the FAQ, and I agree that decorators aren't the answer.

I've done a quick branch to allow defining tasks as classes. A class effectively defines a new namespace, and you can put things inside it with an obvious syntax. It also means you don't need magic mechanisms to pass in keyword arguments for e.g. targets - the method can just pull them off 'self'. Here's a couple of example tasks, roughly based on samples in the documentation:

class task_hello(object):
    """Hello from Python."""
    targets = ['hello.txt']
   
    def run(self):
        with open(self.targets[0], "a") as output:
            output.write("Hello world.")
           
class task_checker(object):
    """Run pyflakes."""
    actions = ['pyflakes sample.py']
    file_dep = ['sample.py']

(In the first example, because there is no 'actions' variable defined, the run() method is used as the action)

I haven't done any of the necessary polish - checks, tests, docs, etc. - just the bare minimum to get those examples working. If you're interested in the implementation, see:

https://bitbucket.org/takluyver/doit/commits/75795b93adfdf0a30d6db9c8173d2bfb0470fc53

Is there any interest in taking this further?

Thanks,
Thomas

[Eduardo Schettino]

On Wed, Dec 5, 2012 at 1:54 AM, Thomas Kluyver <tak...@gmail.com> wrote:

I really like the ideas behind doit - simple, flexible task definitions. The only bit that seems out of place is stuffing everything into a dictionary to define a task.

The idea is that dictionaries are the interface that doit understands.
if you dont like/want to define tasks with a dict go ahead and create the tasks in whatever way you want.
But you will need to convert that to a dict before passing it to doit.

for example once i implemented the SCons interface (http://schettino72.wordpress.com/2011/07/19/faster-scons-on-top-of-doit/).
It is a bit outdated but you should get the idea.

 

I've done a quick branch to allow defining tasks as classes. A class effectively defines a new namespace, and you can put things inside it with an obvious syntax. It also means you don't need magic mechanisms to pass in keyword arguments for e.g. targets - the method can just pull them off 'self'. Here's a couple of example tasks, roughly based on samples in the documentation:

I think there is no "one size fits all" way of defining tasks.
And i think support for other styles needs to be improved...
last release I added support for extending/modifying the task loading mechanism.
Have you seen that?
http://pydoit.org/extending.html#task-loader-customization 

(In the first example, because there is no 'actions' variable defined, the run() method is used as the action)

I haven't done any of the necessary polish - checks, tests, docs, etc. - just the bare minimum to get those examples working. If you're interested in the implementation, see:

https://bitbucket.org/takluyver/doit/commits/75795b93adfdf0a30d6db9c8173d2bfb0470fc53

Is there any interest in taking this further?

I always try to make things customizable instead of adding it to the core doit implementation.
So could you try to implement it as a different TaskLoader?

Than we need to think on a better way to integrate it.
So I am interested on it. It might become part of doit but probably not on doit core implementation.
I guess it should be something like "reporters" where doit ship a few options but let users
define their own.

cheers

Thanks,
Thomas

--

[Eduardo Schettino]

On Wed, Dec 5, 2012 at 1:54 AM, Thomas Kluyver <tak...@gmail.com> wrote:

Is there any interest in taking this further?

Hi Thomas,

I had a new idea to allow easy customization on how tasks are defined.
It supports not only "classes" as you proposed but also "decorators" and "objects".

It is still up to the user to define its own way...
but much better than my previous advice to create a custom TaskLoader.

I wrote the details in a blog post: http://blog.schettino72.net/posts/doit-task-creation.html


cheers,
   Eduardo

[Thomas Kluyver]

On Friday, December 14, 2012 10:10:10 PM UTC, schettino72 wrote:

I had a new idea to allow easy customization on how tasks are defined.
It supports not only "classes" as you proposed but also "decorators" and "objects".

It is still up to the user to define its own way...
but much better than my previous advice to create a custom TaskLoader.

I wrote the details in a blog post: http://blog.schettino72.net/posts/doit-task-creation.html

Hi Eduardo,

Sorry it's taken me so long to respond to this. Thanks for adding support for alternative ways to define tasks.

Would you consider adding the Task base class in your example to doit (and perhaps also the decorator), so that it would be useful without having to include it as boilerplate in the dodo file?

I might also consider a way to define simple tasks directly with a function call:

compile = task(actions=['cc -c main.c'],
               file_dep= ["main.c", "defs.h"],
               targets=['main.o'],
              )

Best wishes,
Thomas

[Eduardo Schettino]

On Mon, Feb 25, 2013 at 9:48 PM, Thomas Kluyver <tak...@gmail.com> wrote:
> On Friday, December 14, 2012 10:10:10 PM UTC, schettino72 wrote:
>
> Would you consider adding the Task base class in your example to doit (and
> perhaps also the decorator), so that it would be useful without having to
> include it as boilerplate in the dodo file?

It was really an example... I didnt gave too much thought on it and
never used them in real life.
Before I add something like this to doit distribution it needs to be
more mature.

I would suggest you create an independent module and push it to pypi.
I could add a link to it in main doit docs, and one day doit might
incorporate it...

>
> I might also consider a way to define simple tasks directly with a function
> call:
>
> compile = task(actions=['cc -c main.c'],
> file_dep= ["main.c", "defs.h"],
> targets=['main.o'],
> )
>

On the first draft of my blog post the task decorator could also be
used as a function call.
I removed it just because I wanted to make the example simpler :)

I was also thinking about creating a task loader to read a file
similar to Makefiles...

cheers.

[Thomas Kluyver]

On 25 February 2013 15:07, Eduardo Schettino <schet...@gmail.com> wrote:

I would suggest you create an independent module and push it to pypi.
I could add a link to it in main doit docs, and one day doit might
incorporate it...

I give you... letsdoit. As in, it lets doit use tasks defined as classes, decorated functions, or function calls.

PyPI: https://pypi.python.org/pypi/letsdoit

Bitbucket: https://bitbucket.org/takluyver/letsdoit

It's MIT licensed, like doit itself, so code can easily be moved across later.

A minor snag I hit while developing this - if you have a class with a create_doit_tasks(self) method, doit will see the class object as well as the instances, and try to call Class.create_doit_tasks(), which throws an error. Perhaps there's a way to check that the function takes no arguments (or one argument, for bound methods), before calling it? You don't really want to just catch the error, because that makes debugging harder if there's a problem.

I also don't see any visible output from the simple example of a Python function printing, but that's the same for standard tasks. Is output suppressed by default? The options in 'doit help run' suggest that it will be shown.

 

I was also thinking about creating a task loader to read a file
similar to Makefiles...

I'd had that idea too ;-)

Thanks,

Thomas

[Eduardo Schettino]

On Tue, Feb 26, 2013 at 9:43 PM, Thomas Kluyver <tak...@gmail.com> wrote:
> On 25 February 2013 15:07, Eduardo Schettino <schet...@gmail.com> wrote:
>>
>> I would suggest you create an independent module and push it to pypi.
>> I could add a link to it in main doit docs, and one day doit might
>> incorporate it...
>
>
> I give you... letsdoit. As in, it lets doit use tasks defined as classes,
> decorated functions, or function calls.
>
> PyPI: https://pypi.python.org/pypi/letsdoit
> Bitbucket: https://bitbucket.org/takluyver/letsdoit
>

great. I will also try to use it in a real project later...

>
> A minor snag I hit while developing this - if you have a class with a
> create_doit_tasks(self) method, doit will see the class object as well as
> the instances, and try to call Class.create_doit_tasks(), which throws an
> error. Perhaps there's a way to check that the function takes no arguments
> (or one argument, for bound methods), before calling it? You don't really
> want to just catch the error, because that makes debugging harder if there's
> a problem.

patches welcome :)

>
> I also don't see any visible output from the simple example of a Python
> function printing, but that's the same for standard tasks. Is output
> suppressed by default?

it is supressed by default. option "verbosity" default to "1", it
should be "2" to display the output.
you can control the overall verbosity and also for individual tasks.

> The options in 'doit help run' suggest that it will
> be shown.

No it doesn't. check the "verbosity" option.

cheers,

[Thomas Kluyver]

On 26 February 2013 21:01, Eduardo Schettino <schet...@gmail.com> wrote:

> A minor snag I hit while developing this - if you have a class with a
> create_doit_tasks(self) method, doit will see the class object as well as
> the instances, and try to call Class.create_doit_tasks(), which throws an
> error. Perhaps there's a way to check that the function takes no arguments
> (or one argument, for bound methods), before calling it? You don't really
> want to just catch the error, because that makes debugging harder if there's
> a problem.

patches welcome :)

Here you go: https://bitbucket.org/schettino72/doit/pull-request/9/only-use-create_doit_tasks-if-it-can-be/diff

 

> I also don't see any visible output from the simple example of a Python

> function printing, but that's the same for standard tasks. Is output
> suppressed by default?

it is supressed by default. option "verbosity" default to "1", it
should be "2" to display the output.
you can control the overall verbosity and also for individual tasks.

> The options in 'doit help run' suggest that it will
> be shown.

No it doesn't. check the "verbosity" option.

Ah, I see. I was looking at:

-o ARG, --output-file=ARG  write output into file [default: stdout]

Does the captured output go anywhere, or is it discarded?

Thanks,

Thomas

[Eduardo Schettino]

What happens to the captured output depends on the "reporter".
On the "default" reporter it is only displayed if the task fails.
The "json" reporter will always display it.
The "zero" reporter will never show it.

cheers

[Roland Puntaier]

After reading through this discussion and

http://blog.schettino72.net/posts/doit-task-creation.html

I wonder whether there are any intentions to integrate this in doit?

letsdoit.py could be included in doit as an additional file

or it could be put into doit's tools.py, or where ever you choose.

I, for one, would welcome it.

Regards, Roland