Feature Suggestions/Thoughts

[Mnemnosi]

So, I've started working on my own personal little website, and
decided to go approach it as a static-generated affair. So I have my
local toolset that goes and does various things, and then I commit the
generated files to git, and when I push them to the server, they are
automatically exported and all is well.

I needed a sort of 'make' tool to glue things together and after
trying out all kinds, doit ... has actually turned out to be
beautiful. Its easy to add and manage tasks, even complex ones--
gluing things together with decorators and 'library' functions I've
written to do certain similar things.

There's two places where it has fallen short for me, though:

1. I generally work out of ~/projects/<project>/, with various sub-
directories in there, and I'm often digging around anywhere in there
when I want to run doit. Having to type "doit -f ~/projects/project/
dodo.py" is annoying.

2. 9 out of 10 times, what I want to happen is for that main dodo file
to be run. Generally, when I'm running those steps that are involved
in generating/testing the site locally. But then when I want to
publish it, there's a a different set of tasks I want to run instead.

Now, how I solved this was to write a wrapper script named "dodo"
checks to see if a 'dodo.py' directory is in my current directory, and
if not it traverses up the tree until it finds one. If it does, it
changes to that directory and runs doit.

Then, if an argument is passed to said script, it checks for
dodo_<arg>.py instead.

It looks like so:

#!/usr/bin/env python2.7

import os, sys

def main():
path = "."

filename = "dodo_" + sys.argv[1] + ".py" if len(sys.argv) > 1
else "dodo.py"

while os.path.abspath(path) != "/":
if os.path.exists(os.path.join(path, filename)):
break
path = path + "/.."

os.chdir(path)
os.execvp("doit", ["dodo", "-f", filename])

if __name__ == "__main__":
main()


Anyways, the wrapper works just fine for me so I don't -need- anything
to change in doit. Just thought I'd share some use-cases I've run into
in using it, see if anyone has any thoughts.

--Mnemnosi

[Eduardo Schettino]

On Wed, May 18, 2011 at 1:50 PM, Mnemnosi <mnem...@lavabit.com> wrote:

I needed a sort of 'make' tool to glue things together and after
trying out all kinds, doit ... has actually turned out to be
beautiful. Its easy to add and manage tasks, even complex ones--
gluing things together with decorators and 'library' functions I've
written to do certain similar things.

great.
 

There's two places where it has fallen short for me, though:

1. I generally work out of ~/projects/<project>/, with various sub-
directories in there, and I'm often digging around anywhere in there
when I want to run doit. Having to type "doit -f ~/projects/project/
dodo.py" is annoying.

this could be a good addition to doit. I only worry about accidentally executing some stuff. if we add a config for it I think it would be ok...
something like:

DOIT_CONFIG = {'allow_???' = True }

so this execute from a subdirectory would be enabled only if explicit asked. what do you think?

 

2. 9 out of 10 times, what I want to happen is for that main dodo file
to be run. Generally, when I'm running those steps that are involved
in generating/testing the site locally. But then when I want to
publish it, there's a a different set of tasks I want to run instead.

Then, if an argument is passed to said script, it checks for
dodo_<arg>.py instead.

typing:
$ doit <arg>
or
$ doit -f dodo_<arg>.py

doesnt make much difference to me...

you could put all tasks in the main dodo.py file. do you know you can control the default executed tasks with "DEFAULT_TASKS" config?

you could also import tasks from dodo_<arg>.py in the main dodo.py?
It is just python:

on dodo.py

from dodo_arg import task_x, task_y

 

Anyways, the wrapper works just fine for me so I don't -need- anything
to change in doit. Just thought I'd share some use-cases I've run into
in using it, see if anyone has any thoughts.

thanks. I would gladly accept a patch for the first item :)

cheers

[Mnemnosi]

On 5/18/11 9:24 AM, Eduardo Schettino wrote:

There's two places where it has fallen short for me, though:

1. I generally work out of ~/projects/<project>/, with various sub-
directories in there, and I'm often digging around anywhere in there
when I want to run doit. Having to type "doit -f ~/projects/project/
dodo.py" is annoying.

this could be a good addition to doit. I only worry about accidentally executing some stuff. if we add a config for it I think it would be ok...
something like:

DOIT_CONFIG = {'allow_???' = True }

so this execute from a subdirectory would be enabled only if explicit asked. what do you think?

That's a good point. Not everyone has my workflow, after all. :) Wouldn't it need to be an environment variable (or ~/.doit.py config-file), or something similar?

Because I'm not sure if its a good idea to traverse to parent directories and execute the dodo.py to look for its DOIT_CONFIG, just to see if it should be allowed to search up the tree. ISTM it should have permission to do that first, to prevent accidental code execution.

you could put all tasks in the main dodo.py file. do you know you can control the default executed tasks with "DEFAULT_TASKS" config?

you could also import tasks from dodo_<arg>.py in the main dodo.py?
It is just python:
on dodo.py

from dodo_arg import task_x, task_y

Yeah, I know you can put the files into the main dodo.py -- in fact, I'm doing the exact opposite in practice, pulling in the 'main' stuff into my dodo_publish.py, and using dependencies to get that stuff executed first.

The issue wasn't about organization of code really, but instead about workflow. I have two or three totally separate and different chains of actions that I tend to want to do in my environment, and 'doit <task>' didn't quite cut it-- given 'doit' ended up executing everything... so I wanted to separate out some stuff to do only when asked.

But! No, I somehow missed the DEFAULT_TASKS thing. That solves the problem just fine :) so #2 is off the list.

Anyways, the wrapper works just fine for me so I don't -need- anything
to change in doit. Just thought I'd share some use-cases I've run into
in using it, see if anyone has any thoughts.

thanks. I would gladly accept a patch for the first item :)

I shall go get my hands dirty and see about making one. :)

-- 

... Matthew
   (Mnemnosi)

[Mnemnosi]

On May 18, 9:24 am, Eduardo Schettino <schettin...@gmail.com> wrote:
> thanks. I would gladly accept a patch for the first item :)

Here's my attempt -- I tried to follow the style and methodology of
the existing package.

https://code.launchpad.net/~mnemnosi/doit/seek_dodo_files

Since this feature is about finding a dodo file, it can't use the
existing DOIT_CONFIG mechanism for configuration -- so I added a
mechanism for 'global' options, where "global" means 'applies to
everything a certain user does with doit'. You may or may not find
that desirable :) A pure-python ~/.doit.py seemed a decent solution.

--matt

[Eduardo Schettino]

On Thu, May 19, 2011 at 1:44 AM, Mnemnosi <mnem...@lavabit.com> wrote:
>
> On 5/18/11 9:24 AM, Eduardo Schettino wrote:
>>
>> There's two places where it has fallen short for me, though:
>>
>> 1. I generally work out of ~/projects/<project>/, with various sub-
>> directories in there, and I'm often digging around anywhere in there
>> when I want to run doit. Having to type "doit -f ~/projects/project/
>> dodo.py" is annoying.
>>
> this could be a good addition to doit. I only worry about accidentally executing some stuff. if we add a config for it I think it would be ok...
> something like:
>
> DOIT_CONFIG = {'allow_???' = True }
>
> so this execute from a subdirectory would be enabled only if explicit asked. what do you think?
>
> That's a good point. Not everyone has my workflow, after all. :) Wouldn't it need to be an environment variable (or ~/.doit.py config-file), or something similar?

I guess this option is better suited as project specific instead of being enabled for all dodo.py i.e.

~/work/dodo.py (some management stuff that i dont want to be executed by mistake
~/work/project_a/dodo.py (project with sub-folders where i want this dodo.py to be used...

 
>
> Because I'm not sure if its a good idea to traverse to parent directories and execute the dodo.py to look for its DOIT_CONFIG, just to see if it should be allowed to search up the tree.

dodo.py dont need to be "executed" it just need to be "imported". if the tasks should not be execute, these will only be called by "mistake" so it doesnt really matter if we are searching up the tree to do nothing...

 
>
> ISTM it should have permission to do that first, to prevent accidental code execution.

sure. we agree on this :)

> Here's my attempt -- I tried to follow the style and methodology of
> the existing package.

sorry. i didnt reply before you've done this... you were too fast :D

> Since this feature is about finding a dodo file, it can't use the
> existing DOIT_CONFIG mechanism for configuration

yes it can. the config is not used to describe the mechanism to find the dodo.py file. dodo.py is found by looking up in the directory. the config is used to decide if the dodo.py found by looking up is allowed to execute its tasks or not. agree?

> But! No, I somehow missed the DEFAULT_TASKS thing. That solves the problem just fine :) so #2 is off the list.

good. you can also create group of tasks like this: http://python-doit.sourceforge.net/dependencies.html#groups

cheers

[Mnemnosi]

> asked. what do you think?

I mused, and thought some sort of global option would be appropriate.
Thus the ~/.doit.py I did in the patch.

Buut--

> > That's a good point. Not everyone has my workflow, after all. :) Wouldn't
> > it need to be an environment variable (or ~/.doit.py config-file), or
> > something similar?
>
> I guess this option is better suited as project specific instead of being
> enabled for all dodo.py i.e.
>
> ~/work/dodo.py (some management stuff that i dont want to be executed by
> mistake
> ~/work/project_a/dodo.py (project with sub-folders where i want this dodo.py
> to be used...

Hmm. That's a point. Okay, how is this for a solution: instead of a
global
~/.doit.py file, "doit" will scan up from the current directory
looking for a
non-invisible "doitcfg.py" file; it would override whatever would have
been set
in ~/.doit.py

So you could do per-project configuration by just putting "doitcfg.py"
at the
base of whatever your project is.

For me, I'd probably just set ~/.doit.py -- because the way I work I'd
almost
always want that. But for others, they could set the whole 'find
dodo_file
in parents' setting on a per-project basis if they'd like.

> > Because I'm not sure if its a good idea to traverse to parent directories
> > and execute the dodo.py to look for its DOIT_CONFIG, just to see if it
> > should be allowed to search up the tree.
>
> dodo.py dont need to be "executed" it just need to be "imported". if the
> tasks should not be execute, these will only be called by "mistake" so it
> doesnt really matter if we are searching up the tree to do nothing...

[tweaking order of replies]

> > Since this feature is about finding a dodo file, it can't use the
> > existing DOIT_CONFIG mechanism for configuration
>
> yes it can. the config is not used to describe the mechanism to find the
> dodo.py file. dodo.py is found by looking up in the directory. the config is
> used to decide if the dodo.py found by looking up is allowed to execute its
> tasks or not. agree?

Well, that's a question of semantics of what "executed" means.
Importing
does in fact execute the file, it just doesn't call any functions in
it. I'm
personally a bit wary about importing files that I don't actually
intend on
using: importing a module can have side-effects -- even if its a good
idea
for one to code in such a way that said side-effects are safe.

But if you don't think that concern is warranted, I can fold the above
idea
into dodo.py: instead of looking for a "doitcfg.py", just search up
from the
current directory always and import any dodo.py's we find, looking for
one
which has DOIT_CONFIG["allow_seek_dodo_file"] (or whatever) set to
True.

I'm a bit wary about that, but only a bit: if you think that's better
I can tweak
the patch/branch. It may be I'm being paranoid.

> > Here's my attempt -- I tried to follow the style and methodology of
> > the existing package.
>
> sorry. i didnt reply before you've done this... you were too fast :D

No worries :D

It was a slow day, and was an excuse to dig into doit's internals and
learn
how things are done.

> > But! No, I somehow missed the DEFAULT_TASKS thing. That solves the problem
> > just fine :) so #2 is off the list.
> good. you can also create group of tasks like this:http://python-doit.sourceforge.net/dependencies.html#groups

Yeah, I'm doing that.

I basically have

DOIT_CONFIG = {"default_tasks": ["build"]}

def task_build():
return {"task_dep": ["headjs", "mootools", "blueprintcss"]}

and the like.

Then a similar, 'task_test' which builds out the website and launches
the test-server; then a task_push that its its own group, building out
a production version of the website, adding it to git and pushing it
off.

And thanks for the great tool :)

--matt

[Eduardo Schettino]

On Thu, May 19, 2011 at 3:54 PM, Mnemnosi <mnem...@lavabit.com> wrote:

> asked. what do you think?

I mused, and thought some sort of global option would be appropriate.
Thus the ~/.doit.py I did in the patch.

I would prefer to keep doit without any global options... If you want a global option you can put some code on all your dodo.py files that read these options from some central location like ~/.doit.py or whatever... but i would prefer to leave this out of doit core. maybe we can add it into doit/tools as optional stuff...
 

my idea is that dodo.py is already a "config" file (at least DOIT_CONFIG) ... so no need another doitcfg.py.

your concern is correct. but if we gonna be paranoid about importing, something like doitcfg.py would have the same problem as you are importing unknown code the same way...
I would just not care about this by now... I already though about supporting a doit.cfg file, but though it was not really necessary. if you want to add a doit.cfg file (INI format) i am ok with that.

 

> > Here's my attempt -- I tried to follow the style and methodology of
> > the existing package.
>
> sorry. i didnt reply before you've done this... you were too fast :D

No worries :D

It was a slow day, and was an excuse to dig into doit's internals and
learn
how things are done.

i hope you didnt have a hard time with doit's internals :)

cheers

[Eduardo Schettino]

On Wed, May 18, 2011 at 1:50 PM, Mnemnosi <mnem...@lavabit.com> wrote:

1. I generally work out of ~/projects/<project>/, with various sub-
directories in there, and I'm often digging around anywhere in there
when I want to run doit. Having to type "doit -f ~/projects/project/
dodo.py" is annoying.

I just took a look into this. I implemented it as command line parameter "--seek-file". so it doesnt require adding a new kind of configuration file.

of course it doesnt make much sense to type this all the type. the idea is that user could create an a shell alias * $alias doit='doit --seek-file' * so instead of asking a user to create a config file he has to create an alias.

the code: (http://bazaar.launchpad.net/~schettino72/doit/trunk/revision/455)

suggestions for a better name instead of "--seek-file" are welcome.

cheers,
  Eduardo