plugin system

71 views
Skip to first unread message

Kostis ankostis

unread,
Mar 31, 2015, 6:20:38 AM3/31/15
to pytho...@googlegroups.com

@schettino72 The introduction of a 2nd configuration file for me is the most problematic point.  I will give 3 examples:

1. Have a `foo/dodo.py` import tasks from another `foo/dodo.py` that depend on their companion `foo/doit.cfg` (ie for a "pickling" db-backend).
  It is too late to read companion `foo/doit.cfg`, the user should have already merge it with its own.

2. Use `doit -d <dir>` option. Which `doit.cfg` should it read? cwd or `-d` dir?

3. What about interactive consoles (ie IPython %doit)? Should they read cwd? home-dir (ie `~/.doit.cfg`), ipython profile-dit, or all of them together in some deterministic format?
   In master this issue is still unresolved.


I have seen this pattern of dual-config-files in maven-1x, they had exactly this setup, one for the project and another one for the plugin-properties, and it was too cumbersome to the users.
Eventually they scrapped the whole thing in maven-2 for a *single* pom.xml file.

And exactly this is a key point about the current architecture:
It complicates the life to the user of the plugin, for the benefit of the plugin developer (where it should be the other way round).



Concerning now your 2 objections to setuptools, the 2nd one (for an extra depepnency to `setuptools`) was invalidated by @kwpolska.

Regarding the 1st, that you have to have the plugin installed somehow, and hence "distributed".
No, it is not neccessary to have a package in *pypi* for to be installed with pip.
You can just use: ``python setup.py develop`` on the private copy of your code.
Of course you have to have a `setup.py` file.


Finally, regarding your question of what i'm trying to achieve with the sample "interface", a general answer would be: giving a chance to plugins to configure themselves in the context of other plugins.
Of course we can discuss this later if you deem worthwhile to proceed to a different approach.


Kind regards,
  kostis


  

Eduardo Schettino

unread,
Mar 31, 2015, 8:16:18 AM3/31/15
to python-doit
On Tue, Mar 31, 2015 at 4:41 PM, Kostis ankostis <anko...@gmail.com> wrote:

I wrote down a bit of my reasoning for the desing decisions here: https://github.com/pydoit/doit/wiki/plugin-system-design
 

@schettino72 The introduction of a 2nd configuration file for me is the most problematic point.  I will give 3 examples:

1. Have a `foo/dodo.py` import tasks from another `foo/dodo.py` that depend on their companion `foo/doit.cfg` (ie for a "pickling" db-backend).
  It is too late to read companion `foo/doit.cfg`, the user should have already merge it with its own.

There is some misunderstanding here. There is no such a thing "companion" `doit.cfg` for a `dodo.py`.
 

2. Use `doit -d <dir>` option. Which `doit.cfg` should it read? cwd or `-d` dir?
We should an option to specify the path of a configuration path when different from CWD.

The option `-d` is an option specific to the `run` command, `doit.cfg` is processed before any command is executed.
So `-d` has no influence whatsoever for `doit.cfg`

 

3. What about interactive consoles (ie IPython %doit)? Should they read cwd? home-dir (ie `~/.doit.cfg`), ipython profile-dit, or all of them together in some deterministic format?
   In master this issue is still unresolved.
I am not sure. Since you implemented this you can pick what you think makes more sense :)
 


I have seen this pattern of dual-config-files in maven-1x, they had exactly this setup, one for the project and another one for the plugin-properties, and it was too cumbersome to the users.
Eventually they scrapped the whole thing in maven-2 for a *single* pom.xml file.

As I explained in the wiki. DOIT_CONFIG in a dodo.py file is not enough so we need a real configuration file.
About having 2 ways I agree it might be confusing...
 

And exactly this is a key point about the current architecture:
It complicates the life to the user of the plugin, for the benefit of the plugin developer (where it should be the other way round).

What if the plugin developer and author are the same person :)
Seriously, I dont think adding an INI file with one line is more complicated than dealing with setuptools.
 



Concerning now your 2 objections to setuptools, the 2nd one (for an extra depepnency to `setuptools`) was invalidated by @kwpolska.

Note that I am not completing discarding setuptools plugins. We can certainly add support for it but without forcing its usage.
 

Regarding the 1st, that you have to have the plugin installed somehow, and hence "distributed".
No, it is not neccessary to have a package in *pypi* for to be installed with pip.
You can just use: ``python setup.py develop`` on the private copy of your code.
Of course you have to have a `setup.py` file.
thats true.
 


Finally, regarding your question of what i'm trying to achieve with the sample "interface", a general answer would be: giving a chance to plugins to configure themselves in the context of other plugins.
 
Yes the current architecture does not allow "plugins to configure themselves in the context of other plugins".
doit components are well defined and independent... Can you give an example that applies to doit?
 
Of course we can discuss this later if you deem worthwhile to proceed to a different approach.

Please note that the current implementation is still very incomplete.
The best way to move forward is to implement real plugins, then we can work on whatever limitations you found.
 

Kind regards,
  kostis

thanks for the feedback.

cheers

Kostis ankostis

unread,
Mar 31, 2015, 11:48:06 AM3/31/15
to pytho...@googlegroups.com


On Tuesday, March 31, 2015 at 2:16:18 PM UTC+2, schettino72 wrote:


On Tue, Mar 31, 2015 at 4:41 PM, Kostis ankostis <anko...@gmail.com> wrote:


Finally, regarding your question of what i'm trying to achieve with the sample "interface", a general answer would be: giving a chance to plugins to configure themselves in the context of other plugins.
 
Yes the current architecture does not allow "plugins to configure themselves in the context of other plugins".
doit components are well defined and independent... Can you give an example that applies to doit?

I have not an code-example as all this is new stuff. 
But I did contemplated about splitting the `cmd_graphx` plugin into ...more plugins.


Currently, cmd_graphx requires *optionally* matplotlib and graphviz, meaning that it can also provide results without them.  The "normal" way to deal with optional requirements is to split them in dependent projects (backends).

So you have cmd_graphx <-- (graphx_matplotlib, graphx_graphviz).

For this, I have at least 2 alternatives: 
  A. Parse the same `doit.cfg` file, or parse a different one.
  
  B. Use the setuptools extension mechanism and let the backends modify available graph-cmd's options (ie add 'matplotlib' into the dict of available outputs).  
  
Lets leave case2 aside for the moment.

A.1) For the 1st case, if I choose to use a 2nd cfg-file, I contribute into the proliferation of config-files, making harder the life of my users, i don't want to do this; and you don't want to do it either, trust me, people hated the guts out of how maven-1 forced them to keep multiple files spread in several folders.

A.2) If I choose to reuse the `doit.cfg` file, there are also 2  alternatives, α) access the parsed file from doit-base (needs modifications to your plugin code), or β) re-parse the file, which means reproducing your reading code.


In all cases (A & B) we need to allow "plugins to configure themselves in the context of other plugins".

But in the A case we are in the "maven-1" world, where for each project the user has to maintain 2 files, the `dodo.py` and the `doit.cfg`; 
that's the "companion cfg-file" i was talking about.  It is not always obvious how how to handle it, ie i still don't know how to resolve the IPython case because it requires the user to manually edit the `doit.cfg` in ipython profile, and I consider this awkward) 


Ciao,
  Kostis

Eduardo Schettino

unread,
Apr 1, 2015, 10:48:07 AM4/1/15
to python-doit
Truth is that I did not understand most of what you are saying but it seems there is confusion: Who write a `doit.cfg` is a plugin user, not a plugin author...
 
The plugins were designed to handle the extension points doit already provides.
You are talking about a plugin to extend something that doit doesnt support at all. This complete out of what i am trying to do.
And I do not agree the `graph` command should be split into many plugins...

You can just detect which libraries are available and expose just those. For example doit checks if multiprocess module is available or not. I dont understand how plugins could improve that.

I suggest you just get the `graph` command working fine first (it still doesnt work for me and I didnt have time to check why) than think if its functionality should be further split.

cheers

Kostis ankostis

unread,
Apr 2, 2015, 1:28:49 PM4/2/15
to pytho...@googlegroups.com


On Wednesday, April 1, 2015 at 4:48:07 PM UTC+2, schettino72 wrote:
On Tue, Mar 31, 2015 at 11:48 PM, Kostis ankostis <anko...@gmail.com> wrote:


On Tuesday, March 31, 2015 at 2:16:18 PM UTC+2, schettino72 wrote:


On Tue, Mar 31, 2015 at 4:41 PM, Kostis ankostis <anko...@gmail.com> wrote:


Finally, regarding your question of what i'm trying to achieve with the sample "interface", a general answer would be: giving a chance to plugins to configure themselves in the context of other plugins.
 
Yes the current architecture does not allow "plugins to configure themselves in the context of other plugins".
doit components are well defined and independent... Can you give an example that applies to doit?

I have not an code-example as all this is new stuff. 
But I did contemplated about splitting the `cmd_graphx` plugin into ...more plugins.


Currently, cmd_graphx requires *optionally* matplotlib and graphviz, meaning that it can also provide results without them.  The "normal" way to deal with optional requirements is to split them in dependent projects (backends).

So you have cmd_graphx <-- (graphx_matplotlib, graphx_graphviz).

For this, I have at least 2 alternatives: 
  A. Parse the same `doit.cfg` file, or parse a different one.
  
  B. Use the setuptools extension mechanism and let the backends modify available graph-cmd's options (ie add 'matplotlib' into the dict of available outputs).  
  
Lets leave case2 aside for the moment.

A.1) For the 1st case, if I choose to use a 2nd cfg-file, I contribute into the proliferation of config-files, making harder the life of my users, i don't want to do this; and you don't want to do it either, trust me, people hated the guts out of how maven-1 forced them to keep multiple files spread in several folders.

A.2) If I choose to reuse the `doit.cfg` file, there are also 2  alternatives, α) access the parsed file from doit-base (needs modifications to your plugin code), or β) re-parse the file, which means reproducing your reading code.


In all cases (A & B) we need to allow "plugins to configure themselves in the context of other plugins".

But in the A case we are in the "maven-1" world, where for each project the user has to maintain 2 files, the `dodo.py` and the `doit.cfg`; 
that's the "companion cfg-file" i was talking about.  It is not always obvious how how to handle it, ie i still don't know how to resolve the IPython case because it requires the user to manually edit the `doit.cfg` in ipython profile, and I consider this awkward) 



Truth is that I did not understand most of what you are saying

Maybe you should read it more carefully then :-)


 
but it seems there is confusion: Who write a `doit.cfg` is a plugin user, not a plugin author...

Why do you say so? 
Can you point me the exact point in my above text where did i seem to work under that confusion??



 
The plugins were designed to handle the extension points doit already provides.
You are talking about a plugin to extend something that doit doesnt support at all.

Can you elaborate on the functionality that "doit doesn't support at all"?


 
This complete out of what i am trying to do.

I believe that facilitating plugin-authors downstream should be part of your duty as the framework author - unless you don't really care about building an ecosystem; but even so, why should you forbid us from building extensible plugins? 

 
And I do not agree the `graph` command should be split into many plugins...

You can just detect which libraries are available and expose just those. For example doit checks if multiprocess module is available or not. I dont understand how plugins could improve that.

Plugins are above and before all, a Dependency; and that is why it makes perfectly sense that it is tied to the distribution systems (setuptools).

Actually `setuptools` have already EXACTLY what you just described, implemented as the ``find_plugins()`` method:

Which by the way explains that `setuptools` already support "private" plugins is some folder (so this part of the reasoning against setuptools on the wiki should eliminated completely).

.
 
I suggest you just get the `graph` command working fine first (it still doesnt work for me and I didnt have time to check why)

Agree,  `cmd_graphx` it is at a very alpha stage, and my time really limited.  If I remember correctly, i had asked you about what went wrong and even suggested some actions to workround or detect the issue, but received no reply.


 
than think if its functionality should be further split.

Repeat: of course it must be split, but i'm too occupied at the momment to do it right.  
The reason is that while `nettworkx` is needed by all graph-backends, it is only one of them is using `matplotlib`.
An the proper tool for handling situations like this or future ones (ie graphviz) , is to avoid importlib-magic and use plugins tied up to the installation machinery. 




What really got my attention in the general problem of "`doit` having now multiple sources of configurations", is 
a) that `dodo.py` was an after thought, and 
b)  the fact that if you can you would drop DOIT_CONFIG for `doit.cfg`.

My question is this:
  What is this setting that cannot be expressed with a python-dictionary and force you to to resort parsing win-ini config files? 


Cheers,
  


cheers
 

Eduardo Schettino

unread,
Apr 8, 2015, 4:24:58 PM4/8/15
to python-doit
Hi,

I completed a good enough implementation of doit plugin system.

At https://github.com/pydoit/doit-plugin-sample there is a minimal plugin to serve as an example.
Of course you need the master from github to test it.

So far on `doit` master there is support for 3 categories of plugins: Command, Reporter and DB backend.
I plan to implement task-loaders plugin before next release...

Answering Kostis concerns:

## setuptools support

I added support for auto-loading plugins that are registered using setuptools.
Local plugins without using setuptools are also supported.
It took only 10 lines of code to integrate support setuptools plugins! [1]

One question: Can I install a package and instruct pip/setuptools to not save the entry points?


## plugins relying on other plugins

It is possible for a plugin to rely/use other plugins that doit has no idea about.
doit will collect the entry points but it will not load/import plugin from categories it does not recognize.
For example the REPORTER plugins are only used by the RUN command.

For the plugin categories doit recognizes the order in which they are loaded/initialized does not matter...
but if for *your* plugin category, this is important, it is up to you to decide when and how you load and initialize
the plugins.


## doit having multiple sources of configuration

Yes, it goes against the Zen of Python "There should be one-- and preferably only one --obvious way to do it."
It doesnt really bother me, and IMO in this case, it is a feature :)
If it bothers you just stick to just one of them.

Kostis ankostis

unread,
Apr 9, 2015, 5:03:29 PM4/9/15
to pytho...@googlegroups.com
Bravo!



On Wednesday, April 8, 2015 at 10:24:58 PM UTC+2, schettino72 wrote:
Hi,

I completed a good enough implementation of doit plugin system.

At https://github.com/pydoit/doit-plugin-sample there is a minimal plugin to serve as an example.
Of course you need the master from github to test it.

So far on `doit` master there is support for 3 categories of plugins: Command, Reporter and DB backend.
I plan to implement task-loaders plugin before next release...

Answering Kostis concerns:

## setuptools support

I added support for auto-loading plugins that are registered using setuptools.
Local plugins without using setuptools are also supported.
It took only 10 lines of code to integrate support setuptools plugins! [1]

One question: Can I install a package and instruct pip/setuptools to not save the entry points?

Not any i have seen of.
But you may choose to  ignore an installed entry-point if instructed by ie `doit.cfg`.
I believe it is easy and necessary to have the capability to exclude installed-plugins.

But in general, plugins should be coded in such way as not to be invasive.




## plugins relying on other plugins

It is possible for a plugin to rely/use other plugins that doit has no idea about.
doit will collect the entry points but it will not load/import plugin from categories it does not recognize.
For example the REPORTER plugins are only used by the RUN command.

For the plugin categories doit recognizes the order in which they are loaded/initialized does not matter...
but if for *your* plugin category, this is important, it is up to you to decide when and how you load and initialize
the plugins.


## doit having multiple sources of configuration

Yes, it goes against the Zen of Python "There should be one-- and preferably only one --obvious way to do it."
It doesnt really bother me, and IMO in this case, it is a feature :)
If it bothers you just stick to just one of them.
I'm afraid that sooner-or-later i will have to resort to the `doit.cfg`.
Are definitely sure that you cannot merge the two files?
Kind regards,


Best Regards,
  Kostis 
 
Reply all
Reply to author
Forward
0 new messages