Plugins in spyder

[Joseph Martinot-Lagarde]

So I wrote 2 plugins for spyder right now (autopep8 and line profiler, and I have a quick draft for running unit tests), and I wonder how I should try to get them integrated in spyder. I see 2 options, which may change the way future plugins are managed :

• Merge them in the official spyder repo. This means that they will need to be maintained by spyder's team (with the help of any contributor on bitbucket including me, of course).
• Keep them separate the eclipse way. In this case there need to be a way to install them in user space rather than in spyder's directory. A page listing available plugins somewhere on the wiki would help discovering them.

Now that the developpement is in bitbucket I'd prefer the first solution, but it depends on how you see the future of spyder !

[Carlos Córdoba]

Hi Joseph,

I'm very interested in including your unit tests plugin. It sounds really cool! I'm not so sure about the others because they seem to provide kind of the same functionality that our pep8 warnings and profiler plugin already do. If you want to maintain them as separate plugins that's fine for us but then we have to give you the opportunity to make them at least pip installable.

I'm not so sure how to do that (perhaps by making spyderplugins a namespace package?) but we have to definitely solve this situation for all plugin authors.

Cheers,
Carlos

El 04/12/13 08:12, Joseph Martinot-Lagarde escribió:

--
You received this message because you are subscribed to the Google Groups "spyder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spyderlib+...@googlegroups.com.
To post to this group, send email to spyd...@googlegroups.com.
Visit this group at http://groups.google.com/group/spyderlib.
For more options, visit https://groups.google.com/groups/opt_out.

[Carlos Córdoba]

After a bit more thought I came to see that the right approach is the one followed by Ninja-IDE:

1. Create a plugin index in Bitbucket.

2. Make Spyder read that index over the web and offer a dialog with the list of available ones.

3. From that dialog the user should be also able to download, install and uninstall a selected plugin (which we would save in ~/.spyder2)

It doesn't seem that hard to do but definitely not for 2.3.

Cheers,
Carlos

El 09/12/13 12:14, Carlos Córdoba escribió:

[Joseph Martinot-Lagarde]

This approach sounds right. the first step is to recognize plugins placed in ~/.spyder2 as copying files here is easy (I know, PR wanted ^^). The dialog would be nice but is not fundamental.

Yeah, my plugins have roughly the same functionnality, but I still think they add some value ! :) The line profiler could be especially useful for people coming from matlab as the ouput is closer to the matlab profiler. In scientific scripts it is very common that a single line takes the most time. Anyway if there's a simple way to install them they don't need to be included in spyder by default.

I'll work on my unittest plugin, I promise ! But I'll be quite busy till next year so it will have to wait...

[Jim Passmore]

For information/reference purposes, another working example using roughly this same approach is the Fiji distribution of ImageJ (open source image analysis software I use regularly, that is based on a plugin model).  The most useful wiki links are below.

• How to follow a 3rd party update site
• How to set up and populate an update site
• List of update sites

Jim

[anatoly techtonik]

I am not a favor of mimicking other approaches. Spyder is a scientific IDE, so
there should some research first. Usability research that will make it better than
others.

About updates sites of ImageJ. I really doubt it is secure. Without https://,
certificates and checksums the update process can easily be hacked to install
altered plugins into your source code tree to steal you next scientific discovery. =)

Guys from distutils made a great effort to protect the communication channel
between user system and PyPI in last year. They didn't have other choice. It is
now complicated and may not be reliable, because some issues are hard to
check. I don't want to mess with that stuff in Spyder. So, no third-party update
sites, but..

To simplify things, there can be registry, but.. it is step 2 (year 2015+). The
registry of approved (signed?) hash#size values comes shipped with Spyder.
hash#size lookup is made against plugin repository/wiki page, to fund a link that
matches hash#size. Even if download site gives you malicious package, if the
hash#size won't match, it will be ignored. Today it is possible to craft files that
match given md5 hash for sure, but I'd like to know if sha1+filesize limitation
makes it practically impossible? Anyway, this is a plan for distant future.

Step 1.
In closest perspective, the plugin discovery should be done by lookup from
certain known locations + paths in Spyder config to reference to unknown
paths (I like using things from repository checkouts). Config can also be used
to quickly enable/disable plugins. I'd like to avoid installation, unpacking and
safety checking. If `pip` can do this securely - I'd fully trust it.

I am against Python-wide plugin installation. Plugins have nothing to do with
Python installations - I may run multiple Spyder versions (flavors) with the
same interpreter and I don't want plugins to popup from nowhere and crash
IDE due to incompatible API.

So, let's concentrate on lookup order first, then on config with custom paths,
then think about everything else. `installation` will be dialog with optional
autodiscovery, but with obligatory manual way to add plugin by specifying a
path to it.

Going with Bitbucket /contrib/ or /plugins/ IMHO is a good way for the starter
of third-party plugins and work on decoupled API.

[Gonzalo A. PEÑA CASTELLANOS]

What is the current state of this discussion of plugins in spyder?

I have been using django and django-cms lately and I know Spyder is not a framework (like django is), but would it make sense to provide some decoupling of a spyder base and the different plugins so that they could be pip installable and maybe easier to maintain?

I am right now in the process of adding a conda package manager plugin but I have been thinking of other plugins that could make work easier in many other fields/directions that could leverage on a core. On the long run when Spyder moves to github (which is something that is almost agreed on as I understand) and hopefully more contributors jump on board we should make it easier to develop spyder and plugins for spyder.

Making all the plugins live inside the main repo will make in the future spyder much harder to maintain (or at least more annoying?). Just as Django has a ton of packages that enhance functionality and play (in theory) nicely together, and depend on spyder, could we make something on this direction happen maybe by doing a bit of metaclassing to simplify the definition of plugin base classes?

Right now there is a compatibility library for py2 and py3 that lives within spyder but there is also 'six' so if six takes care of that spyder does not have to.... 

We could have something like 

pip install spyder-plugin-conda

pip install spyder-plugin-x

pip install spyder-plugin-y

and spyder should be able to autodiscover this plugins and allow for enabling or disabling them (dialog...) 

Spyder plugins should be able to run by themselves (but depend on other modules inside spyderlib or even by separating modules from spyder that deal with this, the 'six' example...) 

We could even have something like spyder-qtwidgets or whatever and have a set of widgets defined in there.

I am also aware there are few developers/contributors right now and this seems like a daunting move, but in the long run I think it would pay off tremendously as plugins can be maintained by their own developers, without having the core of spyder contributors worry about this, except for having a clear and hopefully not so volatile plugin-api.

Cheers

[Joseph Martinot-Lagarde]

The state has stayed where we left it.

pip-installable modules are good, but it should be possible to install
them without having admin rights.

Plugins have made the success of many softwares (firefox and eclipse
come to mind) , and relying on users for less-used features would be
nice too.

Now if only I could find some free time...

> To simplify things, there can be registry, *but.. it is step 2*

> (year 2015+). The
> registry of approved (signed?) hash#size values comes shipped with
> Spyder.
> hash#size lookup is made against plugin repository/wiki page, to
> fund a link that
> matches hash#size. Even if download site gives you malicious
> package, if the
> hash#size won't match, it will be ignored. Today it is possible to
> craft files that
> match given md5 hash for sure, but I'd like to know if sha1+filesize
> limitation
> makes it practically impossible? Anyway, this is a plan for distant
> future.
>

> *Step 1*.

> --
> You received this message because you are subscribed to the Google
> Groups "spyder" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to
> spyderlib+...@googlegroups.com

> <mailto:spyderlib+...@googlegroups.com>.

> To post to this group, send email to
> spyd...@googlegroups.com

> <mailto:spyd...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/spyderlib.

> For more options, visit https://groups.google.com/d/optout.

[Gonzalo A. PEÑA CASTELLANOS]

Hi Joseph

Yes indeed, it would be nice like that... if we still want to go in the conda way of things, this "problem" would be solved.. at least for science (our biggest user community) 

conda install spyder-plugin-conda 
conda install spyder-plugin-pylint 
conda install spyder-plugin-profiler

Or even better just using the conda package manager... 

Even if we do not find the time to make it happen (the actual code) I think it would really help if we managed to have some brainstorming session (IRC or google hangouts... or the like..) at least to gather some "consensus" or random ideas?

And it would be nice also if this could be done in some sort of wiki or whatever to start looking at a common idea... (github wiki? or bitbucket?) 

What are the requirements, the needs the current bottlenecks and the roadmap. What would be a good folder structure for a plugin.. (locale, images... etc..)

The current plugin API was a great improvement on the plugin definition, but I would like to see some simplification of the p_package.py modules.

For development of plugins (while they are not pip or conda installable) there should be some spyder --plugin-dev option on startup to be able to load a plugin based on specific folder location and test it...

Lots of ideas to test :)

Cheers

[Gonzalo A. PEÑA CASTELLANOS]

Hi guys/gals (anyone reading this)

What are your views on using namespace packages for spyder?

Something like what is used for this project pyQode?