Question about the 'develop' command

33 views
Skip to first unread message

shoulderhigher

unread,
Mar 30, 2011, 11:56:49 AM3/30/11
to the-fellowship-...@googlegroups.com
Hi, guys:

I want to take the 'SetuptoolsFeatures' as my Google summer
project, in which a basic requirement is to add a 'develop' command
for the Distutils2. I have read the developer documents of setuptools
and found that it also has a 'develop' command, so I think there will
be many spirits and thoughts that we can learned from such solution in
setuptools. Setuptools allows developer to deploy his projects for use
in a common directory or staging area, but without copying any files.
In addition, it creates a special .egg-link file in the deployment
directory, that links to the project's source code.
But I'm still wondering what is the usage of such directory and
how does it work as expected?

I'm preparing my application these days, so I just want to make
it clear that what's the key point of the 'develop' command of
Disutils2. Hope someone can give me helps, thank you very much.

Best wishes,
higery

Carl Meyer

unread,
Mar 30, 2011, 12:15:33 PM3/30/11
to the-fellowship-...@googlegroups.com
Hello,

On 03/30/2011 11:56 AM, shoulderhigher wrote:
> I want to take the 'SetuptoolsFeatures' as my Google summer
> project, in which a basic requirement is to add a 'develop' command
> for the Distutils2. I have read the developer documents of setuptools
> and found that it also has a 'develop' command, so I think there will
> be many spirits and thoughts that we can learned from such solution in
> setuptools. Setuptools allows developer to deploy his projects for use
> in a common directory or staging area, but without copying any files.
> In addition, it creates a special .egg-link file in the deployment
> directory, that links to the project's source code.
> But I'm still wondering what is the usage of such directory and
> how does it work as expected?

In terms of importing the packages that you "setup.py develop", the
egg-link file is actually irrelevant. When you "setup.py develop"
setuptools also adds an entry to the easy-install.pth file in the
site-packages directory, pointing to the source code location. Python
processes these pth files at startup (in the stdlib "site.py" module")
and adds paths contained in them to sys.path (see the documentation for
the site module: http://docs.python.org/library/site.html).

It's worth noting that this pth file technique generally works in most
cases, but it isn't very careful to mirror what will happen with a real
installation. If you have additional Python packages alongside the one
that your setup.py is actually supposed to install, those additional
packages will also be available as top-level imports on sys.path after
you "setup.py develop." This can be a bit surprising if they have
generic names (e.g. "tests"), and it can also hide bugs in your setup.py
(if it doesn't actually install all the packages/modules it should,
they'll still be importable under "setup.py develop" anyway). So it
might be worth putting some thought into possible ways we could improve
on setuptools' implementation of "setup.py develop" - for instance, on
platforms that support it, symlinking might be a much better approach.
Also, handling of data-files will be a tricky area.

The .egg-link file is not used by Python itself at all, but is used by
setuptools to be able to detect the develop-installed project later if
you use the APIs in pkg_resources to get the full "working set" of
installed projects.

Hope this helps gives some pointers for further digging! Good luck on
your GSoC application.

Carl

Tarek Ziadé

unread,
Mar 30, 2011, 12:55:42 PM3/30/11
to the-fellowship-...@googlegroups.com
Hey,

besides what Carl said, my first idea on this was to work on a develop
command that :

- works with the notion of "project environment"
- have one pth file per project in the user home
- allow the addition or removal of directories in those pth files
- allow the activation of one "project environment" by using
environment variables

Technically speaking, it means:

- generating .dist-info dirs in place
- read the setup,cfg files to detect what the project contains in
order to link the scrips etc

But that's just a first braindump.

Cheers
Tarek

--
Tarek Ziadé | http://ziade.org

Xu Dehai

unread,
Mar 30, 2011, 10:41:44 PM3/30/11
to the-fellowship-...@googlegroups.com
Hi,

    Many thanks for Carl and Tarek's response.

    If I have used 'develop' command to install a project, then how should I do if I want to take a look at what such software or package will perform after some modifications on its source code.
    Document says that we can rerun develop command after we make any changes to setup script or C extensions to make things up-to-date; does it means that old files will be replaced with modified ones?


Best wishes,
higery



2011/3/31 Tarek Ziadé <ziade...@gmail.com>

higery

unread,
May 7, 2011, 11:22:12 AM5/7/11
to the-fellowship-...@googlegroups.com
Hi:

These days I have time to consider your opinions more carefully, and I still have some questions according to your following thought.
 
- works with the notion of "project environment"
Yes.

- have one pth file per project in the user home
Yes.

- allow the addition or removal of directories in those pth files
Yes.
 
- allow the activation of one "project environment" by using
environment variables
Automatically by Python or manually?
 

Technically speaking, it means:

- generating .dist-info dirs in place
.dist-info dirs is just used by 'uninstall' command in Distutils2, right?
 
- read the setup,cfg files to detect what the project contains in
order to link the scrips etc
 Does setup.cfg files have anything to do with import functionality? Accoring to your above thought of .pth files per project, we can import the module as expect now; it seems that there is not a necessarity to read the setup.cfg.


Best regards,
higery




Floris Bruynooghe

unread,
May 9, 2011, 5:13:05 PM5/9/11
to the-fellowship-...@googlegroups.com
Hi

On 30 March 2011 17:55, Tarek Ziadé <ziade...@gmail.com> wrote:
> besides what Carl said, my first idea on this was to work on a develop
> command that :
>
> - works with the notion of "project environment"
> - have one pth file per project in the user home
> - allow the addition or removal of directories in those pth files
> - allow the activation of one "project environment" by using
> environment variables

[...]


> But that's just a first braindump.

I've always been scared to terrified of setuptools' use of .pth files
so would be rather keen to avoid this situation in d2/packaging. As I
understand it you propose the .pth files to only contain directories
rather then magic (in the form of "import" lines in them), which is
obviously a much better start. Still, I wonder if it is worth taking
a step back and thinking about the problem it tries to solve.

As I understand it what it tries to solve is to have the packages and
modules of the distribution you're developing available in your
interpreter environment, i.e. on sys.path. This this to make it more
convenient when you're developing two depending packages together. To
me this is essentially managing PYTHONPATH for you.

The first question coming to mind then is that it overlaps with uses
of virtualenv or the like. So is this a feature that should live
there (which just leaves creating .dist-info in place as d2's job)?
Or should d2 perhaps have it's own mini-virtualenv like shell
modification which basically exports PYTHONPATH and prepends some
directories to it. The benefit of something like that is that it does
not change the normal non-development environment of the user.

Maybe there's other neat solutions here which might be better, but for
now I'm just wondering if I'm the only one to think that maybe it's
worth looking behind just copying setuptools develop command.

Regards
Floris

--
Debian GNU/Linux -- The Power of Freedom
www.debian.org | www.gnu.org | www.kernel.org

Éric Araujo

unread,
May 11, 2011, 11:26:40 AM5/11/11
to the-fellowship-...@googlegroups.com
Hi,

>> - allow the activation of one "project environment" by using
>> environment variables
> Automatically by Python or manually?

“By using environment variables” means it’s a manual shell operation.
It happens before Python is run. “Activation” is a virtualenv concept
though, not a setuptools one IIRC.

>> - generating .dist-info dirs in place
> .dist-info dirs is just used by 'uninstall' command in Distutils2,
> right?

Nope. Go back to PEP 376 :)

>> - read the setup,cfg files to detect what the project contains in
>> order to link the scrips etc
> Does setup.cfg files have anything to do with import functionality?
> Accoring to your above thought of .pth files per project, we can
> import the module as expect now; it seems that there is not a
> necessarity to read the setup.cfg.

You’re right; setup.cfg lists the modules (.py files, packages,
extension modules written in C) and scripts contained in the project;
the .pth file is what will let Python find the modules. Search for
“pth” in the Python docs and stdlib (modules site and maybe pkgutil
too).

Éric Araujo

unread,
May 11, 2011, 11:52:53 AM5/11/11
to the-fellowship-...@googlegroups.com
Hi,

Thanks Floris for your feedback on this.

> I've always been scared to terrified of setuptools' use of .pth files

Rightly so.

> so would be rather keen to avoid this situation in d2/packaging. As
> I
> understand it you propose the .pth files to only contain directories
> rather then magic (in the form of "import" lines in them), which is
> obviously a much better start.

I personally will not accept any code that abuses “import” support in
pth files to execute arbitrary code :) There’s got to be a better way,
given that we can edit things in site.py and other standard library
modules.

> Still, I wonder if it is worth taking a step back and thinking about
> the problem it tries to solve.
>
> As I understand it what it tries to solve is to have the packages and
> modules of the distribution you're developing available in your
> interpreter environment, i.e. on sys.path. This this to make it more
> convenient when you're developing two depending packages together.

Yes.

> To me this is essentially managing PYTHONPATH for you.

It’s more than PYTHONPATH: the environment variable only persists in
one shell (unless you set it in your shell config file, but let’s not do
that from Python code), whereas pth files are created once and found by
any Python in any shell (or without any shell).

To restate the problem as I understand it: I have clones of spamproject
and hamlib in different directories, spamproject depends on hamlib, so I
need a way to make hamlib visible to the Python used to run e.g. the
test suite of spamproject, but I don’t want to install hamlib.

> The first question coming to mind then is that it overlaps with uses
> of virtualenv or the like. So is this a feature that should live
> there (which just leaves creating .dist-info in place as d2's job)?

There is some overlap in goals, yes. Your mention of virtualenv made
me think of a limitation of the develop feature: it’s not isolated! I
want to make hamlib importable from spamproject, but I may not want any
invocation of Python to find hamlib, because of startup speed issues of
version conflicts. So here’s an idea: an option to the develop command
to control where to install the pth file. (I’m not sure if this is the
same than setuptools’ egg-path option to develop.)

> Or should d2 perhaps have it's own mini-virtualenv like shell
> modification which basically exports PYTHONPATH and prepends some
> directories to it.

I don’t think it’s possible to add shell-specific code in the stdlib.
Also note that a subset of virtualenv’s functionality is being added to
CPython 3.3, unrelated to packaging.

> Maybe there's other neat solutions here which might be better, but
> for
> now I'm just wondering if I'm the only one to think that maybe it's
> worth looking behind just copying setuptools develop command.

Sure! There’s still a bit of time before the coding period starts, we
can discuss. The key thing is to provide users with the same useful
features as develop, but we’re free to improve, extend or limit the
scope and implementation.

higery

unread,
May 15, 2011, 4:39:21 AM5/15/11
to the-fellowship-...@googlegroups.com
Hi,
 
“By using environment variables” means it’s a manual shell operation.  It happens before Python is run.  “Activation” is a virtualenv concept though, not a setuptools one IIRC.

I think such environment variables should be added by 'python setup.py develop' command.

Since 'Activation' is a virtualenv concept, then it maybe necessary to add an option to develop command, right? For instance, "python setup.py develop --activate" makes sense.
 


Best regards,
higery


Éric Araujo

unread,
May 15, 2011, 6:30:59 AM5/15/11
to the-fellowship-...@googlegroups.com
Hi,

>> “By using environment variables” means it’s a manual shell operation. It
>> happens before Python is run. “Activation” is a virtualenv concept though,
>> not a setuptools one IIRC.
> I think such environment variables should be added by 'python setup.py
> develop' command.

If it’s technically possible for Python to set variables in the calling
shell in a way that works across platforms and shells, sure, it will be
a nice extra functionality. I think you should first focus on the
specification of develop itself.

> Since 'Activation' is a virtualenv concept, then it maybe necessary to add
> an option to develop command, right?

With that comment I didn’t want to talk about implementation details,
just to say that you won’t find mentions of activation in the setuptools
code or docs because it’s a virtualenv feature.

Cheers

Floris Bruynooghe

unread,
May 15, 2011, 6:51:18 AM5/15/11
to the-fellowship-...@googlegroups.com
On 11 May 2011 16:52, Éric Araujo <win...@netwok.org> wrote:
> I personally will not accept any code that abuses “import” support in pth
> files to execute arbitrary code :)

Glad to hear!

> To restate the problem as I understand it: I have clones of spamproject and
> hamlib in different directories, spamproject depends on hamlib, so I need a
> way to make hamlib visible to the Python used to run e.g. the test suite of
> spamproject, but I don’t want to install hamlib.

ack

>> The first question coming to mind then is that it overlaps with uses
>> of virtualenv or the like.  So is this a feature that should live
>> there (which just leaves creating .dist-info in place as d2's job)?
>
> There is some overlap in goals, yes.  Your mention of virtualenv made me
> think of a limitation of the develop feature: it’s not isolated!  I want to
> make hamlib importable from spamproject, but I may not want any invocation
> of Python to find hamlib, because of startup speed issues of version
> conflicts.

Indeed, the isolation issue is one of the niggles I have with it.
When I develop myself I often have a released version in the user
site-packages but put an in-development version on sys.path somehow
(I'm clearly not using the develop command currently ;-)).

> So here’s an idea: an option to the develop command to control
> where to install the pth file.  (I’m not sure if this is the same than
> setuptools’ egg-path option to develop.)

Here's another idea which came to mind when reading your version of
the problem description: tell the setup.{py,cfg} of spamproject where
the directory of the hamlib you want is. Then the test command can
ensure the dist-info files of hamlib are build and add it to the
sys.path during the test run, keeping it completely isolated.
Similarly the develop command could then start you an interactive
python interpreter with sys.path set (and dist-info created) so you
can reach both hamlib and spamproject. But when you leave that
interpreter it's all gone again. None of this requires any .pth
files!

I realise that having to specify the location of hamlib in spamproject
might be a sticking point.

Carl Meyer

unread,
May 15, 2011, 10:40:12 AM5/15/11
to the-fellowship-...@googlegroups.com
On 05/15/2011 05:51 AM, Floris Bruynooghe wrote:
> On 11 May 2011 16:52, Éric Araujo <win...@netwok.org> wrote:

>> To restate the problem as I understand it: I have clones of spamproject and
>> hamlib in different directories, spamproject depends on hamlib, so I need a
>> way to make hamlib visible to the Python used to run e.g. the test suite of
>> spamproject, but I don’t want to install hamlib.

I don't agree with this as a problem statement for the "develop"
command. Spamproject is irrelevant, and introducing it into the problem
statement just confuses the purpose of "develop". A "develop" command
that is only usable for this use case, e.g. if its used via the
setup.cfg of spamproject, does not meet my use cases for "develop",
which don't necessarily involve dependencies of other projects at all.

IMO the problem statement for a "develop" command is simply this: I want
an in-place install. In other words, "develop" should function just like
a real installation of the package, in that it should be considered
installed for all purposes including dependency computation, scripts
should be available, and all modules and packages should be importable,
but the source code should not be copied to any other location; changes
made to the original source tree should be immediately available for
import. And it should not depend on transient modification of my shell,
either - it should be a persistent condition on the filesystem that is
reversed only if I explicitly ask for the "develop"-installed project to
be uninstalled.

The current setuptools feature meets these requirements, except that the
actual importability of packages and modules is simplistic and doesn't
work with complex uses of the "package_dir" keyword arg. I'd hope we
could make the d2 version more robust.

>>> The first question coming to mind then is that it overlaps with uses
>>> of virtualenv or the like. So is this a feature that should live
>>> there (which just leaves creating .dist-info in place as d2's job)?
>>
>> There is some overlap in goals, yes. Your mention of virtualenv made me
>> think of a limitation of the develop feature: it’s not isolated! I want to
>> make hamlib importable from spamproject, but I may not want any invocation
>> of Python to find hamlib, because of startup speed issues of version
>> conflicts.

No. "Lack of isolation" is NOT a limitation of "setup.py develop". It's
an orthogonal concern which is already handled perfectly adequately by
virtualenv (which hopefully we'll have in the stdlib for 3.3). There is
no reason why "setup.py develop" should scope-creep and gain
isolation-related features; it should simply be concerned with
performing an in-place installation, and should not care about the
destination of that installation, any more than a regular pysetup
install does.

To address the mentioned use case, you should simply create a virtualenv
called "spamproject-test" and then "develop"-install hamlib into that
virtualenv, rather than trying to pollute develop-install with
half-baked virtualenv-like features.

> Indeed, the isolation issue is one of the niggles I have with it.
> When I develop myself I often have a released version in the user
> site-packages but put an in-development version on sys.path somehow
> (I'm clearly not using the develop command currently ;-)).

Once again, virtualenv can already handle this just fine.

>> So here’s an idea: an option to the develop command to control
>> where to install the pth file. (I’m not sure if this is the same than
>> setuptools’ egg-path option to develop.)
>
> Here's another idea which came to mind when reading your version of
> the problem description: tell the setup.{py,cfg} of spamproject where
> the directory of the hamlib you want is. Then the test command can
> ensure the dist-info files of hamlib are build and add it to the
> sys.path during the test run, keeping it completely isolated.
> Similarly the develop command could then start you an interactive
> python interpreter with sys.path set (and dist-info created) so you
> can reach both hamlib and spamproject. But when you leave that
> interpreter it's all gone again. None of this requires any .pth
> files!

Please, no. This is just scope confusion all over the place, IMO.
"Develop" should not be tied to testing, and it should not be temporary,
and it should not be tied to some other packages setup.cfg. Any one of
these changes would make "develop" useless for the majority of my
current (daily) use of it.

Pth files are actually perfectly built for the use case of "develop",
and I think we should use them for it. Using them does not require
abusing them with executable code, like setuptools does (just to force
its eggs to the top of sys.path, which is unnecessary and anti-social IMO).

Carl Meyer

unread,
May 15, 2011, 10:50:07 AM5/15/11
to The fellowship of the packaging
Hi Tarek,

On Mar 30, 11:55 am, Tarek Ziadé <ziade.ta...@gmail.com> wrote:
> besides what Carl said, my first idea on this was to work on a develop
> command that :
>
> - works with the notion of "project environment"
> - have one pth file per project in the user home
> - allow the addition or removal of directories in those pth files
> - allow the activation of one "project environment" by using
> environment variables

I don't like this approach. Again, the entire concept of "project
environment" and activation of them seems like a half-baked re-
implementation of virtualenv, and I don't see the point of that. I
already use virtualenv and will continue to use it, I don't want d2 to
start trying to manage these pseudo-virtualenvs on my behalf. That
will just make my workflow more complicated and difficult.

Basically, I want something much closer to setuptools' existing
develop-install, just tweaked a bit for robustness. Setuptools'
existing approach is widely used and useful; I don't see why it needs
this complete re-thinking.

Carl

Éric Araujo

unread,
May 18, 2011, 1:07:14 PM5/18/11
to the-fellowship-...@googlegroups.com
Hi,

Thanks Carl for weighing in and clearing some misconceptions of mine.

Le 15/05/2011 16:40, Carl Meyer a écrit :
> On 05/15/2011 05:51 AM, Floris Bruynooghe wrote:
>> On 11 May 2011 16:52, Éric Araujo <win...@netwok.org> wrote:

> IMO the problem statement for a "develop" command is simply this: I want
> an in-place install. In other words, "develop" should function just like
> a real installation of the package, in that it should be considered
> installed for all purposes including dependency computation, scripts
> should be available, and all modules and packages should be importable,
> but the source code should not be copied to any other location; changes
> made to the original source tree should be immediately available for
> import.

Okay. I tried to think about more defined use cases that “I want an
in-place install”, i.e. think about why you want that and whether there
are solutions for those use cases that may have a user interface and an
implementation quite different than the setuptools develop command, but
I now think I was wrong, and your restatement of the feature is what we
should start with. (You forgot to mention that dist-info API should
find the distribution too.)

> And it should not depend on transient modification of my shell

Agreed.

> The current setuptools feature meets these requirements, except that the
> actual importability of packages and modules is simplistic and doesn't
> work with complex uses of the "package_dir" keyword arg. I'd hope we
> could make the d2 version more robust.

For one thing, there’s only one package_root option in d2 (it applies to
modules too), so with less complexity and more tests we can get to
confident robustness.

(I’d like to kill the “root package” concept in the d2 docs and just
talk about the top level for modules instead.)

> No. "Lack of isolation" is NOT a limitation of "setup.py develop". It's
> an orthogonal concern which is already handled perfectly adequately by
> virtualenv (which hopefully we'll have in the stdlib for 3.3).

Thanks for stating clearly it’s orthogonal. +1 on not concerning us
with that.

> Pth files are actually perfectly built for the use case of "develop",
> and I think we should use them for it. Using them does not require
> abusing them with executable code, like setuptools does

Good to hear you confirm it’s feasible.

Higery: Have you searched the ML archives for “develop”? There were one
of two starts of discussion about it, during the last of which I said:

> I have a suspicion the install command (or a subcommand) already has
> most of what it takes to do this. I’m still trying to understand the
> extra_path argument (http://bugs.python.org/issue901727), which may or
> may not give us 90% of what’s needed to implement this feature.

Cheers

Carl Meyer

unread,
May 18, 2011, 1:50:07 PM5/18/11
to the-fellowship-...@googlegroups.com
Hi Éric,

On 05/18/2011 12:07 PM, Éric Araujo wrote:
> Okay. I tried to think about more defined use cases that “I want an
> in-place install”, i.e. think about why you want that and whether there
> are solutions for those use cases that may have a user interface and an
> implementation quite different than the setuptools develop command,

Sure, that's a reasonable thing to do.

> but
> I now think I was wrong, and your restatement of the feature is what we
> should start with.

Good. :-)

> (You forgot to mention that dist-info API should
> find the distribution too.)

Yes.

>> The current setuptools feature meets these requirements, except that the
>> actual importability of packages and modules is simplistic and doesn't
>> work with complex uses of the "package_dir" keyword arg. I'd hope we
>> could make the d2 version more robust.
> For one thing, there’s only one package_root option in d2 (it applies to
> modules too), so with less complexity and more tests we can get to
> confident robustness.
>
> (I’d like to kill the “root package” concept in the d2 docs and just
> talk about the top level for modules instead.)

Excellent, I hadn't followed that discussion. That will make things a
lot simpler.

>> Pth files are actually perfectly built for the use case of "develop",
>> and I think we should use them for it. Using them does not require
>> abusing them with executable code, like setuptools does
> Good to hear you confirm it’s feasible.

If we needed to support complex package_dir hierarchies, we might need
multiple pth entries. But if there's just one package_root, having that
root in a pth file should be sufficient.

One remaining issue with using pth this way is that it's possible for
"develop" to give you more stuff importable than you actually want, if
you have Python modules/packages under your package_root, but excluded
in your setup.cfg. This would be an advantage of symlinks, in that we
could just symlink over the specific packages/modules listed in
setup.cfg. But I'm not sure the "extra stuff" problem is really that big
a deal; it's probably not worth two separate implementations of
"develop", one for OSes with symlink and one for those without.

> Higery: Have you searched the ML archives for “develop”? There were one
> of two starts of discussion about it, during the last of which I said:
>
>> I have a suspicion the install command (or a subcommand) already has
>> most of what it takes to do this. I’m still trying to understand the
>> extra_path argument (http://bugs.python.org/issue901727), which may or
>> may not give us 90% of what’s needed to implement this feature.

Looking forward to your work on this feature, Higery!

Carl

higery

unread,
May 19, 2011, 10:43:34 PM5/19/11
to the-fellowship-...@googlegroups.com
Hi, Eric:
 
Higery: Have you searched the ML archives for “develop”?  There were one
of two starts of discussion about it, during the last of which I said:

> I have a suspicion the install command (or a subcommand) already has
> most of what it takes to do this. I’m still trying to understand the
> extra_path argument (http://bugs.python.org/issue901727), which may or
> may not give us 90% of what’s needed to implement this feature.

 Yes, I paid attention to your discussing in bug tracker(Issue 8668).
These days' talk about develop command in this thread have helped me a lot and  made me more clear to know what I should do in future coding days.

Best wishes to everyone here,
higery




higery

unread,
May 19, 2011, 10:45:34 PM5/19/11
to the-fellowship-...@googlegroups.com
Hi, Carl:


Looking forward to your work on this feature, Higery!

Thank you very much for your useful opinions and I will try my best to work on this feature.

Best regards,
higery


Éric Araujo

unread,
Jun 15, 2011, 10:45:56 AM6/15/11
to the-fellowship-...@googlegroups.com
Hi folks,

I have another question about the develop command for people who use it
a lot. What do you think about the name? Do you want to keep it as is,
to make clear it’s the same thing as the setuptools command (but working
with packaging/distutils2), or are you open to finding a better name?

I say better because to me, develop is not a great description.
Developing is what I do with my text editor, my VCS and my bug tracker.
This command is in my eyes a kind of install command.

Thoughts?

Michael Foord

unread,
Jun 15, 2011, 11:06:23 AM6/15/11
to the-fellowship-...@googlegroups.com
To me it implies "make available the under development version", so develop is fine for me (and has the right connotations). But yes, it is "a kind of install command".

Michael Foord


--
http://www.voidspace.org.uk/

May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing http://www.sqlite.org/different.html

Carl Meyer

unread,
Jun 15, 2011, 12:49:06 PM6/15/11
to the-fellowship-...@googlegroups.com
Hi,

Pip uses the term "editable", which I think makes some sense. But really
all the variations including "install" that I can think of end up being
long and unwieldy: "install_editable"? Ugh. One word is better than two.
Really, "develop" is not a bad name, and I think there's some benefit to
continuity of terminology with setuptools when there isn't a good reason
for a break.

Carl

Erik Bray

unread,
Jun 15, 2011, 12:42:43 PM6/15/11
to the-fellowship-...@googlegroups.com
On Wed, Jun 15, 2011 at 11:06 AM, Michael Foord <fuzz...@gmail.com> wrote:
>
>
> On 15 June 2011 15:45, Éric Araujo <win...@netwok.org> wrote:
>>
>> Hi folks,
>>
>> I have another question about the develop command for people who use it
>> a lot.  What do you think about the name?  Do you want to keep it as is,
>> to make clear it’s the same thing as the setuptools command (but working
>> with packaging/distutils2), or are you open to finding a better name?
>>
>> I say better because to me, develop is not a great description.
>> Developing is what I do with my text editor, my VCS and my bug tracker.
>>  This command is in my eyes a kind of install command.
>>
>> Thoughts?
>
> To me it implies "make available the under development version", so develop
> is fine for me (and has the right connotations). But yes, it is "a kind of
> install command".

I wouldn't object to making it an option to the install command.
Something like `pysetup install --develop`. Seems simpler to me than
having an entirely different command.

Erik

Carl Meyer

unread,
Jun 15, 2011, 1:11:58 PM6/15/11
to the-fellowship-...@googlegroups.com
On 06/15/2011 11:42 AM, Erik Bray wrote:
> I wouldn't object to making it an option to the install command.
> Something like `pysetup install --develop`. Seems simpler to me than
> having an entirely different command.

This is what pip does (except the option is -e/--editable), and I
thought of proposing it. The issue is that this option severely
restricts the valid arguments to "install" when it's used (no PyPI
project names, just local directories), and it feels a little odd for an
option to have that effect on a command.

In pip, you can pass either a local directory or a VCS url to "install
-e". In pysetup, that would be just a local directory, since pip's VCS
backends aren't going into the stdlib.

Carl

Éric Araujo

unread,
Jun 16, 2011, 10:00:42 AM6/16/11
to the-fellowship-...@googlegroups.com
Le 15/06/2011 19:11, Carl Meyer a écrit :
> On 06/15/2011 11:42 AM, Erik Bray wrote:
>> I wouldn't object to making it an option to the install command.
>> Something like `pysetup install --develop`. Seems simpler to me than
>> having an entirely different command.
>
> This is what pip does (except the option is -e/--editable), and I
> thought of proposing it. The issue is that this option severely
> restricts the valid arguments to "install" when it's used (no PyPI
> project names, just local directories), and it feels a little odd for an
> option to have that effect on a command.

You’re confusing pysetup actions and pysetup commands (higery, you
too!). The things that can operate on project names, tarballs, etc. are
commands; for example, pysetup list and pysetup install. The things
that operate on a source directory with a setup.cfg are commands (just
like distutils commands), for example pysetup run build_ext and pysetup
run install_lib.

develop is a commmand, not an action, so a new install_* name or an
option to the existing install_dist command would work for me.

Regards

Pierre-Yves David

unread,
Jun 17, 2011, 5:10:53 AM6/17/11
to the-fellowship-...@googlegroups.com
On Thu, Jun 16, 2011 at 04:00:42PM +0200, Éric Araujo wrote:
> The things that can operate on project names, tarballs, etc. are
> commands;
> […]

> The things that operate on a source directory with a setup.cfg are commands

Looks like your confused them too :-)

--
Pierre-Yves David

http://www.logilab.fr/

signature.asc

Éric Araujo

unread,
Jun 17, 2011, 1:04:51 PM6/17/11
to the-fellowship-...@googlegroups.com
Le 17/06/2011 11:10, Pierre-Yves David a écrit :
> On Thu, Jun 16, 2011 at 04:00:42PM +0200, Éric Araujo wrote:
>> The things that can operate on project names, tarballs, etc. are
>> commands;
>> […]
>> The things that operate on a source directory with a setup.cfg are commands
> Looks like your confused them too :-)

Ha ha :) So : The things that can operate on project names, tarballs,
etc. are actions; the things that operate on a source directory with a
setup.cfg are commands.

higery

unread,
Jun 21, 2011, 12:00:30 PM6/21/11
to the-fellowship-...@googlegroups.com
Hi, folks:


Through the above discussion, I think there are two things that we have made clear:

1) 'develop' is just a kind of command rather than an action
2) 'develop' should not be taken as an option of install
cause we 'develop' a project just from a local source directory instead of a PYPI or VCS, and if it's an option of install, it will restrict the valid arguments to install action (from Carl's saying)

So, I agree with Eric's suggestion of taking 'develop' as a command and using a kind of install_* as its name. And for me, I will not take the 'editable' as its keyword, because 'editable' is just an option of Easyinstall in setuptools, what's more, 'editable' does not have an obvious heuristic meaning and is a little confusing.


There are two candidates:
1) install_develop
2) install_development

The second one is better in my opinion, because we should take the last word after '_' as a noun instead of a verb in packaging(e.g. install_lib, install_scripts, install_data, install_distinfo) , and 'development' will let users get a bit thought that its project will be installed in a 'development mode', which is in keeping with the saying on setuptools' online documents and comments in its source code. Look at http://peak.telecommunity.com/DevCenter/setuptools#development-mode

But we also should clarify here that, although the name of develop command takes 'install' as the prefix, install_development should not be taken as a subcommand of the install action.


Do you agree with me? Any thoughts?


Best wishes to everyone,
higery


Pierre-Yves David

unread,
Jun 21, 2011, 1:02:37 PM6/21/11
to the-fellowship-...@googlegroups.com
On Wed, Jun 22, 2011 at 12:00:30AM +0800, higery wrote:
> But we also should clarify here that, although the name of develop command
> takes 'install' as the prefix, install_development should not be taken as a
> subcommand of the install action.

This seems a good reason not to choose ``install_`` as a prefix for this command.

signature.asc

Éric Araujo

unread,
Jun 21, 2011, 2:03:36 PM6/21/11
to the-fellowship-...@googlegroups.com
Le 21/06/2011 19:02, Pierre-Yves David a écrit :
> On Wed, Jun 22, 2011 at 12:00:30AM +0800, higery wrote:
>> But we also should clarify here that, although the name of develop command
>> takes 'install' as the prefix, install_development should not be taken as a
>> subcommand of the install action.
> This seems a good reason not to choose ``install_`` as a prefix for this command.

Agreed. Let’s keep a develop command for now, and we’ll see later if
it’s better to rename it to install_dist --develop.

Michael Foord

unread,
Jun 21, 2011, 6:51:43 PM6/21/11
to the-fellowship-...@googlegroups.com
On 21 June 2011 17:00, higery <shoulde...@gmail.com> wrote:
Hi, folks:


Through the above discussion, I think there are two things that we have made clear:

1) 'develop' is just a kind of command rather than an action
2) 'develop' should not be taken as an option of install
cause we 'develop' a project just from a local source directory instead of a PYPI or VCS, and if it's an option of install, it will restrict the valid arguments to install action (from Carl's saying)

So, I agree with Eric's suggestion of taking 'develop' as a command and using a kind of install_* as its name. And for me, I will not take the 'editable' as its keyword, because 'editable' is just an option of Easyinstall in setuptools, what's more, 'editable' does not have an obvious heuristic meaning and is a little confusing.


There are two candidates:
1) install_develop
2) install_development

The second one is better in my opinion,

You're saying we should move from a seven letter command (for which I didn't see any evidence proferred that it was confusing anyone in practise) to a 14 letter one? -1

Michael
 
because we should take the last word after '_' as a noun instead of a verb in packaging(e.g. install_lib, install_scripts, install_data, install_distinfo) , and 'development' will let users get a bit thought that its project will be installed in a 'development mode', which is in keeping with the saying on setuptools' online documents and comments in its source code. Look at http://peak.telecommunity.com/DevCenter/setuptools#development-mode

But we also should clarify here that, although the name of develop command takes 'install' as the prefix, install_development should not be taken as a subcommand of the install action.


Do you agree with me? Any thoughts?


Best wishes to everyone,
higery


Tarek Ziadé

unread,
Jun 21, 2011, 7:08:56 PM6/21/11
to the-fellowship-...@googlegroups.com
On Wed, Jun 22, 2011 at 12:51 AM, Michael Foord <fuzz...@gmail.com> wrote:
>
>
> On 21 June 2011 17:00, higery <shoulde...@gmail.com> wrote:
>>
>> Hi, folks:
>>
>>
>> Through the above discussion, I think there are two things that we have
>> made clear:
>>
>> 1) 'develop' is just a kind of command rather than an action
>> 2) 'develop' should not be taken as an option of install
>> cause we 'develop' a project just from a local source directory instead of
>> a PYPI or VCS, and if it's an option of install, it will restrict the valid
>> arguments to install action (from Carl's saying)
>>
>> So, I agree with Eric's suggestion of taking 'develop' as a command and
>> using a kind of install_* as its name. And for me, I will not take the
>> 'editable' as its keyword, because 'editable' is just an option of
>> Easyinstall in setuptools, what's more, 'editable' does not have an obvious
>> heuristic meaning and is a little confusing.
>>
>>
>> There are two candidates:
>> 1) install_develop
>> 2) install_development
>>
>> The second one is better in my opinion,
>
> You're saying we should move from a seven letter command (for which I didn't
> see any evidence proferred that it was confusing anyone in practise) to a 14
> letter one? -1

For the command name itself it does not really matter imo.

now I really think we want an action for this in pysetup, and well, a
pysetup install option seemed nice:

$ pysetup install --inplace

I am not sure why this would be a real problem

Cheers
Tarek
--
Tarek Ziadé | http://ziade.org

Tarek Ziadé

unread,
Jun 21, 2011, 7:12:01 PM6/21/11
to the-fellowship-...@googlegroups.com

Because people don't want to run sudo to install something inplace..
which makes sense

Installing something inplace is something a user does for himself, so
there's a notion of "local"

How would you implement Setuptools' develop without touching the main
site-packages at all ?


>
> Carl

higery

unread,
Jun 21, 2011, 9:24:27 PM6/21/11
to the-fellowship-...@googlegroups.com
How would you implement Setuptools' develop without touching the main
site-packages at all ?

I think it's impossible if we just use the pure code. The 'developed' project source should always be importable, so we should write the path information into a .pth file on site-packages or the os environment variable.

As Tarek said, to activate a project, we will always configure the environment variables manually rather than using code, so do you think it's a little boring? We can append a path to os.environ in the source code, but it would not work across shells and platforms.

Then how about just creating a kind of 'packaing.pth' file in site-packages, and write the source directory path to it when 'develop' is used?


Regards,
higery

 

Carl Meyer

unread,
Jun 22, 2011, 12:08:14 AM6/22/11
to the-fellowship-...@googlegroups.com
Hi Tarek,

On 06/21/2011 06:12 PM, Tarek Ziad� wrote:
>>> - works with the notion of "project environment"
>>> - have one pth file per project in the user home
>>> - allow the addition or removal of directories in those pth files
>>> - allow the activation of one "project environment" by using
>>> environment variables
>>
>> I don't like this approach. Again, the entire concept of "project
>> environment" and activation of them seems like a half-baked re-
>> implementation of virtualenv, and I don't see the point of that. I
>> already use virtualenv and will continue to use it, I don't want d2 to
>> start trying to manage these pseudo-virtualenvs on my behalf. That
>> will just make my workflow more complicated and difficult.
>>
>> Basically, I want something much closer to setuptools' existing
>> develop-install, just tweaked a bit for robustness. Setuptools'
>> existing approach is widely used and useful; I don't see why it needs
>> this complete re-thinking.
>
> Because people don't want to run sudo to install something inplace..
> which makes sense

Which people? Is there a record of such complaints somewhere? I haven't
heard any such comments regarding pip's "install --editable", which is
just a wrapper around "setup.py develop". But if we made it impossible
to "install --editable" into system site-packages, I would guess we
would have complaints about that unnecessary restriction.

> Installing something inplace is something a user does for himself, so
> there's a notion of "local"

I don't agree. The notion of installing in-place is orthogonal to the
question of whether that install happens at the system level, the user
level, or the virtualenv level.

I have seen people use "pip install --editable" to install things in
their system Python: why would you want to outlaw such usage?

> How would you implement Setuptools' develop without touching the main
> site-packages at all ?

Why would I want to? If someone wants to develop-install something into
the system site-packages, they should be perfectly able to do that (and
it should require sudo). If they don't want to do that, they use --user
or a virtualenv. No different from a regular install.

Carl

Alain Leufroy

unread,
Jun 22, 2011, 3:52:02 AM6/22/11
to the-fellowship-...@googlegroups.com
> For the command name itself it does not really matter imo.
>
> now I really think we want an action for this in pysetup, and well, a
> pysetup install option seemed nice:
>
> $ pysetup install --inplace
>
> I am not sure why this would be a real problem

Is this option shall be deferent from the common '--prefix' command ?

--
Alain Leufroy
LOGILAB, Paris (France)
http://www.logilab.fr
Informatique scientifique & gestion de connaissances

Tarek Ziadé

unread,
Jun 22, 2011, 3:58:50 AM6/22/11
to the-fellowship-...@googlegroups.com
On Wed, Jun 22, 2011 at 9:52 AM, Alain Leufroy <alain....@logilab.fr> wrote:
>> For the command name itself it does not really matter imo.
>>
>> now I really think we want an action for this in pysetup, and well, a
>> pysetup install option seemed nice:
>>
>> $ pysetup install --inplace
>>
>> I am not sure why this would be a real problem
>
> Is this option shall be deferent from the common '--prefix' command ?

well obviously, some options will be incompatible with each other, but
that's just a set of check in the action I guess

>
>
>
> --
> Alain Leufroy
> LOGILAB, Paris (France)
> http://www.logilab.fr
> Informatique scientifique & gestion de connaissances
>

--
Tarek Ziadé | http://ziade.org

Tarek Ziadé

unread,
Jun 22, 2011, 4:33:54 AM6/22/11
to the-fellowship-...@googlegroups.com
On Wed, Jun 22, 2011 at 6:08 AM, Carl Meyer <ca...@oddbird.net> wrote:
...

>> Because people don't want to run sudo to install something inplace..
>> which makes sense
>
> Which people? Is there a record of such complaints somewhere?

Me in the first place. Virtualenv != develop


> I haven't
> heard any such comments regarding pip's "install --editable", which is
> just a wrapper around "setup.py develop". But if we made it impossible
> to "install --editable" into system site-packages, I would guess we
> would have complaints about that unnecessary restriction.

I am not sure to follow this. I think we need to define what's the
develop command and set aside virtualenv for a second.

When I run a develop on a package to work on it, I am not necessarily
planning to share it with the whole system, with everyone. I am not
either necessarily willing to use virtualenv.

So the use case is:

1/ I am in a project source tree
2/ I run "develop" in it
3/ the project seems installed, for me and potentially for the others,
I can work on it, because it's in my python path

That's a different use case than virtualenv, where I install a whole
environment for one or several packages etc.

So, how would you implement this ?

Cheers
Tarek

Alain Leufroy

unread,
Jun 22, 2011, 4:40:20 AM6/22/11
to the-fellowship-...@googlegroups.com
Hi,


>>>> - works with the notion of "project environment"
>>>> - have one pth file per project in the user home
>>>> - allow the addition or removal of directories in those pth files
>>>> - allow the activation of one "project environment" by using
>>>> environment variables
>>>
>>> I don't like this approach. Again, the entire concept of "project
>>> environment" and activation of them seems like a half-baked re-
>>> implementation of virtualenv, and I don't see the point of that. I
>>> already use virtualenv and will continue to use it, I don't want d2 to
>>> start trying to manage these pseudo-virtualenvs on my behalf. That
>>> will just make my workflow more complicated and difficult.
>>>
>>> Basically, I want something much closer to setuptools' existing
>>> develop-install, just tweaked a bit for robustness. Setuptools'
>>> existing approach is widely used and useful; I don't see why it needs
>>> this complete re-thinking.

+1

>> Because people don't want to run sudo to install something inplace..
>> which makes sense
> Which people? Is there a record of such complaints somewhere? I haven't
> heard any such comments regarding pip's "install --editable", which is
> just a wrapper around "setup.py develop". But if we made it impossible
> to "install --editable" into system site-packages, I would guess we
> would have complaints about that unnecessary restriction.
>
>> Installing something inplace is something a user does for himself, so
>> there's a notion of "local"
>
> I don't agree. The notion of installing in-place is orthogonal to the
> question of whether that install happens at the system level, the user
> level, or the virtualenv level.
>
> I have seen people use "pip install --editable" to install things in
> their system Python: why would you want to outlaw such usage?
>
>> How would you implement Setuptools' develop without touching the main
>> site-packages at all ?
>
> Why would I want to? If someone wants to develop-install something into
> the system site-packages, they should be perfectly able to do that (and
> it should require sudo). If they don't want to do that, they use --user
> or a virtualenv. No different from a regular install.
>


First of all, I think that making the system 'under development' is definitively
a bad idea. It just a point of view about the concept which take sens for the
command (or action) label/option. So, It may be a bad idea to join 'install' and
'develop', and 'pysetup install_*' or 'pysetup install --*' are dangerous
(taking appart that it allows any code to be executed with powered privilege).
For example, I see many people saying that 'develop' is interesting as it sets
the development environment close to de production. However this is not true
beacause it just adds a link to the dev folder (far from an trully installed
package).

Lets see what can be "develop":

1. download sources from PyPI or from control version with usefull shortcut to
define revision, tag, version ...
2. add an entry to the .pth in the site-packages folder
3. do not remove the source folder

So, what about spliting 'develop' ?

1. => setup source packagname/url
2. => pytsetup paths --add/remove/check/user/system

By this way, the system can be set 'under dev' if needed. ``pysetup paths`` will
be able to manage python system paths or user system paths.

Pierre-Yves David

unread,
Jun 22, 2011, 7:03:26 AM6/22/11
to the-fellowship-...@googlegroups.com

I would line up with several other response on this thread.

*** Leaves the system sites-package alone when you develop! ***

I think the develop command should default to alter an *as much local as
possible* .pth by default. Specific flag could be added to enable the package
at a wider scope.

Adding all developed command in "developing.pth" dedicated to that purpose
seems the way to go.

The main question is which path to used for this "developing.pth". Here is a
small brain storming about way to find it.

1) looking for a "developing.pth" in the parent directory of the project directory,
2) doing (1) recursively,
3) looking for a "DEVELOPINGPTH" env variable (that specific tool like virtual would set),
4) looking for a "developing.pth" the user home directory,
5) iterate over sys.path until you find a "developing.pth" file.

Various notes:
-----------------

Although solutions (1)//(2) (with (3) in backup) are probably very
convenient:
- those solutions require to alter import mechanism of python to look a
"developing.pth" in the current directory hierarchy.
- this can be a security threats
- could have some import performance impact by looking in multiple
directory (but it's basically O(1) and have probably less impact that
happening tones of directory to sys.path).

(1)//(2) allow to just remove a directory hierarchy without worring about
side effect.

By default we won't create "developing.pth" file except for (4) (and (1)
with a specific flag.)

My view
-----------

* I like the combo of (3) + (4),
* (1) is tempting,
* A better (5) could much simpler (and reused existing and well tested mechanism)

-----

For information, not using egg I never had a imperative needs for the develop
commands. I always used the PYTHONPATH environment variable either adding path
to PYTHONPATH or making symbolic link directory part of my user PYTHONPATH. It
work damn well.


I
see the `pysetup develop`[1] command as

(1) an helper to including//excluding source project in sys.path.
(instead of adding symlink to ~/pythonpath by hand)
(2) a way to transport the including//excluding information across various shell.
(where adding stuff to PYTHON requires to alter ~/.profile and reset//source all shells)


--
Pierre-Yves David

http://www.logilab.fr/


[1] whatever the name will be.

signature.asc

Carl Meyer

unread,
Jun 22, 2011, 11:13:39 AM6/22/11
to the-fellowship-...@googlegroups.com
Hi Tarek,

On 06/22/2011 03:33 AM, Tarek Ziad� wrote:
> On Wed, Jun 22, 2011 at 6:08 AM, Carl Meyer <ca...@oddbird.net> wrote:
> ...
>>> Because people don't want to run sudo to install something inplace..
>>> which makes sense
>>
>> Which people? Is there a record of such complaints somewhere?
>
> Me in the first place.

I've seen numerous uses of "setup.py develop" in the wild that your
proposal would break. It is widely used today to install unreleased
versions of dependencies for actual web project deployments, direct from
DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
day about whether that's a good practice or not (I don't do it myself),
but ultimately that's a pointless discussion: the fact is that many,
many people find it useful and are using it in that way. I don't think
it is the job of packaging to tell them they are wrong and outlaw their use.

I will give you one example of a similar use I do make of it. I have a
private VPS that hosts a number of Mercurial repositories. I have some
custom plugins and hooks that need to run for all users of that server.
These plugins and hooks are frequently updated and I don't want to have
to bump version numbers and make releases all the time. So I just have
an easily-updateable clone of the repo containing the plugins and hooks,
and it is "setup.py develop" installed into the system site-packages, so
that the system Mercurial can see it.

"setup.py develop" is an existing feature in setuptools/distribute; I'd
venture to say that apart from install_requires it is the most popular
feature of setuptools/distribute. We are not designing something new in
a vacuum. It's our responsibility to actually look around and see how
people are using the existing feature, and take that as a guide for what
the feature ought to do.

>> I haven't
>> heard any such comments regarding pip's "install --editable", which is
>> just a wrapper around "setup.py develop". But if we made it impossible
>> to "install --editable" into system site-packages, I would guess we
>> would have complaints about that unnecessary restriction.
>
> I am not sure to follow this. I think we need to define what's the
> develop command and set aside virtualenv for a second.

I didn't say anything about virtualenv here. I am saying that you are
proposing to take an existing widely-used feature of
setuptools/distribute (and by extension, pip), and make it more
complicated and less flexible. The burden of proof is on you to
demonstrate why it is critical that we make these changes, which will
break existing uses of the feature. "Me for one" is not adequate - it
indicates that you have one particular narrow view of the purpose of
"develop", and you want to design a new complex and limited feature
around your particular use-case, without considering the wide variety of
already-existing uses in the wild.

> When I run a develop on a package to work on it, I am not necessarily
> planning to share it with the whole system, with everyone.

So install it with --user in ~/.local, then, just like you would for any
installation that you didn't want to share with all users.

> I am not either necessarily willing to use virtualenv.

So don't, if you don't want to.

But your original proposal for "develop" said you wanted a "project
environment" that can be "activated". This is a precise description of
virtualenv. If that's what you want, I don't understand why you would
propose to re-implement a limited version of virtualenv instead of just
using it.

> So the use case is:
>
> 1/ I am in a project source tree
> 2/ I run "develop" in it
> 3/ the project seems installed, for me and potentially for the others,
> I can work on it, because it's in my python path

If you want it to seem installed for both you and others, then
develop-install it into the one place where all of you will see it: the
site-packages for the Python install you are all using.

If you want it to seem installed for just you, then install it into your
~/.local.

> That's a different use case than virtualenv, where I install a whole
> environment for one or several packages etc.

Your case is not a different use case than virtualenv, it's a subset of
the possible use cases for virtualenv.

> So, how would you implement this ?

As I've said, I would keep the question of "location and scope of
installation - i.e. *where* is something installed" entirely orthogonal
from the question of *how* it's installed (i.e. develop-install vs
normal install). By doing this we give people maximum flexibility to
mix-and-match for their use case, rather than limiting them to only your
specific use case.

Python already has options for setting *location and scope* of
installation. You can install to the site-packages for the python
installation, and then everyone who uses that installation will see it.
You can install to ~/.local/, and then only you will see it. If you want
more fine-grained control, you can use a virtualenv (this is so widely
used it may as well be a built-in feature of Python, and by the looks of
it in 3.3 it will be, thanks to Vinay's work). If you think there should
be another *location and scope* option that we don't currently have,
please outline what it is (but keep it separate from the question of
*how* a package is installed).

And I would implement *develop* as simply an installation which links
(whether via FS link or pth file) the in-place development checkout
files rather than copying them to the installation destination, but is
otherwise just like a normal install, and has no special opinions or
restrictions regarding where it is installed to.

The combination of these two orthogonal sets of simple features allows
you to have your use case, and allows others to mix and match
differently for their use cases. This is better than having complicated
features that muddle orthogonal concepts together.

I think I've said this now a couple times as clearly as I know how - if
it's still not making sense, perhaps we should take this to
distutils-sig, perhaps some broader input could clarify the issues.

Carl

Alain Leufroy

unread,
Jun 22, 2011, 12:06:42 PM6/22/11
to the-fellowship-...@googlegroups.com
On Wed, 22 Jun 2011, Carl Meyer wrote:

> Date: Wed, 22 Jun 2011 10:13:39 -0500
> From: Carl Meyer <ca...@oddbird.net>
> Reply-To: the-fellowship-...@googlegroups.com
> To: the-fellowship-...@googlegroups.com
> Subject: Re: Question about the 'develop' command


>
> Hi Tarek,
>
> On 06/22/2011 03:33 AM, Tarek Ziad� wrote:
>> On Wed, Jun 22, 2011 at 6:08 AM, Carl Meyer <ca...@oddbird.net> wrote:
>> ...
>>>> Because people don't want to run sudo to install something inplace..
>>>> which makes sense
>>>
>>> Which people? Is there a record of such complaints somewhere?
>>
>> Me in the first place.
>
> I've seen numerous uses of "setup.py develop" in the wild that your
> proposal would break. It is widely used today to install unreleased
> versions of dependencies for actual web project deployments, direct from
> DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
> day about whether that's a good practice or not (I don't do it myself),
> but ultimately that's a pointless discussion: the fact is that many,
> many people find it useful and are using it in that way. I don't think
> it is the job of packaging to tell them they are wrong and outlaw their use.

"many people is doing wrong things" is not a good argument.
Pysetup is simple to use. Updating a package is not more complicated than
updating the repo (merging too), less than security lacks.

>
> I will give you one example of a similar use I do make of it. I have a
> private VPS that hosts a number of Mercurial repositories. I have some
> custom plugins and hooks that need to run for all users of that server.
> These plugins and hooks are frequently updated and I don't want to have
> to bump version numbers and make releases all the time. So I just have
> an easily-updateable clone of the repo containing the plugins and hooks,
> and it is "setup.py develop" installed into the system site-packages, so
> that the system Mercurial can see it.

It is a bad develop usage. There are many better/easyer/saftyer solutions for
that: small script/Hg hook/cron that increases the version number/add a
tag/update installed packages.

You cannot share in prod changesets that are under heavy development.

If you realy want it, it is better to add a #changeset (instead of version tag)
attribute to url checking in pysetup


>
> "setup.py develop" is an existing feature in setuptools/distribute; I'd
> venture to say that apart from install_requires it is the most popular
> feature of setuptools/distribute. We are not designing something new in
> a vacuum. It's our responsibility to actually look around and see how
> people are using the existing feature, and take that as a guide for what
> the feature ought to do.

Same as point 1. It is better to change the feature.
There is a need and a nasty solution, it's time to improve it.

>>> I haven't
>>> heard any such comments regarding pip's "install --editable", which is
>>> just a wrapper around "setup.py develop". But if we made it impossible
>>> to "install --editable" into system site-packages, I would guess we
>>> would have complaints about that unnecessary restriction.
>>
>> I am not sure to follow this. I think we need to define what's the
>> develop command and set aside virtualenv for a second.
>
> I didn't say anything about virtualenv here. I am saying that you are
> proposing to take an existing widely-used feature of
> setuptools/distribute (and by extension, pip), and make it more
> complicated and less flexible. The burden of proof is on you to
> demonstrate why it is critical that we make these changes, which will
> break existing uses of the feature. "Me for one" is not adequate - it
> indicates that you have one particular narrow view of the purpose of
> "develop", and you want to design a new complex and limited feature
> around your particular use-case, without considering the wide variety of
> already-existing uses in the wild.

edit a site-package/nasty.pth then 'build --inplace': not more complicated
Or make a clone of you repository in site-packages, then push modificatons

>
>> When I run a develop on a package to work on it, I am not necessarily
>> planning to share it with the whole system, with everyone.
> So install it with --user in ~/.local, then, just like you would for any
> installation that you didn't want to share with all users.
>
>> I am not either necessarily willing to use virtualenv.
>
> So don't, if you don't want to.
>
> But your original proposal for "develop" said you wanted a "project
> environment" that can be "activated". This is a precise description of
> virtualenv. If that's what you want, I don't understand why you would
> propose to re-implement a limited version of virtualenv instead of just
> using it.

+1

>
>> So the use case is:
>>
>> 1/ I am in a project source tree
>> 2/ I run "develop" in it
>> 3/ the project seems installed, for me and potentially for the others,
>> I can work on it, because it's in my python path
>
> If you want it to seem installed for both you and others, then
> develop-install it into the one place where all of you will see it: the
> site-packages for the Python install you are all using.
>
> If you want it to seem installed for just you, then install it into your
> ~/.local.
>
>> That's a different use case than virtualenv, where I install a whole
>> environment for one or several packages etc.
>
> Your case is not a different use case than virtualenv, it's a subset of
> the possible use cases for virtualenv.
>
>> So, how would you implement this ?
>
> As I've said, I would keep the question of "location and scope of
> installation - i.e. *where* is something installed" entirely orthogonal
> from the question of *how* it's installed (i.e. develop-install vs
> normal install). By doing this we give people maximum flexibility to
> mix-and-match for their use case, rather than limiting them to only your
> specific use case.

Flexibility is already ok (.pth, pythonstartup, PYTHONPATH, sys.path,
update, links ...). It is not a question about flexibility, it is about
simplicity.

> Python already has options for setting *location and scope* of
> installation. You can install to the site-packages for the python
> installation, and then everyone who uses that installation will see it.
> You can install to ~/.local/, and then only you will see it. If you want
> more fine-grained control, you can use a virtualenv (this is so widely
> used it may as well be a built-in feature of Python, and by the looks of
> it in 3.3 it will be, thanks to Vinay's work). If you think there should
> be another *location and scope* option that we don't currently have,
> please outline what it is (but keep it separate from the question of
> *how* a package is installed).
>
> And I would implement *develop* as simply an installation which links
> (whether via FS link or pth file) the in-place development checkout
> files rather than copying them to the installation destination, but is
> otherwise just like a normal install, and has no special opinions or
> restrictions regarding where it is installed to.
>
> The combination of these two orthogonal sets of simple features allows
> you to have your use case, and allows others to mix and match
> differently for their use cases. This is better than having complicated
> features that muddle orthogonal concepts together.
>
> I think I've said this now a couple times as clearly as I know how - if
> it's still not making sense, perhaps we should take this to
> distutils-sig, perhaps some broader input could clarify the issues.

It seems that 'develop' only simplify the importation paths settings.
So, a better solution is to implement a feature that help on this.

>
> Carl

Carl Meyer

unread,
Jun 22, 2011, 12:59:33 PM6/22/11
to the-fellowship-...@googlegroups.com
Hi Alain,

On 06/22/2011 11:06 AM, Alain Leufroy wrote:
> On Wed, 22 Jun 2011, Carl Meyer wrote:
>> I've seen numerous uses of "setup.py develop" in the wild that your
>> proposal would break. It is widely used today to install unreleased
>> versions of dependencies for actual web project deployments, direct from
>> DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
>> day about whether that's a good practice or not (I don't do it myself),
>> but ultimately that's a pointless discussion: the fact is that many,
>> many people find it useful and are using it in that way. I don't think
>> it is the job of packaging to tell them they are wrong and outlaw
>> their use.
>
> "many people is doing wrong things" is not a good argument.
> Pysetup is simple to use. Updating a package is not more complicated than
> updating the repo (merging too), less than security lacks.

Not really true if you consider the hassle of having to make and release
the package, too (especially if it's a third-party package and there
hasn't been a release recently).

As I said, I think it's pointless to argue this. People have reasons for
doing what they do - because they find it easier and simpler, and
perhaps aren't aware of the downsides. If pysetup makes it harder to do
what they are already doing successfully with setuptools/distribute,
guess what their answer will be? Don't switch to pysetup, keep using
setuptools/distribute.

Even if we all agree that develop-installing stuff into site-packages on
a server is a bad idea, that doesn't mean there aren't still good use
cases for doing it on a local shared development machine/VM, or on a
custom-compiled version of Python in your home directory, or...

I think it is useful to be able to easily do in-place-editable
installations of Python packages, and I don't see any reason why the
target location of such installations should be forcibly restricted.
We're consenting adults here.

IOW, I don't think there is a problem with the existing
setuptools/distribute "develop" feature that needs to be fixed (except
more detailed issues with how it interacts with the current package_dirs
setup kwarg).

>> I will give you one example of a similar use I do make of it. I have a
>> private VPS that hosts a number of Mercurial repositories. I have some
>> custom plugins and hooks that need to run for all users of that server.
>> These plugins and hooks are frequently updated and I don't want to have
>> to bump version numbers and make releases all the time. So I just have
>> an easily-updateable clone of the repo containing the plugins and hooks,
>> and it is "setup.py develop" installed into the system site-packages, so
>> that the system Mercurial can see it.
>
> It is a bad develop usage. There are many better/easyer/saftyer
> solutions for that: small script/Hg hook/cron that increases the version
> number/add a tag/update installed packages.

Perhaps better and safer - certainly not easier.

> You cannot share in prod changesets that are under heavy development.

Sure you can - there are tradeoffs to doing so. You have to evaluate the
tradeoffs in your own case.

> If you realy want it, it is better to add a #changeset (instead of
> version tag)
> attribute to url checking in pysetup

This is what pip's VCS backends already provide as a wrapper around the
existing setuptools/distribute "develop" feature.

>> "setup.py develop" is an existing feature in setuptools/distribute; I'd
>> venture to say that apart from install_requires it is the most popular
>> feature of setuptools/distribute. We are not designing something new in
>> a vacuum. It's our responsibility to actually look around and see how
>> people are using the existing feature, and take that as a guide for what
>> the feature ought to do.
>
> Same as point 1. It is better to change the feature.
> There is a need and a nasty solution, it's time to improve it.

So what is your proposal for improvement, and what problem with the
existing feature would it solve?

>> As I've said, I would keep the question of "location and scope of
>> installation - i.e. *where* is something installed" entirely orthogonal
>> from the question of *how* it's installed (i.e. develop-install vs
>> normal install). By doing this we give people maximum flexibility to
>> mix-and-match for their use case, rather than limiting them to only your
>> specific use case.
>
> Flexibility is already ok (.pth, pythonstartup, PYTHONPATH, sys.path,
> update, links ...). It is not a question about flexibility, it is about
> simplicity.

Right. The develop command as it already exists in setuptools/distribute
is nothing more than a convenience for easily making use of existing
Python features. It doesn't do anything you couldn't do yourself with a
bit of work.

I think it is a useful convenience; much easier than manually setting up
pth files or PYTHONPATH. I would like to see this convenience remain as
flexible as possible, and not get bogged down with restrictions due to
one specific vision of how it ought to be used.

>> Python already has options for setting *location and scope* of
>> installation. You can install to the site-packages for the python
>> installation, and then everyone who uses that installation will see it.
>> You can install to ~/.local/, and then only you will see it. If you want
>> more fine-grained control, you can use a virtualenv (this is so widely
>> used it may as well be a built-in feature of Python, and by the looks of
>> it in 3.3 it will be, thanks to Vinay's work). If you think there should
>> be another *location and scope* option that we don't currently have,
>> please outline what it is (but keep it separate from the question of
>> *how* a package is installed).
>>
>> And I would implement *develop* as simply an installation which links
>> (whether via FS link or pth file) the in-place development checkout
>> files rather than copying them to the installation destination, but is
>> otherwise just like a normal install, and has no special opinions or
>> restrictions regarding where it is installed to.
>>
>> The combination of these two orthogonal sets of simple features allows
>> you to have your use case, and allows others to mix and match
>> differently for their use cases. This is better than having complicated
>> features that muddle orthogonal concepts together.
>>
>> I think I've said this now a couple times as clearly as I know how - if
>> it's still not making sense, perhaps we should take this to
>> distutils-sig, perhaps some broader input could clarify the issues.
>
> It seems that 'develop' only simplify the importation paths settings.
> So, a better solution is to implement a feature that help on this.

Develop *is* a feature that helps with this. It lets you put your
current project checkout onto the path, in a durable and convenient way.
What is the problem with it that you are trying to solve?

Carl

alain

unread,
Jun 22, 2011, 5:56:56 PM6/22/11
to the-fellowship-...@googlegroups.com

Hi Carl,
----- Message d'origine -----

> Hi Alain,
>
> On 06/22/2011 11:06 AM, Alain Leufroy wrote:
> > On Wed, 22 Jun 2011, Carl Meyer wrote:
> > > I've seen numerous uses of "setup.py develop" in the wild that your
> > > proposal would break. It is widely used today to install unreleased
> > > versions of dependencies for actual web project deployments, direct
> > > from DVCS checkout, and updateable by simply "<vcs> pull". We could
> > > argue all day about whether that's a good practice or not (I don't
> > > do it myself), but ultimately that's a pointless discussion: the
> > > fact is that many, many people find it useful and are using it in
> > > that way. I don't think it is the job of packaging to tell them they
> > > are wrong and outlaw their use.
> >
> > "many people is doing wrong things" is not a good argument.
> > Pysetup is simple to use. Updating a package is not more complicated
> > than updating the repo (merging too), less than security lacks.
>
> Not really true if you consider the hassle of having to make and release
> the package, too (especially if it's a third-party package and there
> hasn't been a release recently).
>
> As I said, I think it's pointless to argue this. People have reasons for
> doing what they do - because they find it easier and simpler, and
> perhaps aren't aware of the downsides. If pysetup makes it harder to do
> what they are already doing successfully with setuptools/distribute,
> guess what their answer will be? Don't switch to pysetup, keep using
> setuptools/distribute.

I'd like to answer yes. It's a nasty solution and we're talking to push it in the standard library. We can make an external tools that doing it, based on packaging.


> Even if we all agree that develop-installing stuff into site-packages on
> a server is a bad idea, that doesn't mean there aren't still good use
> cases for doing it on a local shared development machine/VM, or on a
> custom-compiled version of Python in your home directory, or...

So there is no problem, for example, to put a clone there and to make an install --inplace


> I think it is useful to be able to easily do in-place-editable
> installations of Python packages, and I don't see any reason why the
> target location of such installations should be forcibly restricted.

Because system installed files shall be sure and not under development.


> We're consenting adults here.

It's not a good argument for me but it's not about computing :P

'develop' simply add/remove .pth file entries during first install/remove and do an install --inplace.
So, what about a "pysetup source" and a "pysetup paths"?
I think it's more powerfull and people will explicitly link underdev/non-secure code to the system.


>
> IOW, I don't think there is a problem with the existing
> setuptools/distribute "develop" feature that needs to be fixed (except
> more detailed issues with how it interacts with the current package_dirs
> setup kwarg).

It works well.


>
> > > I will give you one example of a similar use I do make of it. I have
> > > a private VPS that hosts a number of Mercurial repositories. I have
> > > some custom plugins and hooks that need to run for all users of that
> > > server. These plugins and hooks are frequently updated and I don't
> > > want to have to bump version numbers and make releases all the time.
> > > So I just have an easily-updateable clone of the repo containing the
> > > plugins and hooks, and it is "setup.py develop" installed into the
> > > system site-packages, so that the system Mercurial can see it.
> >
> > It is a bad develop usage. There are many better/easyer/saftyer
> > solutions for that: small script/Hg hook/cron that increases the
> > version number/add a tag/update installed packages.
>
> Perhaps better and safer - certainly not easier.

ok,
I personally add repos one place with a simple script that add them to the PYTHONPATH. I juste have to clone a new repo or update an existing one - no need to make 'develop' (for pure python package).


> > You cannot share in prod changesets that are under heavy development.
>
> Sure you can - there are tradeoffs to doing so. You have to evaluate the
> tradeoffs in your own case.
>
> > If you realy want it, it is better to add a #changeset (instead of
> > version tag)
> > attribute to url checking in pysetup
>
> This is what pip's VCS backends already provide as a wrapper around the
> existing setuptools/distribute "develop" feature.

It's a different feature for me. pip split develop process. It allows to specify an url where to find sources (instead of PyPI) and that url can point to a repo (allowing to automaticly up to a particular tag/branch ...). That quite awesome :D


> > > "setup.py develop" is an existing feature in setuptools/distribute;
> > > I'd venture to say that apart from install_requires it is the most
> > > popular feature of setuptools/distribute. We are not designing
> > > something new in a vacuum. It's our responsibility to actually look
> > > around and see how people are using the existing feature, and take
> > > that as a guide for what the feature ought to do.
> >
> > Same as point 1. It is better to change the feature.
> > There is a need and a nasty solution, it's time to improve it.
>
> So what is your proposal for improvement, and what problem with the
> existing feature would it solve?

I prefere to split develop:
- a "source" command for donwloding sources as pip do it
- a "paths" command for managing paths where to find python packages/module. Pierre Yves David has send an interresting mail about that.


> > > As I've said, I would keep the question of "location and scope of
> > > installation - i.e. *where* is something installed" entirely
> > > orthogonal from the question of *how* it's installed (i.e.
> > > develop-install vs normal install). By doing this we give people
> > > maximum flexibility to mix-and-match for their use case, rather than
> > > limiting them to only your specific use case.
> >
> > Flexibility is already ok (.pth, pythonstartup, PYTHONPATH, sys.path,
> > update, links ...). It is not a question about flexibility, it is about
> > simplicity.
>
> Right. The develop command as it already exists in setuptools/distribute
> is nothing more than a convenience for easily making use of existing
> Python features. It doesn't do anything you couldn't do yourself with a
> bit of work.

+1

>
> I think it is a useful convenience; much easier than manually setting up
> pth files or PYTHONPATH. I would like to see this convenience remain as
> flexible as possible, and not get bogged down with restrictions due to
> one specific vision of how it ought to be used.

+1
A good tool to manage paths shall be more powerfull and perhaps easier.

Doug Hellmann

unread,
Jun 23, 2011, 9:00:45 AM6/23/11
to the-fellowship-...@googlegroups.com

On Jun 22, 2011, at 12:08 AM, Carl Meyer wrote:

> Hi Tarek,


>
> On 06/21/2011 06:12 PM, Tarek Ziadé wrote:
>>>> - works with the notion of "project environment"
>>>> - have one pth file per project in the user home
>>>> - allow the addition or removal of directories in those pth files
>>>> - allow the activation of one "project environment" by using
>>>> environment variables
>>>
>>> I don't like this approach. Again, the entire concept of "project
>>> environment" and activation of them seems like a half-baked re-
>>> implementation of virtualenv, and I don't see the point of that. I
>>> already use virtualenv and will continue to use it, I don't want d2 to
>>> start trying to manage these pseudo-virtualenvs on my behalf. That
>>> will just make my workflow more complicated and difficult.
>>>
>>> Basically, I want something much closer to setuptools' existing
>>> develop-install, just tweaked a bit for robustness. Setuptools'
>>> existing approach is widely used and useful; I don't see why it needs
>>> this complete re-thinking.
>>
>> Because people don't want to run sudo to install something inplace..
>> which makes sense
>
> Which people? Is there a record of such complaints somewhere? I haven't
> heard any such comments regarding pip's "install --editable", which is
> just a wrapper around "setup.py develop". But if we made it impossible
> to "install --editable" into system site-packages, I would guess we
> would have complaints about that unnecessary restriction.

I have no such complaint. It makes sense to require sudo to install globally. There are several options for installing to other places that do not require higher privileges (~/.local, virtualenv, etc.). Each has a different purpose.

>
>> Installing something inplace is something a user does for himself, so
>> there's a notion of "local"
>
> I don't agree. The notion of installing in-place is orthogonal to the
> question of whether that install happens at the system level, the user
> level, or the virtualenv level.
>
> I have seen people use "pip install --editable" to install things in
> their system Python: why would you want to outlaw such usage?
>
>> How would you implement Setuptools' develop without touching the main
>> site-packages at all ?
>
> Why would I want to? If someone wants to develop-install something into
> the system site-packages, they should be perfectly able to do that (and
> it should require sudo). If they don't want to do that, they use --user
> or a virtualenv. No different from a regular install.

Indeed. Our application cannot work in a virtualenv (for a variety of reasons too complex to get into here). If I lose the ability to install into the global site-packages in "development mode", then the mode is entirely useless for me for this application and my team will have to create a completely new workflow.

Doug

Doug Hellmann

unread,
Jun 23, 2011, 9:05:48 AM6/23/11
to the-fellowship-...@googlegroups.com

On Jun 22, 2011, at 12:06 PM, Alain Leufroy wrote:

> On Wed, 22 Jun 2011, Carl Meyer wrote:
>
>> Date: Wed, 22 Jun 2011 10:13:39 -0500
>> From: Carl Meyer <ca...@oddbird.net>
>> Reply-To: the-fellowship-...@googlegroups.com
>> To: the-fellowship-...@googlegroups.com
>> Subject: Re: Question about the 'develop' command
>> Hi Tarek,
>>

>> On 06/22/2011 03:33 AM, Tarek Ziadé wrote:
>>> On Wed, Jun 22, 2011 at 6:08 AM, Carl Meyer <ca...@oddbird.net> wrote:
>>> ...
>>>>> Because people don't want to run sudo to install something inplace..
>>>>> which makes sense
>>>>
>>>> Which people? Is there a record of such complaints somewhere?
>>>
>>> Me in the first place.
>>
>> I've seen numerous uses of "setup.py develop" in the wild that your
>> proposal would break. It is widely used today to install unreleased
>> versions of dependencies for actual web project deployments, direct from
>> DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
>> day about whether that's a good practice or not (I don't do it myself),
>> but ultimately that's a pointless discussion: the fact is that many,
>> many people find it useful and are using it in that way. I don't think
>> it is the job of packaging to tell them they are wrong and outlaw their use.
>
> "many people is doing wrong things" is not a good argument.
> Pysetup is simple to use. Updating a package is not more complicated than
> updating the repo (merging too), less than security lacks.

As I said in an earlier email, our app doesn't work in virtualenv or under buildout. We use development mode with the global site-packages directory on development systems, then deploy using real packages to production. AFAICT, we're not "doing wrong things." It has certainly been working well for us for the past couple of years.

Doug

Alain Leufroy

unread,
Jun 23, 2011, 9:09:38 AM6/23/11
to the-fellowship-...@googlegroups.com
On Thu, 23 Jun 2011, Doug Hellmann wrote:

> Date: Thu, 23 Jun 2011 09:00:45 -0400
> From: Doug Hellmann <doug.h...@gmail.com>


> Reply-To: the-fellowship-...@googlegroups.com
> To: the-fellowship-...@googlegroups.com
> Subject: Re: Question about the 'develop' command
>
>

> On Jun 22, 2011, at 12:08 AM, Carl Meyer wrote:
>
>> Hi Tarek,
>>

No chance :P, manually add the entry in a .pth?

>
> Doug

Tarek Ziadé

unread,
Jun 23, 2011, 9:15:47 AM6/23/11
to the-fellowship-...@googlegroups.com
On Wed, Jun 22, 2011 at 5:13 PM, Carl Meyer <ca...@oddbird.net> wrote:
> Hi Tarek,
>
> On 06/22/2011 03:33 AM, Tarek Ziadé wrote:
>> On Wed, Jun 22, 2011 at 6:08 AM, Carl Meyer <ca...@oddbird.net> wrote:
>> ...
>>>> Because people don't want to run sudo to install something inplace..
>>>> which makes sense
>>>
>>> Which people? Is there a record of such complaints somewhere?
>>
>> Me in the first place.
>
> I've seen numerous uses of "setup.py develop" in the wild that your
> proposal would break.

why break ? can't we find a way so all use cases work ?

ISTM that you're saying: if you don't use sudo OR virtualenv OR
per-user site-package, you can't use develop. That sounds very
restrictive.

It boils down to : I want to "develop" a project without having to
install it in a site-package, or have super-powers.

> It is widely used today to install unreleased
> versions of dependencies for actual web project deployments, direct from
> DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
> day about whether that's a good practice or not (I don't do it myself),
> but ultimately that's a pointless discussion: the fact is that many,
> many people find it useful and are using it in that way. I don't think
> it is the job of packaging to tell them they are wrong and outlaw their use.

You are using string words here. No one said we wanted packaging to
tell people they are wrong :d
They're not wrong, they just use a tool the way it works to fulfill a use case.

I think we need to pause here, back up a little bit, and write down
the uses cases we want to fulfill,
then come back to the "how". Otherwise it's going to be an endless
discussion because we're all have ways to deal with this use cases
using our ways and preferred tools and istm that narrows the
discussion.

And bringing the discussion to distutils-sig is just going to dilute
the discussion

Could someone that works on develop start up a wiki page on the use
cases (without mentioning any tool or technique for now.)

Cheers
Tarek

Tarek Ziadé

unread,
Jun 23, 2011, 9:24:34 AM6/23/11
to the-fellowship-...@googlegroups.com
On Thu, Jun 23, 2011 at 3:05 PM, Doug Hellmann <doug.h...@gmail.com> wrote:
...

>
> As I said in an earlier email, our app doesn't work in virtualenv or under buildout. We use development mode with the global site-packages directory on development systems, then deploy using real packages to production. AFAICT, we're not "doing wrong things." It has certainly been working well for us for the past couple of years.

I am not sure who started with this "you're doing it wrong" ... :D

What I know is that all of us have ways to solve problems with the
existing eco-system, and that what works for some don't work for some
others. I really think we can come up with an elegant solution that
can work for all, but we're missing a list of use case to start up
with imho.

higery, since you're doing the GSOC for this, maybe you can start a
wiki page that collects all the uses cases

Cheers

higery

unread,
Jun 23, 2011, 9:45:05 AM6/23/11
to the-fellowship-...@googlegroups.com
higery, since you're doing the GSOC for this, maybe you can start a
wiki page that collects all the uses cases

OK. But I can not make sure the cases I collect is the right and real situation as guys showed in the mailing list, however, people can enhance after I complete a draft I think.


Regards,
higery



Tarek Ziadé

unread,
Jun 23, 2011, 9:53:44 AM6/23/11
to the-fellowship-...@googlegroups.com

yes I think the plan can be:

1. you initiate
2. people here add more details if they see missing use cases or mistakes
3. eventually we make sure we have a snapshot of "developer stories"
e.g. what everyone does to work with a package in "dev mode"


>
>
> Regards,
> higery

Pierre-Yves David

unread,
Jun 23, 2011, 9:54:40 AM6/23/11
to the-fellowship-...@googlegroups.com

Yes, start the process then show it to the python community and collect the
feedback.

signature.asc

Doug Hellmann

unread,
Jun 23, 2011, 1:33:01 PM6/23/11
to the-fellowship-...@googlegroups.com

Why would I do something manually that I can do with a command today? I don't understand why the existing behavior needs to change.

Doug

Doug Hellmann

unread,
Jun 23, 2011, 1:42:24 PM6/23/11
to the-fellowship-...@googlegroups.com

On Jun 23, 2011, at 9:15 AM, Tarek Ziadé wrote:

> On Wed, Jun 22, 2011 at 5:13 PM, Carl Meyer <ca...@oddbird.net> wrote:
>> Hi Tarek,
>>
>> On 06/22/2011 03:33 AM, Tarek Ziadé wrote:
>>> On Wed, Jun 22, 2011 at 6:08 AM, Carl Meyer <ca...@oddbird.net> wrote:
>>> ...
>>>>> Because people don't want to run sudo to install something inplace..
>>>>> which makes sense
>>>>
>>>> Which people? Is there a record of such complaints somewhere?
>>>
>>> Me in the first place.
>>
>> I've seen numerous uses of "setup.py develop" in the wild that your
>> proposal would break.
>
> why break ? can't we find a way so all use cases work ?
>
> ISTM that you're saying: if you don't use sudo OR virtualenv OR
> per-user site-package, you can't use develop. That sounds very
> restrictive.
>
> It boils down to : I want to "develop" a project without having to
> install it in a site-package, or have super-powers.

I don't understand what that means. What would the new form of the command do different from the current form?

>> It is widely used today to install unreleased
>> versions of dependencies for actual web project deployments, direct from
>> DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
>> day about whether that's a good practice or not (I don't do it myself),
>> but ultimately that's a pointless discussion: the fact is that many,
>> many people find it useful and are using it in that way. I don't think
>> it is the job of packaging to tell them they are wrong and outlaw their use.
>
> You are using string words here. No one said we wanted packaging to
> tell people they are wrong :d
> They're not wrong, they just use a tool the way it works to fulfill a use case.
>
> I think we need to pause here, back up a little bit, and write down
> the uses cases we want to fulfill,
> then come back to the "how". Otherwise it's going to be an endless
> discussion because we're all have ways to deal with this use cases
> using our ways and preferred tools and istm that narrows the
> discussion.

Good idea.

I want to run some equivalent of the current "python setup.py develop" and have the project I am working on added to the "current" python path in a way that meets these requirements:

1. All applications using the "current" python interpreter see it, without having to use an environment variable (our app spawns sub-processes from a daemon, and changing the environment for the daemon isn't necessary now).
2. I do not need to manually edit any files to configure the path for "current" python interpreter (even if they are modified by running the "python setup.py develop" command.
3. An application using the "current" python interpreter does not need to be run from any particular directory on the filesystem in order to see the changes or my package (again, spawning sub-processes from daemons makes this difficult).
4. All entry points in my project, including console scripts, are registered correctly with the "current" interpreter.
5. I can edit files in my project and re-run the interpreter to have the changes picked up, without having to redo the "python setup.py develop" step.

In the above, when I say "current" for the interpreter, I mean the one I use when I run "python setup.py develop" (or its equivalent).

Can someone explain the proposed changes, and how they might (or might not) impact the above use case?

Tarek Ziadé

unread,
Jun 23, 2011, 1:54:54 PM6/23/11
to the-fellowship-...@googlegroups.com
On Thu, Jun 23, 2011 at 7:42 PM, Doug Hellmann <doug.h...@gmail.com> wrote:
...
>>
>> It boils down to : I want to "develop" a project without having to
>> install it in a site-package, or have super-powers.
>
> I don't understand what that means. What would the new form of the command do different from the current form?

Let you specify the scope of the develop. In other words where it gets
"installed" and what environments are impacted.

1 - do "develop" in a "project environment", no site-packages
involved, and just have the package included in the path
2 - do "develop" in a site-packages. So that can be in the global
Python one, the per-user one, or a virtualenv-ed one.

ISTM that #1 is what Alain and al. want, #2 is what Carl and you do
(with different flavors)

for 1- I don't want to define technically how it could work yet, just
say that one box can have n different projects.


>
>>> It is widely used today to install unreleased
>>> versions of dependencies for actual web project deployments, direct from
>>> DVCS checkout, and updateable by simply "<vcs> pull". We could argue all
>>> day about whether that's a good practice or not (I don't do it myself),
>>> but ultimately that's a pointless discussion: the fact is that many,
>>> many people find it useful and are using it in that way. I don't think
>>> it is the job of packaging to tell them they are wrong and outlaw their use.
>>
>> You are using string words here. No one said we wanted packaging to
>> tell people they are wrong :d
>> They're not wrong, they just use a tool the way it works to fulfill a use case.
>>
>> I think we need to pause here, back up a little bit, and write down
>> the uses cases we want to fulfill,
>> then come back to the "how". Otherwise it's going to be an endless
>> discussion because we're all have ways to deal with this use cases
>> using our ways and preferred tools and istm that narrows the
>> discussion.
>
> Good idea.
>
> I want to run some equivalent of the current "python setup.py develop" and have the project I am working on added to the "current" python path in a way that meets these requirements:
>
> 1. All applications using the "current" python interpreter see it, without having to use an environment variable (our app spawns sub-processes from a daemon, and changing the environment for the daemon isn't necessary now).
> 2. I do not need to manually edit any files to configure the path for "current" python interpreter (even if they are modified by running the "python setup.py develop" command.
> 3. An application using the "current" python interpreter does not need to be run from any particular directory on the filesystem in order to see the changes or my package (again, spawning sub-processes from daemons makes this difficult).
> 4. All entry points in my project, including console scripts, are registered correctly with the "current" interpreter.
> 5. I can edit files in my project and re-run the interpreter to have the changes picked up, without having to redo the "python setup.py develop" step.
>
> In the above, when I say "current" for the interpreter, I mean the one I use when I run "python setup.py develop" (or its equivalent).

How do you change the "current" python interpreter, if you have 2
separate projects ?

Doug Hellmann

unread,
Jun 23, 2011, 4:00:46 PM6/23/11
to the-fellowship-...@googlegroups.com

On Jun 23, 2011, at 1:54 PM, Tarek Ziadé wrote:

> On Thu, Jun 23, 2011 at 7:42 PM, Doug Hellmann <doug.h...@gmail.com> wrote:
> ...
>>>
>>> It boils down to : I want to "develop" a project without having to
>>> install it in a site-package, or have super-powers.
>>
>> I don't understand what that means. What would the new form of the command do different from the current form?
>
> Let you specify the scope of the develop. In other words where it gets
> "installed" and what environments are impacted.
>
> 1 - do "develop" in a "project environment", no site-packages
> involved, and just have the package included in the path
> 2 - do "develop" in a site-packages. So that can be in the global
> Python one, the per-user one, or a virtualenv-ed one.
>
> ISTM that #1 is what Alain and al. want, #2 is what Carl and you do
> (with different flavors)
>
> for 1- I don't want to define technically how it could work yet, just
> say that one box can have n different projects.

What would actually happen for #1? I don't understand how that is useful, so I'm sure I'm missing something.

Unfortunately for our application, that isn't possible. We take over the entire system, and have a single copy of the interpreter that must run globally (yeah, yeah, legacy issues). So, I run develop in two different sandboxes using the same interpreter, and it swaps the .pth entry in the global site-packages to point to the right place.

Normally, one would use something like virtualenv or buildout (I don't use buildout, so it may not apply, but it seems like it is in the category of tools that meet this need). With virtualenv, I really do want my site-packages updated to point to the development sandbox. As I said above, though, I don't really understand how it is useful to do anything else, so I may be missing something.

Doug

higery

unread,
Jun 23, 2011, 11:08:57 PM6/23/11
to the-fellowship-...@googlegroups.com
Hi, all:


I just created a simple page to collect the use cases of develop command , which we all talked about before in this list. The page mainly concerns the storis about what the user really want the develop command to do when a user uses it in his project. But I think this page is still not complete and not tell all the conditions that the develop command should behave, so don't hesitate to help enhance it, thank you.

Link:     http://wiki.python.org/moin/UsecasesOfDevelop


Best Regards,
higery

Doug Hellmann

unread,
Jun 24, 2011, 7:12:57 AM6/24/11
to the-fellowship-...@googlegroups.com
Thanks for collecting those stories. The versions based on my comments are accurate.

Doug

Reply all
Reply to author
Forward
0 new messages