Sharing code between different projects?

[andrea crotti]

I am in the situation where I am working on different projects that
might potentially share a lot of code.

I started to work on project A, then switched completely to project B
and in the transiction I copied over a lot of code with the
corresponding tests, and I started to modify it.

Now it's time to work again on project A, but I don't want to copy
things over again.

I would like to design a simple and nice way to share between projects,
where the things I want to share are simple but useful things as for
example:

class TempDirectory:
"""Create a temporary directory and cd to it on enter, cd back to
the original position and remove it on exit
"""
def __init__(self):
self.oldcwd = getcwd()
self.temp_dir = mkdtemp()

def __enter__(self):
logger.debug("create and move to temp directory %s" % self.temp_dir)
return self.temp_dir

def __exit__(self, type, value, traceback):
# I first have to move out
chdir(self.oldcwd)
logger.debug("removing the temporary directory and go back to
the original position %s" % self.temp_dir)
rmtree(self.temp_dir)


The problem is that there are functions/classes from many domains, so it
would not make much sense to create a real project, and the only name I
could give might be "utils or utilities"..

In plus the moment the code is shared I must take care of versioning and
how to link different pieces together (we use perforce by the way).

If then someone else except me will want to use these functions then of
course I'll have to be extra careful, designing really good API's and so
on, so I'm wondering where I should set the trade-off between ability to
share and burden to maintain..

Anyone has suggestions/real world experiences about this?

[Chris Angelico]

On Tue, Aug 14, 2012 at 2:53 AM, andrea crotti
<andrea....@gmail.com> wrote:
> The problem is that there are functions/classes from many domains, so it
> would not make much sense to create a real project, and the only name I
> could give might be "utils or utilities"..

There's actually much merit in a generic utilities module. Keep things
nicely segregated (ideally such that you know what things depend on
what other, but at very least keep track of where one ends and another
begins - that's trivial if everything's "one function" or "one class",
but less so when you have a family of related functions), and then you
can consider promoting one block of code to stand-alone module. But in
the meantime, you have a single module used in two places, even if it
doesn't have a very clear definition as yet.

ChrisA

[andrea crotti]

2012/8/13 Rob Day <rober...@merton.oxon.org>:
> I'd just create a module - called shared_utils.py or similar - and import
> that in both projects. It might be a bit messy if there's no 'unifying
> theme' to the module - but surely it'd be a lot less messy than your
> TempDirectory class, and anyone else who knows Python will understand
> 'import shared_utils' much more easily.
>
> I realise you might not want to say, but if you could give some idea what
> sort of projects these are, and what sorts of code you're trying to share,
> it might make things a bit clearer.
>
> I'm not really sure what your concerns about 'versioning and how to link
> different pieces together' are - what d you think could go wrong here?
>

It's actually not so simple..

Because the two projects live in different parts of the repository
with different people allowed to work on them, and they have to run on
different machines..

In plus I'm using perforce which doesn't have any svn:externals-like
thing as far as I know.. The thing I should do probably is to set up
workspace (which contains *absolute* paths of the machines) with the
right setting to make module available in the right position.

Second problem is that one of the two projects has a quite insane
requirement, which is to be able to re-run itself on a specific
version depending on a value fetched from the database.

This becomes harder if divide code around, but in theory I can use the
changeset number which is like a SVN revision so this should be fine.

The third problem is that from the moment is not just me using these
things, how can I be sure that changing something will not break
someone else code?

I have unit tests on both projects plus the tests for the utils, but
as soon as I separate them it becomes harder to test everything..

So well everything can have a solution probably, I just hope it's
worth the effort..

Another thing which would be quite cool might be a import hook which
fetches things from the repository when needed, with a simple
bootstrap script for every project to be able to use this feature, but
it only makes sense if I need this kind of feature in many projects.

[Jean-Michel Pichavant]

> The problem is that there are functions/classes from many domains, so it
> would not make much sense to create a real project, and the only name I
> could give might be "utils or utilities"..
>

> In plus the moment the code is shared I must take care of versioning and
> how to link different pieces together (we use perforce by the way).
>
> If then someone else except me will want to use these functions then of
> course I'll have to be extra careful, designing really good API's and so
> on, so I'm wondering where I should set the trade-off between ability to
> share and burden to maintain..
>
> Anyone has suggestions/real world experiences about this?
>

I can think of logilab-common (http://www.logilab.org/848/)

Having a company-wide python module properly distributed is one to
achieve your goal. Without distributing your module to the public,
there's a way to have a pypi-like server runnning on your private network :

http://pypi.python.org/pypi/pypiserver/

JM

Note : looks like pypi.python.org is having some trouble, the above link is broken. Search for recent announcement about pypiserver.

[Steven D'Aprano]

On Mon, 13 Aug 2012 17:53:32 +0100, andrea crotti wrote:

> I started to work on project A, then switched completely to project B
> and in the transiction I copied over a lot of code with the
> corresponding tests, and I started to modify it.
>
> Now it's time to work again on project A, but I don't want to copy
> things over again.
>
> I would like to design a simple and nice way to share between projects,
> where the things I want to share are simple but useful things as for
> example:

[...]

> The problem is that there are functions/classes from many domains, so it
> would not make much sense to create a real project, and the only name I
> could give might be "utils or utilities"..

I feel your pain. "Copy and paste" is perhaps the simplest, most
seductive, and *worst* of the developer anti-patterns.

I wish I had a solution to share with you, because I'm in the same
position, but I don't.

The best I have come up with is a combination of:

1) For really small code snippets, just forget about sharing code. If you
need a five-line function Foo in ten projects, just re-write them (well,
copy and paste them...) each time. Forget about trying to keep them
syncronised.


2) For bigger functions/classes, put them in their own module which you
can share across multiple projects.


3) Consider having a "utilities" module to include all the assorted bits
and pieces.


If your projects are in-house, then a utilities module makes more sense.
If you're writing libraries for independent release, not so much.



--
Steven

[andrea crotti]

2012/8/14 Jean-Michel Pichavant <jeanm...@sequans.com>:

>
> I can think of logilab-common (http://www.logilab.org/848/)
>
> Having a company-wide python module properly distributed is one to achieve
> your goal. Without distributing your module to the public, there's a way to
> have a pypi-like server runnning on your private network :
>
> http://pypi.python.org/pypi/pypiserver/
>
> JM
>
> Note : looks like pypi.python.org is having some trouble, the above link is
> broken. Search for recent announcement about pypiserver.
>

Thanks, yes we need something like this..
I'll copy the name probably, I prefer "common" to "utils/utilities"..

[Cameron Simpson]

On 13Aug2012 17:53, andrea crotti <andrea....@gmail.com> wrote:
| I am in the situation where I am working on different projects that
| might potentially share a lot of code.
|

| I started to work on project A, then switched completely to project B
| and in the transiction I copied over a lot of code with the
| corresponding tests, and I started to modify it.
|
| Now it's time to work again on project A, but I don't want to copy
| things over again.

[...]
| The problem is that there are functions/classes from many domains, so it
| would not make much sense to create a real project, and the only name I
| could give might be "utils or utilities"..
|

| In plus the moment the code is shared I must take care of versioning and
| how to link different pieces together (we use perforce by the way).

[...]

Having just skimmed this thread, one thing I haven't quite seen suggested is
this:

Really do make a third "utilities" project, and treat "the project" and
"deploy" as separate notions. So to actually run/deploy project A's code
you'd have a short script that copied project A and the utilities project
code into a tree and ran off that. Or even a simple process/script to
update the copy of "utilities" in "project A"'s area.

So you don't "share" code on an even handed basis but import the
"utilities" library into each project as needed.

I do this (one my own very small scale) in one of two ways:

- as needed, copy the desired revision of utilities into the project's
library space and do perforce's equivalent of Mercurial's addremove
on that library tree (comment "update utilities to revision X").

- keep a perforce work area for the utilities in your project A area,
where your working project A can hook into it with a symlink or some
deploy/copy procedure as suggested above.
With this latter one you can push back into the utilities library
from your "live" project, because you have a real checkout. So:

projectAdir
projectA-perforce-checkout
utilities-perforce-checkout
projectBdir
projectB-perforce-checkout
utilities-perforce-checkout

Personally I become more and more resistent to cut/paste even for small
things as soon as multiple people use it; you will never get to backport
updates to even trivial code to all the copies.

Cheers,
--
Cameron Simpson <c...@zip.com.au>

The mere existence of a problem is no proof of the existence of a solution.
- Yiddish Proverb

[andrea crotti]

2012/8/14 Cameron Simpson <c...@zip.com.au>:

>
> Having just skimmed this thread, one thing I haven't quite seen suggested is
> this:
>
> Really do make a third "utilities" project, and treat "the project" and
> "deploy" as separate notions. So to actually run/deploy project A's code
> you'd have a short script that copied project A and the utilities project
> code into a tree and ran off that. Or even a simple process/script to
> update the copy of "utilities" in "project A"'s area.
>
> So you don't "share" code on an even handed basis but import the
> "utilities" library into each project as needed.
>
> I do this (one my own very small scale) in one of two ways:
>
> - as needed, copy the desired revision of utilities into the project's
> library space and do perforce's equivalent of Mercurial's addremove
> on that library tree (comment "update utilities to revision X").
>
> - keep a perforce work area for the utilities in your project A area,
> where your working project A can hook into it with a symlink or some
> deploy/copy procedure as suggested above.
> With this latter one you can push back into the utilities library
> from your "live" project, because you have a real checkout. So:
>
> projectAdir
> projectA-perforce-checkout
> utilities-perforce-checkout
> projectBdir
> projectB-perforce-checkout
> utilities-perforce-checkout
>

Thanks, is more or less what I was going to do.. But I would not use
symlinks and similar things, because then every user should set it up
accordingly.

Potentially we could instead use the perforce API to change the
workspace mappings at run-time, and thus "force" perforce to checkout
the files in the right place..

There is still the problem that people should checkout things from two
places all the time instead of one..

> Personally I become more and more resistent to cut/paste even for small
> things as soon as multiple people use it; you will never get to backport
> updates to even trivial code to all the copies.
>
> Cheers,

Well sure, but on the other end as soon as multiple people use it you
can't change any of the public functions signatures without being
afraid that you'll break something..

[andrea crotti]

Also looking at logilab-common I thought that it would be great if we
could actually make this "common" library even open source, and use it
as one of the other many external libraries.

Since Python code is definitively not the the core business of this
company I might even convince them, but the problem is that then all
the internal people working on it would not be able to use the
standard tools that they use with everything else..

Did anyone manage to convince his company to do something similar?

[Miki Tebeka]

> In plus I'm using perforce which doesn't have any svn:externals-like

You can probably use views to this (http://www.perforce.com/perforce/r12.1/manuals/cmdref/o.views.html).

> Second problem is that one of the two projects has a quite insane
> requirement, which is to be able to re-run itself on a specific
> version depending on a value fetched from the database.

You can probably play with PYTHONPATH to do that.

> The third problem is that from the moment is not just me using these
> things, how can I be sure that changing something will not break
> someone else code?

That's always a problem with libraries you distribute, no matter what is the mechanism for distribution.

If you go the views/link way. You can link to tags instead of trunk (or the perforce equivalent) and then client can still to know "good" version. If you go the PyPi route, you can specify package version in setup.py dependencies (foo==0.10.2)

[Miki Tebeka]

> In plus I'm using perforce which doesn't have any svn:externals-like

You can probably use views to this (http://www.perforce.com/perforce/r12.1/manuals/cmdref/o.views.html).

> Second problem is that one of the two projects has a quite insane
> requirement, which is to be able to re-run itself on a specific
> version depending on a value fetched from the database.

You can probably play with PYTHONPATH to do that.

> The third problem is that from the moment is not just me using these
> things, how can I be sure that changing something will not break
> someone else code?

[Jean-Michel Pichavant]

SVN allows to define external dependencies, where one repository will
actually checkout another one at a specific version. If SVN does it, I
guess any decent SCM also provide such feature.

Assuming our project is named 'common', and you have 2 projects A and B :

A
- common@rev1

B
- common@rev2

Project A references the lib as "A.common", B as "B.common". You need to
be extra carefull to never reference common as 'common' in any place.

JM

[andrea crotti]

2012/8/16 Jean-Michel Pichavant <jeanm...@sequans.com>:

>
> SVN allows to define external dependencies, where one repository will
> actually checkout another one at a specific version. If SVN does it, I guess
> any decent SCM also provide such feature.
>
> Assuming our project is named 'common', and you have 2 projects A and B :
>
> A
> - common@rev1
>
> B
> - common@rev2
>
> Project A references the lib as "A.common", B as "B.common". You need to be
> extra carefull to never reference common as 'common' in any place.
>
> JM
>

Unfortunately I think you guess wrong
http://forums.perforce.com/index.php?/topic/553-perforce-svnexternals-equivalent/
Anyway with views and similar things is not that hard to implement the
same thing..

[andrea crotti]

2012/8/16 andrea crotti <andrea....@gmail.com>:

>
>
> Unfortunately I think you guess wrong
> http://forums.perforce.com/index.php?/topic/553-perforce-svnexternals-equivalent/
> Anyway with views and similar things is not that hard to implement the
> same thing..

I'm very happy to say that I finally made it!

It took 3 hours to move / merge a few thousand lines around but
everything seems to work perfectly now..

At the moment I'm just using symlinks, I'll see later if something
smarter is necessary, thanks to everyone for the ideas.

[yo...@cocycles.com]

I would try bit:
https://github.com/teambit/bit

Hope it helps.

[dieter]

> On Monday, August 13, 2012 at 7:53:32 PM UTC+3, andrea crotti wrote:
>> I am in the situation where I am working on different projects that
>> might potentially share a lot of code.
>>
>> I started to work on project A, then switched completely to project B
>> and in the transiction I copied over a lot of code with the
>> corresponding tests, and I started to modify it.
>>
>> Now it's time to work again on project A, but I don't want to copy
>> things over again.

I use so called "namespace packages" for this. Those are "virtual" packages
"containing" a set of related "real" packages - individually manageable
on PyPI. The "real" packages describe in their dependencies on which
other packages they depend, among them (potentially) "friend" packages.

An example is "dm.zope.rpc" and "dm.zope.rpc.wsdl_suds". In this case,
"dm.zope.rpc" provides general rpc infrastructure for the web
application framework Zope and implementations for some elementary
rpc protocols and "dm.zipe.rpc.wsdl_suds" uses this infrastructure
for the implementation of a WSDL based rpc protocol by means of "suds".