python module, virtualenv/pip handling

[inebri...@gmail.com]

RelEng puppet folks,

I'm a long time Mozilla user, and wanted to thank you for releasing the PuppetAgain code. We're running a modified (somewhat hacked to death by a colleague, unfortunately) version of the python module, and it's been immensely helpful.

I wanted to pass along a post that I made to puppet-users today, in case it's of any interest:

Hello, community,

I work for a python/Django shop (we run supposedly one of the largest
Django apps out there), and we're just starting to use Puppet for
handling python stuff (and hopefully application deploys, eventually).
We're currently using a hacked up version of Mozilla RelEng's excellent
and quite helpful python module to handle virtualenv creation and pip
package installation - but the module (or at least our version of it)
doesn't handle requirements files, and uses a define to pip install
packages, so a given package can only be installed in one venv on a node.

Is anyone aware of a better, more complete python/virtualenv module?
I've seen stankevich/python on the forge, which handles requirements
files but doesn't fix the issue with only being able to install a given
package once per node (which, AFAIK, can be fixed with a native provider
but not a defined type). If not, does anyone have interest in
collaborating or contributing to a more feature-complete
python/virtualenv module?

If anyone's interested in collaborating, or can suggest an existing
module (stankevich's? or moz releng?) to start with, I've spoken with
one of the pip/virtualenv maintainers and he's interested in finding
something suitable that could become the "official" module.

Thanks for any advice, input or suggestions,
Jason Antman
jantman
CMGdigital

[Justin Wood]

Hi,

Wonderful to hear of your interest in our "PuppetAgain" and its utility to you (that is one of the reasons we made it public from the start). That said as far as I can tell our define for our python module does use the venv directory itself as a part of our naming scheme for specific python packages [1].

Are you able to show what you have in your modified copy (if possible anyway) and I'm happy to look at it and help identify a way you may be able to fix it to do what you need.

Also we'd likely be very open to any upstreaming of relevant changes to our module. You can file bug(s) on our bugzilla [2] instance as you see fit. While our module's/code is designed strictly to assist needs for Mozilla, improvements that don't detract from our needs and make our puppet modules more robust will be a welcome change you as a contributor can make.

Thank You,
~Justin Wood (Callek)

[1] - http://mxr.mozilla.org/build/source/puppet-manifests/modules/python/manifests/virtualenv.pp#244
[2] - https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=RelOps:%20Puppet

> _______________________________________________
> tools-puppetagain mailing list
> tools-pu...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-puppetagain
>

[Jason Antman]

Justin,

Yeah, I'm very glad that it was made public... it's one of my go-to
places when I can't figure out how I should do something; we also used a
bunch of ideas in our new Puppet3 multi-master rollout, including some
of the certificate handling ideas for multiple separate puppet stacks,
and I'm going to be exploring secret storage next.

Having looked at the python module both at the link you mentioned and at
https://github.com/mozilla/build-puppet-manifests/tree/master/modules/python,
I'm not longer sure I can really say that our module is "heavily
modified" as opposed to "inspired by".

Ahhhh yeah, I see, the inject-venv-path-into-title-and-strip method. I
considered that (actually without seeing, or at least remembering, your
module) but found it to be a bit of a kludge, plus making it harder to
use Puppet's native list-as-namevar-becomes-multiple-resources
functionality. It *appears* (from looking at the puppetlabs-inifile
module) that impementing this as a native puppet/ruby type or provider
would provide a way to work around this problem. Alas, nobody on my team
has much skill with ruby.

Sure, I've posted the module we're using at:
https://github.com/jantman/puppet-python-test

Speaking of your "are you able to show what you have" comment, is there
a specific license that the PuppetAgain stuff is released under? We're a
near 100% open source shop (our main "product" is a Django application),
so we operate under the assumption that anything we modify should (as
time allows) be submitted back upstream.

Cool, good to hear. One of the members of our automation team is a
pip/virtualenv maintainer, so we're trying to come up with something
that will become a (semi) official module. I'll spend some more time
looking over your current code (is the github repo I referenced up to
date?) and see if we can use that as the base; with the one caveat that
we'd likely want to either rename the module to 'virtualenv' or
'python-virtualenv', or attempt to expand it to also (optionally) handle
installing the python OS packages.

Thanks very much for the information and reply.
-Jason Antman
jantman

On 11/01/2013 10:46 PM, Justin Wood wrote:
> Hi,
>
> Wonderful to hear of your interest in our "PuppetAgain" and its utility to you (that is one of the reasons we made it public from the start). That said as far as I can tell our define for our python module does use the venv directory itself as a part of our naming scheme for specific python packages [1].
>
> Are you able to show what you have in your modified copy (if possible anyway) and I'm happy to look at it and help identify a way you may be able to fix it to do what you need.
>
> Also we'd likely be very open to any upstreaming of relevant changes to our module. You can file bug(s) on our bugzilla [2] instance as you see fit. While our module's/code is designed strictly to assist needs for Mozilla, improvements that don't detract from our needs and make our puppet modules more robust will be a welcome change you as a contributor can make.
>
> Thank You,
> ~Justin Wood (Callek)
>
> [1] - http://mxr.mozilla.org/build/source/puppet-manifests/modules/python/manifests/virtualenv.pp#244
> [2] - https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=RelOps:%20Puppet
>
> ----- Original Message -----
>> From: inebri...@gmail.com
>> To: tools-pu...@lists.mozilla.org
>> Sent: Friday, November 1, 2013 10:28:29 PM
>> Subject: python module, virtualenv/pip handling
>>

[Dustin J. Mitchell]

PuppetAgain is licensed under the Mozilla Public License. There's a copyright header on every file - you can't miss it :)

Implementing some of the Python support using native types would make it more flexible. And we have some Ruby expertise. Would you mind opening up a Mozilla Bugzilla bug to describe what you'd like to see, and we can try to make it happen? http://tinyurl.com/pcao4yo should put the bug in the right place.

If, together, we can make that Python module "cool" enough, it might be worth trying to isolate it from the rest of PuppetAgain and shipping it as a distinct module.

Dustin

[Jason Antman]

Dustin,

Sorry for the delayed reply - prod bugs wait for no man.

Yeah I supposed I should've seen that :)

I'll be opening the bug in a few days - one of my co-workers is taking a
crack at fixing http://projects.puppetlabs.com/issues/7286 which adds
virtualenv support to the puppet core "pip" provider, and would fix at
least the same package in multiple venvs issue. Once I see how that pans
out, I'll create the bug. I'm already essentially tracking this in the
PL bug tracker and our internal ticketing system.

I'll ping back when I know how our internal efforts work out.

That would be wonderful. We're an all-python shop and pretty let down
with the lack of a complete, working community puppet module. One of the
guys on my team is a pip/virtualenv maintainer (qwcode) and is just
building his puppet skills, and is really interested in having something
(at least semi-) official that pip/virtualenv can point to.

Thanks,
Jason