Re: vendorized cherrypy & tornado causing fun times

[Chad Whitacre]

Dan,

[Note: Moving to aspen-users list.]

Those libraries are vendorized because I like providing production-quality batteries out of the box. In the CherryPy case especially the WSGIServer was designed to be usable standalone, and in fact has been split off as the Cheroot project:

    https://bitbucket.org/cherrypy/cheroot

Paul and I have talked about removing Tornado from aspen core. Paul has added three stdlib-based templating. One of those would be the default instead of Tornado. We could do something similar with the networking engine, though I really don't like the idea of setting people loose with BaseHTTPServer. I hear war stories coming out of Heroku of Django deployments with 50 dynos, each running a single-threaded manage.py runserver. I'd like for Aspen not to be a "toy" out of the box.

Can you say more about our py3k approach in general? I understand there are a few different ways to handle the migration. CherryPy supports 3.1+, so depending on our strategy we can maybe keep that vendored at least? It looks like Tornado 2 adds Python 3.2 support as well.

What's our migration strategy? Are we writing a single codebase targeting 2 and 3, or are we maintaining separate trees?

chad

On Sun, Aug 12, 2012 at 11:44 PM, Daniel Buch <d...@meatballhat.com> wrote:

Hi Chad & Paul,

I've been working on py3k compatibility and some of the most common
offenders are in the `aspen._cherrypy` and `aspen._tornado` vendorized
libraries.  Rather than munge them in place like I've been doing, I'd
much rather list them as explicit installation requirements (or
extras) and let distutils/distribute do its thing.  I'm assuming,
though, that they're vendored for good reasons and unvendoring is
something I shouldn't be doing in a vacuum.

What can you tell me about why these libraries are vendored rather
than installation requirements?  Is it possible to unvendor them, and
if so should I be doing this in my py3k compatibility branch?

Much thanks,
--
~Dan

[Daniel Buch]

Hi List!

> Those libraries are vendorized because I like providing production-quality
> batteries out of the box. In the CherryPy case especially the WSGIServer was
> designed to be usable standalone, and in fact has been split off as the
> Cheroot project:
>
> https://bitbucket.org/cherrypy/cheroot

Totally understood wrt production batteries. I've been known to copy
CherryPy's WSGIServer into internal company projects (pre-Cheroot.)
What are your feelings toward the distribute/setuptools dependency resolution
step? I'm showing my lack of experience with Heroku and AppEngine, but is
`setup.py` execution not an option in environments like that?

> Paul and I have talked about removing Tornado from aspen core. Paul has
> added three stdlib-based templating. One of those would be the default
> instead of Tornado. We could do something similar with the networking
> engine, though I really don't like the idea of setting people loose with
> BaseHTTPServer. I hear war stories coming out of Heroku of Django
> deployments with 50 dynos, each running a single-threaded manage.py
> runserver. I'd like for Aspen not to be a "toy" out of the box.

+1 for not being a toy out of the box. Even if you were to
unvendorize Tornado, would you consider it a candidate for a
switchable extra a la distribute/setuptools' `extras_require` hook?

> Can you say more about our py3k approach in general? I understand there are
> a few different ways to handle the migration. CherryPy supports 3.1+, so
> depending on our strategy we can maybe keep that vendored at least? It looks
> like Tornado 2 adds Python 3.2 support as well.
>
> What's our migration strategy? Are we writing a single codebase targeting 2
> and 3, or are we maintaining separate trees?
>

I'm personally not a fan of depending on 2to3 or 3to2 at installation
time. Call it paranoia. Forking the project is also un-fun. I much
prefer to maintain a single project that's compatible with 2.6 through
3.2+, which is what I've started to do in my `py3k-compat` branch,
although I'm still getting familiar with the codebase and don't
consider the work to be especially merge-worthy.

[Chad Whitacre]

Dan,

What are your feelings toward the distribute/setuptools dependency resolution
step?  I'm showing my lack of experience with Heroku and AppEngine, but is
`setup.py` execution not an option in environments like that?

Heroku's not the problem. Doubt AppEngine is either.

I don't trust packaging machinery in general and Python's in particular. I think distusetupiputetools is complicated and fragile. Personally, I want to keep Aspen away from it as much as possible, so the smell doesn't rub off. When someone lands in dependency hell I want Aspen to be as far away as possible.

 

Even if you were to unvendorize Tornado, would you consider it a candidate for a
switchable extra a la distribute/setuptools' `extras_require` hook?

What's the case for using extras_require? To me it seems like clutter and complication. If I want pystache, how is it better to specify aspen[pystache] (and cross myself) than to simply specify pystache directly myself?

If I'm honest, I'm down on it, man. Open to having it out though if you want to make the case.

chad

[Daniel Buch]

> Heroku's not the problem. Doubt AppEngine is either.
>
> I don't trust packaging machinery in general and Python's in particular. I
> think distusetupiputetools is complicated and fragile. Personally, I want to
> keep Aspen away from it as much as possible, so the smell doesn't rub off.
> When someone lands in dependency hell I want Aspen to be as far away as
> possible.

I can certainly understand wanting to stay away from
distusetupiputetools because of fragility concerns. I don't think
this means you have to keep dependencies in your source tree, though.
What about adding a packaging-time step to vendorize the dependencies?
You wouldn't even have to use distusetupiputetools to do it, so the
smell would stay minimal.

> What's the case for using extras_require? To me it seems like clutter and
> complication. If I want pystache, how is it better to specify
> aspen[pystache] (and cross myself) than to simply specify pystache directly
> myself?

Given your aversion to distusetupiputetools, I'd say it's not worth
considering. I'll avoid suggesting such things in the future ;-)

[Chad Whitacre]

Dan,

 

I can certainly understand wanting to stay away from
distusetupiputetools because of fragility concerns.  I don't think
this means you have to keep dependencies in your source tree, though.
What about adding a packaging-time step to vendorize the dependencies?

Nice idea. There's a release.sh script that I use to make new releases. What would be our algorithm for vendorizing at packaging time?

Can we pin the versions of the dependencies? I believe in manual dependency upgrades. This manual upgrade should be easier than it is at present, though. In another project where I was depending on Aspen, I had a switch.sh script that took an Aspen version number and updated the vendorized Aspen dependency in the downstream project. Maybe we have an upgrade script that takes {cherrypy,tornado} and a version number? Though if the vendorizing happens at packaging time I guess it's not an issue: I guess we would just bundle whatever is in the environment where release.sh is run?

I was also going to inquire further about the troubles specifically related to porting to Python 3. Do we maybe have an old version of CherryPyWSGIServer that predates cherrypy's Python 3 compatibility? Would a brute-force upgrade of _cherrypy (and _tornado and also _mimeparse, it turns out) possibly unblock us for now?

chad 

[Daniel Buch]

>> I can certainly understand wanting to stay away from
>> distusetupiputetools because of fragility concerns. I don't think
>> this means you have to keep dependencies in your source tree, though.
>> What about adding a packaging-time step to vendorize the dependencies?
>
> Nice idea. There's a release.sh script that I use to make new releases. What
> would be our algorithm for vendorizing at packaging time?
>
> Can we pin the versions of the dependencies? I believe in manual dependency
> upgrades. This manual upgrade should be easier than it is at present,
> though. In another project where I was depending on Aspen, I had a switch.sh
> script that took an Aspen version number and updated the vendorized Aspen
> dependency in the downstream project. Maybe we have an upgrade script that
> takes {cherrypy,tornado} and a version number? Though if the vendorizing
> happens at packaging time I guess it's not an issue: I guess we would just
> bundle whatever is in the environment where release.sh is run?

If you want to avoid any pip or distribute machinery, I'd say the
release.sh script should download any dependencies directly and copy
over files exactly as desired. Upgrading the deps would be a matter
of changing the URLs. Is this similar to what you're describing?

> I was also going to inquire further about the troubles specifically related
> to porting to Python 3. Do we maybe have an old version of
> CherryPyWSGIServer that predates cherrypy's Python 3 compatibility? Would a
> brute-force upgrade of _cherrypy (and _tornado and also _mimeparse, it turns
> out) possibly unblock us for now?

Yes. Upgrading those deps would be a good idea, which I can do. It's
not so much that I'm blocked, but that I want to better understand the
design philosophy. I understand it better now, yay!

[Daniel Buch]

On Thu, Aug 16, 2012 at 7:58 AM, Daniel Buch <d...@meatballhat.com> wrote:
>>> I can certainly understand wanting to stay away from
>>> distusetupiputetools because of fragility concerns. I don't think
>>> this means you have to keep dependencies in your source tree, though.
>>> What about adding a packaging-time step to vendorize the dependencies?
>>
>> Nice idea. There's a release.sh script that I use to make new releases. What
>> would be our algorithm for vendorizing at packaging time?
>>
>> Can we pin the versions of the dependencies? I believe in manual dependency
>> upgrades. This manual upgrade should be easier than it is at present,
>> though. In another project where I was depending on Aspen, I had a switch.sh
>> script that took an Aspen version number and updated the vendorized Aspen
>> dependency in the downstream project. Maybe we have an upgrade script that
>> takes {cherrypy,tornado} and a version number? Though if the vendorizing
>> happens at packaging time I guess it's not an issue: I guess we would just
>> bundle whatever is in the environment where release.sh is run?
>
> If you want to avoid any pip or distribute machinery, I'd say the
> release.sh script should download any dependencies directly and copy
> over files exactly as desired. Upgrading the deps would be a matter
> of changing the URLs. Is this similar to what you're describing?

Or maybe something like this which would serve as a developer tool to
periodically update the vendorized deps (???):

https://github.com/meatballhat/aspen/blob/dependency-absorption/absorb-deps.sh

[Chad Whitacre]

Dan,

Or maybe something like this which would serve as a developer tool to
periodically update the vendorized deps (???):

    https://github.com/meatballhat/aspen/blob/dependency-absorption/absorb-deps.sh

Yes! That's what I envisioned.

However, after adding "set -e" to the top of the script, I get this output:

$ ./absorb-deps.sh 

readlink: illegal option -- f

usage: readlink [-n] [file ...]

usage: dirname path

$ 

The man page for readlink/stat specifies a -f option, so I'm not sure what's going on. I'm on Mac OS 10.6.8, and `which readlink` gives me /usr/bin/readlink. I can't `locate` another one.

chad

[Daniel Buch]

> Yes! That's what I envisioned.
>
> However, after adding "set -e" to the top of the script, I get this output:
>
> $ ./absorb-deps.sh
> readlink: illegal option -- f
> usage: readlink [-n] [file ...]
> usage: dirname path
> $
>
> The man page for readlink/stat specifies a -f option, so I'm not sure what's
> going on. I'm on Mac OS 10.6.8, and `which readlink` gives me
> /usr/bin/readlink. I can't `locate` another one.

Cat's out of the bag. I'm using a Linux desktop.

This should work better good nice nice:

https://github.com/meatballhat/aspen/blob/dependency-absorption/absorb-deps.sh

If this is what you have in mind, what do you think about removing the
cherrypy, tornado, and mimeparse files from the source tree and using
this script at development time and release time? I could add some
checksumming in case we're worried about upstream sneaking in patches
without bumping the versions.

[Chad Whitacre]

Dan,

 

Cat's out of the bag.  I'm using a Linux desktop.

Now to flush out which one. ;-)

This should work better good nice nice:

    https://github.com/meatballhat/aspen/blob/dependency-absorption/absorb-deps.sh

Yeah, that works, thanks. I went back and checked: In scripts where I've tried to do something similar (setting a base/top dir) I've just done it relatively using `dirname $0`. I've never actually used readlink before. Supposedly the -f option has been added to FreeBSD recently, where there is in any case an alternative in realpath(1). I don't find realpath on Mac OS, and I misread the man page earlier: when called as stat it takes a -f, but not when called as realpath.

Oh, well. Hooray for Python! :-)

 

If this is what you have in mind, what do you think about removing the
cherrypy, tornado, and mimeparse files from the source tree and using
this script at development time and release time?

+1

 

I could add some checksumming in case we're worried about upstream 

sneaking in patches without bumping the versions.

Sure seems like we'd be doing our job well if we did that. :-)

chad 

[Daniel Buch]

Warning: this response has very little to do with Aspen.

>> Cat's out of the bag. I'm using a Linux desktop.
>
> Now to flush out which one. ;-)

Xubuntu, mostly. (I also use OSX quite a bit at work and on work laptop.)

> Yeah, that works, thanks. I went back and checked: In scripts where I've
> tried to do something similar (setting a base/top dir) I've just done it
> relatively using `dirname $0`. I've never actually used readlink before.
> Supposedly the -f option has been added to FreeBSD recently, where there is
> in any case an alternative in realpath(1). I don't find realpath on Mac OS,
> and I misread the man page earlier: when called as stat it takes a -f, but
> not when called as realpath.
>
> Oh, well. Hooray for Python! :-)

For all I know it's since been fixed, but I've run into problems in
the past with `dirname $0` on RedHat variants including CentOS and
Fedora. That's when I picked up the `readlink` trick, which falls
flat on OSX and BSD (and Solaris, but who cares!)

>> I could add some checksumming in case we're worried about upstream
>> sneaking in patches without bumping the versions.
>
>
> Sure seems like we'd be doing our job well if we did that. :-)

I'll add that, then! :-D

[Chad Whitacre]

That's when I picked up the `readlink` trick, which falls
flat on OSX and BSD (and Solaris, but who cares!)

How about SmartOS?!

chad

[Daniel Buch]

Are you trolling me?

[Chad Whitacre]

Well, I'm genuinely interested. So ... no? :-)

chad

[Daniel Buch]

On Sun, Aug 19, 2012 at 9:04 AM, Chad Whitacre <ch...@zetaweb.com> wrote:
> Well, I'm genuinely interested. So ... no? :-)
>

Sorry!
From what I can tell, SmartOS's readlink implementation behaves like
Linux's, at least in the context of *bash* scripts. I'm assuming
YMMV.

[Chad Whitacre]

Dan,

I made a ticket for this:

    https://github.com/whit537/aspen/issues/100

chad

[Daniel Buch]

The work isn't done since the target directories are still in git and
there's no integration with development-time or release-time actions,
but it's a start (??):

https://github.com/whit537/aspen/pull/101

> --
> You received this message because you are subscribed to the Google Groups
> "aspen-users" group.
> To post to this group, send email to aspen...@googlegroups.com.
> To unsubscribe from this group, send email to
> aspen-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/aspen-users?hl=en.