TurboGears joins the PylonsProject

[Mark Ramm]

I've been working through a lot of feedback from the TurboGears
mailing list about how we should respond to the Repoze.bfg / Pylons
merger.   We've had a couple of IRC chats, I've spent a lot of time
talking to various people in the community, and even more talking to
Ben Bangert and Chris McDonough.

And I believe we've come to a solid consensus, and that it's time to
make a decision.

I didn't want to rush this, and very much wanted to weigh all the
options appropriately. But now that that thinking is done, it's time
to jump in with both feet.

TurboGears will be merging into the Pylons Project

To understand why we are making this move and what this means, a bit
of background is needed.

We built TurboGears 2 on top of a the Pylons framework, and have been
working with Ben and the Pylons folks for a couple of years now.
Pylons is a lightweight framework that is neither full stack nor
opinionated. The Pylons team has recently joined up with the
repoze.bfg folks to create a new low-level non-opinionated web
application development library called Pyramid which is based on
repoze.bfg and now part of the larger Pylons Project.

You can use Pyramid to build web applications right now. It has more
plug points, and is in many ways more flexible and extendable than the
original Pylons and should provide an even better foundation for a
full stack framework.

Fortunately the Pylons Project leaders recognize that a "low level"
framework is not enough. Most web developers need to get things done
quickly, and want a full stack setup and ready to go, so they can
immediately start developing features rather than infrastructure.

That's where we come in.  TurboGears was the pioneer of the "full
stack" set of integrated components approach among the modern Python
web frameworks, and we have already developed many full stack tools.

So, our first step will be to add the TurboGears2 package to the
legacy support in the Pylons Project. So, the Pylons 1.x and
Turbogears2 packages will be maintained side by side as part of a
single overall project. This change won't impact the TG2 code, except
that we will be officially acknowledging that both codebases are
tightly linked and now are part of a single project.

Maintaining the existing Pylons1+tg code will only be one part of the
larger Pylons Project.

Ultimately, we will also be working with the Pylons Project folks to
create a new generation of rapid application development tools on top
of Pyramid, as part of the TurboGears "Full Stack" philosophy. We will
help to pick default templating engines, default session support,
default data persistence mechanisms, integrate widget libraries and
build high level tools like OAuth or OpenID support.

The main goal of this merger will be to reach across framework
boundaries, work together with with Repoze, Pylons, and other web
framework developers to build a set of high level tools that make
building complex, modern web applications easier and faster. There's a
lot we can learn from each-other, and even more that we can do if we
work through our differences and find new ways to collaborate.

Over the next few weeks we'll be moving repositories, migrating
tickets, consolidating infrastructure, and discussing many exciting
new developments.

Thanks for your support. It's been a great ride, and I have a
feeling that the best is yet to come.

--
Mark Ramm

--

PS. Don't worry, TurboGears 1 will also continue to be maintained,
and supported, and we'll even have a major release soon. We don't
have all the details worked out yet, but you should also expect some
anouncments over the comming weeks about whatever changes that we'll
be making to that infrastructure.

[roopeshv]

may be its time to update http://www.turbogears.org/2.1/docs/main/WhatsNew.html#why-not-merge-with-pylons

[Mark Ramm]

Yea, we'll definitely be updating the docs in the next few days.

In this case the logic is actually very much the same, which is why
we're not merging into one package, but unlike before we *are* merging
into one project that supports both packages.

--Mark Ramm

On Mon, Dec 27, 2010 at 9:29 PM, roopeshv <roopeshv...@gmail.com> wrote:
> may be its time to update
> http://www.turbogears.org/2.1/docs/main/WhatsNew.html#why-not-merge-with-pylons
>

> --
> You received this message because you are subscribed to the Google Groups
> "TurboGears" group.
> To post to this group, send email to turbo...@googlegroups.com.
> To unsubscribe from this group, send email to
> turbogears+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/turbogears?hl=en.
>

--
Mark Ramm-Christensen
email: mark at compoundthinking dot com
blog: www.compoundthinking.com/blog

[Alice McGregor]

> TurboGears will be merging into the Pylons Project.

… and in the darkness bind them. ;^P

- Alice.

(Sorry, first thought that entered my head was of the One Ring.
Otherwise, congratulations for consolidating! Larger developer pools
are almost always a good thing.)

[Blaine]

On Dec 27, 8:41 pm, Alice McGregor <al...@gothcandy.com> wrote:
> > TurboGears will be merging into the Pylons Project.
>
> … and in the darkness bind them.  ;^P

One framework to rule them all.

[Timuçin Kızılay]

Thank you for informing us.

Now my question is :
Where is the Pyramid in that merger?
will the new framework is the sum of Pylons, BFG and Turbogears?
What will be the new name? Or it'll still be named Pylons?

My TG2.1 project is about to enter production stage and I'm looking
forward to start a new project using the new framework.

I'm not qualified enough to contribute to the project but I can only use
the code and find possible bugs while using end report them. I'm
grateful for all the things the developers done.

[Mengu]

hi timucin,

"the pylons project" is an umbrella name. pyramid is the new
framework.

[Mark Ramm]

Let me try to list the moving parts:

Pylons Project --> The overarching organization

* Pylons 1.0 --> package that is the legacy pylons support
* Turbogears2 --> package that is the legacy turbogears support
* Pyramid --> Rebranded and expanded repoze.bfg

The Pylons Project is going to do two things:

1) Provide solid legacy support for Pylons and TurboGears2 stacks
2) Collaborate around Pyramid and things built on top of Pyramid

Big new development will be centered around Pyramid, but we'll be
solidifying, cleaning up, and managing legacy code in best way that we
can.

--Mark Ramm

2010/12/28 Timuçin Kızılay <t...@savaskarsitlari.org>:

[Jd]

Thanks for clarification and unification of projects is always great,
assuming the transitions are taken care of.
Given that the TG2 stack would be maintained and well supported is a
big relief.

We have been using TG2 2.0.3 version for our OSS product for a while
and have been overall satisfied. Have few comments and questions that
we hope would be taken in the positive spirit.

* Labeling dependencies on release : This has been biggest pain for
us. i.e When we depend on TG2 2.0.3 stack,, we expect that all
dependencies are frozen, this way we can develop, test and release the
product against a particular stack. TG2 itself having dependencies
"foo >= x.x." should be changed to "foo = x.x" once the framework is
labeled/released. (I know that this is multilevel and all components
dependencies also need to be made fixed, but doing this seems right at
"providing framework"/TG2/Pyramid level). We support our application
on a wide variety of linux platforms, having atleast dependencies
fixed will cut down one axis of complexity. I hope something like this
can be done in upcoming TG2 releases as well as the new Pyramid
project.

* This will also allow other contributors to package TG2 properly and
build application packages that can be easily tested/validated and
packaged.
( and not run in to issues like
http://groups.google.com/group/turbogears/browse_thread/thread/6f209bd6a22d66c5/791918d036f1e1b4?lnk=gst&q=Ubuntu+10.10#791918d036f1e1b4
)
Making turbogears/Pyramid available and installable via native
commands like yum/apt-get is huge positive point for the product.

* It is mentioned that new work will be undertaken under Pyramid
project. Is there going to be a way to move a TG2 project to Pyramid ?
This way we can take advantage of new enhancements in made available
down the line. Do we know what this is going to look like ?

Thanks
/Jd

On Dec 28 2010, 9:12 am, Mark Ramm <mark.mchristen...@gmail.com>
wrote:

[Diez Roggisch]

>We have been using TG2 2.0.3 version for our OSS product for a while
>and have been overall satisfied. Have few comments and questions that
>we hope would be taken in the positive spirit.
>
>* Labeling dependencies on release : This has been biggest pain for
>us. i.e When we depend on TG2 2.0.3 stack,, we expect that all
>dependencies are frozen, this way we can develop, test and release the
>product against a particular stack. TG2 itself having dependencies
>"foo >= x.x." should be changed to "foo = x.x" once the framework is
>labeled/released. (I know that this is multilevel and all components
>dependencies also need to be made fixed, but doing this seems right at
>"providing framework"/TG2/Pyramid level). We support our application
>on a wide variety of linux platforms, having atleast dependencies
>fixed will cut down one axis of complexity. I hope something like this
>can be done in upcoming TG2 releases as well as the new Pyramid
>project.

The current approach of index-pages takes pretty much care of that. And I don't think your suggestion is viable. It makes it impossible to update dependencies without an explicit release of TG2 itself.

Using setup.cfg to constrain setuptools to fetch only specific versions of packages is easy enough, and allows for the individual flexibility needed.

>* This will also allow other contributors to package TG2 properly and
>build application packages that can be easily tested/validated and
>packaged.
> ( and not run in to issues like
>http://groups.google.com/group/turbogears/browse_thread/thread/6f209bd6a22d66c5/791918d036f1e1b4?lnk=gst&q=Ubuntu+10.10#791918d036f1e1b4
>)
> Making turbogears/Pyramid available and installable via native
>commands like yum/apt-get is huge positive point for the product.

I beg to differ. It is a big positive point for products that depend on it, alas existing web-apps that are released. AFAIK some red-hat stuff is based on TG1.

It's not a positive point for ongoing development of your own app. You are stuck with out-dated and possibly buggy versions because of the release-policies for a OS-distro.

The numbers of times where system-installed packages broke our development are countless. So either we have to remove pulled in dependencies of e.g. zope.transaction, or use virtualenvs with --no-site-packages.

I understand the desire of a one-size-fits-all package management. It's just not realistic, at least right now.

>* It is mentioned that new work will be undertaken under Pyramid
>project. Is there going to be a way to move a TG2 project to Pyramid ?
>This way we can take advantage of new enhancements in made available
>down the line. Do we know what this is going to look like ?

I'm not part of the TG3 (in lieu of a better name) efforts, but from previous experience with TG1 + TG2 I say: don't hold your hopes up high on this. Being backwards-compatible is difficult even in linear development. Switching horses on the go (as the move to repoze.bfg essentially means) makes it nightmarish.

You have to weigh the benefits of moving to a new stack & rewriting parts of your code vs. just keep on going with a working system. I *hope* test-code is migratable. If anything, that would be where I'd put effort into. Because then you can cover you app with tests, and see if & which break after a transition.

Diez
___________________________________________________________
Neu: WEB.DE De-Mail - Einfach wie E-Mail, sicher wie ein Brief!
Jetzt De-Mail-Adresse reservieren: https://produkte.web.de/go/demail02

[Jd]

> The current approach of index-pages takes pretty much care of that. And I don't think your suggestion is viable. It makes it impossible to update dependencies without an explicit release of TG2 itself.



>
> Using setup.cfg to constrain setuptools to fetch only specific versions of packages is easy enough, and allows for the individual flexibility needed.
>

So is it possible to create a set of (config) files that "freezes" the
dependencies at the time of release. And small doc update that can
allow application developers and packagers to pick right set of
dependencies.

[ Btw, How to interpret RELEASE of a particular version of TG2 ? Does
it mean that it will work with ANY future version of the dependent
components ? We know that is not true. ]

> >* This will also allow other contributors to package TG2 properly and
> >build application packages that can be easily tested/validated and
> >packaged.
> >  ( and not run in to issues like

> >http://groups.google.com/group/turbogears/browse_thread/thread/6f209b...

> >)
> >  Making turbogears/Pyramid available and installable via native
> >commands like yum/apt-get is huge positive point for the product.
>
> I beg to differ. It is a big positive point for products that depend on it, alas existing web-apps that are released. AFAIK some red-hat stuff is based on TG1.
>

Distribution is one of the axis considered for choosing a development
platform. We have heard from multiple consumers that egg files
deployment and management is not acceptable and they would consider
the product only if it is available in .rpm or .deb package format.

Even from developers point of view, being able to get an environment
where they can start getting their hands wet is also important. (See
wiki updates on documents on the install page the number of updates
and workarounds for people to get TG2 stack installed)

> It's not a positive point for ongoing development of your own app. You are stuck with out-dated and possibly buggy versions because of the release-policies for a OS-distro.
>

Yeah, but at the same time.. consider where each developer end up with
different stack dependencies while each setting up their environment
trying to install a particular version of TG2. This has actually
happened with us.. and consumed considerable time. Similarly when QA
sets up their environment they can get a different stack as well !!

> The numbers of times where system-installed packages broke our development are countless. So either we have to remove pulled in dependencies of e.g. zope.transaction, or use virtualenvs with --no-site-packages.
>

This happens if there is no static definition of a particular version
of TG2. In perfect world you would do your development against a
particular version of TG2 and that means all dependencies having
particular version. The same would be available from the distribution.
So there should not be any difference.
Now, I do understand that you might want to pick and choose some
dependencies depending on either bug fixes or enhancements in that
package. But you can make that choice *explicitly* and use mechanism
like virtualenv.

> I understand the desire of a one-size-fits-all package management. It's just not realistic, at least right now.
>
> >* It is mentioned that new work will be undertaken under Pyramid
> >project. Is there going to be a way to move a TG2 project to Pyramid ?
> >This way we can take advantage of new enhancements in made available
> >down the line. Do we know what this is going to look like ?
>
> I'm not part of the TG3 (in lieu of a better name) efforts, but from previous experience with TG1 + TG2 I say: don't hold your hopes up high on this. Being backwards-compatible is difficult even in linear development. Switching horses on the go (as the move to repoze.bfg essentially means) makes it nightmarish.
>
> You have to weigh the benefits of moving to a new stack & rewriting parts of your code vs. just keep on going with a working system. I *hope* test-code is migratable. If anything, that would be where I'd put effort into. Because then you can cover you app with tests, and see if & which break after a transition.
>

No encouraging but, Thanks for the heads up :)

[Diez Roggisch]

>> Using setup.cfg to constrain setuptools to fetch only specific versions of packages is easy enough, and allows for the individual flexibility needed.
>>
>
>So is it possible to create a set of (config) files that "freezes" the
>dependencies at the time of release. And small doc update that can
>allow application developers and packagers to pick right set of
>dependencies.

Not exactly. But you can contstrain setuptools to install eggs only from a specific directory or website, and forbid other sources.

https://bitbucket.org/deets/easterbunny/src/882d2328438f/README

>
>[ Btw, How to interpret RELEASE of a particular version of TG2 ? Does
>it mean that it will work with ANY future version of the dependent
>components ? We know that is not true. ]

No, but TG2 works e.g. with SQLAlchemy 0.5 and 0.6. Your approach would prevent that from happening.

>Distribution is one of the axis considered for choosing a development
>platform. We have heard from multiple consumers that egg files
>deployment and management is not acceptable and they would consider
>the product only if it is available in .rpm or .deb package format.
>
>Even from developers point of view, being able to get an environment
>where they can start getting their hands wet is also important. (See
>wiki updates on documents on the install page the number of updates
>and workarounds for people to get TG2 stack installed)

We have developers up and running (if they have a working linux) in under half an hour.

And of course a DEB or RPM is nice, if that's the way you chose to deploy. But that has nothing to do with my point. You can certainly create a DEB containing a full versionset, with dependend eggs. The problem is to try & use the shipped dependencies of your distro. That's a desaster waiting to happen.

>
>
>> It's not a positive point for ongoing development of your own app. You are stuck with out-dated and possibly buggy versions because of the release-policies for a OS-distro.
>>
>
>Yeah, but at the same time.. consider where each developer end up with
>different stack dependencies while each setting up their environment
>trying to install a particular version of TG2. This has actually
>happened with us.. and consumed considerable time. Similarly when QA
>sets up their environment they can get a different stack as well !!
>

This doesn't happen for us, see above. We use a specialized version of EggBasket (a PyPI clone), but later I found out that we even didn't need to do that.

All of our devs install different version-sets of eggs for different working-copies, e.g. when actually upgrading certain packages on a feature branch.

Also ,deployment is easy, and allows for rolling back as well.

>> The numbers of times where system-installed packages broke our development are countless. So either we have to remove pulled in dependencies of e.g. zope.transaction, or use virtualenvs with --no-site-packages.
>>
>
>This happens if there is no static definition of a particular version
>of TG2. In perfect world you would do your development against a
>particular version of TG2 and that means all dependencies having
>particular version. The same would be available from the distribution.
>So there should not be any difference.
>Now, I do understand that you might want to pick and choose some
>dependencies depending on either bug fixes or enhancements in that
>package. But you can make that choice *explicitly* and use mechanism
>like virtualenv.

Only if you happen to not have conflicting packages. And what happens if an OS-upgrade or installation of a 3rd-party tool pulls one in? That has crashed running systems more than once for us.

And if you concede that you sometimes need venvs, why not go with it in the first place & save yourself the trouble?

Diez
___________________________________________________________
NEU: FreePhone - kostenlos mobil telefonieren und surfen!
Jetzt informieren: http://produkte.web.de/go/webdefreephone

[Toshio Kuratomi]

On Mon, Jan 03, 2011 at 09:36:22PM +0100, Diez Roggisch wrote:
> >Distribution is one of the axis considered for choosing a development
> >platform. We have heard from multiple consumers that egg files
> >deployment and management is not acceptable and they would consider
> >the product only if it is available in .rpm or .deb package format.
> >
> >Even from developers point of view, being able to get an environment
> >where they can start getting their hands wet is also important. (See
> >wiki updates on documents on the install page the number of updates
> >and workarounds for people to get TG2 stack installed)
>
> We have developers up and running (if they have a working linux) in under half an hour.
>
> And of course a DEB or RPM is nice, if that's the way you chose to deploy. But that has nothing to do with my point. You can certainly create a DEB containing a full versionset, with dependend eggs. The problem is to try & use the shipped dependencies of your distro. That's a desaster waiting to happen.
>

Generalizations are bad because they ignore that there's different needs
that different strategies are geared for and the different tools that
underly those strategies.

If you're creating a web application for use inside of an organization where
you're also the full-time sysadmin for the service, virtualenvs may be the
best way to manage that.

If you're handing off to a dedicated IT department that deploys to a stable
system (one where API compatibility is ensured even when packages are
updated), building to use the system packages that will be available on that
system may be a requirement imposed by IT.

If you're developing an application that will be released for people in many
different organizations to run (for example Drupal or mediawiki [Is it bad
that all the apps that I can think of that fit this role off the top of my
head are php? :-( ]) then you have to write code that can deal with a wide
variety of different versions since the route most users will take to get
this software will be what's packaged for their operating system.

A little unrelated, putting all of the eggs you depend on into a single
deb/rpm really is a recipe for disaster as it means that you need to manage
the builds for all of the packages should you need to make changes to just
one. It's much better to take the packages from your distro and modify them
to suit your needs and then create a package repository to pull your packages
from. That way you: 1) Gain the benefit of seeing how your system
maintainers have built their packages to work around quirks that they've
discovered in each of the packages. 2) When you discover something that you
must fix, you're able to fix only that one package and issue an update
rather than updating everything in the rpm, possibly introducing unexpected
changes.

-Toshio

[Diez Roggisch]

-----Ursprüngliche Nachricht-----
Von: "Toshio Kuratomi" <a.ba...@gmail.com>
Gesendet: 03.01.2011 23:59:23
An: turbo...@googlegroups.com
Betreff: Re: [TurboGears] Re: TurboGears joins the PylonsProject

>On Mon, Jan 03, 2011 at 09:36:22PM +0100, Diez Roggisch wrote:
>> >Distribution is one of the axis considered for choosing a development
>> >platform. We have heard from multiple consumers that egg files
>> >deployment and management is not acceptable and they would consider
>> >the product only if it is available in .rpm or .deb package format.
>> >
>> >Even from developers point of view, being able to get an environment
>> >where they can start getting their hands wet is also important. (See
>> >wiki updates on documents on the install page the number of updates
>> >and workarounds for people to get TG2 stack installed)
>>
>> We have developers up and running (if they have a working linux) in under half an hour.
>>
>> And of course a DEB or RPM is nice, if that's the way you chose to deploy. But that has nothing to do with my point. You can certainly create a DEB containing a full versionset, with dependend eggs. The problem is to try & use the shipped dependencies of your distro. That's a desaster waiting to happen.
>>
>Generalizations are bad because they ignore that there's different needs
>that different strategies are geared for and the different tools that
>underly those strategies.

This discussion is in the context of TG2 development and deployment.

>
>If you're creating a web application for use inside of an organization where
>you're also the full-time sysadmin for the service, virtualenvs may be the
>best way to manage that.

I'm not the sysadmin. But I am of course working together with him. And he's quite happy with the solution so far :)

>
>If you're handing off to a dedicated IT department that deploys to a stable
>system (one where API compatibility is ensured even when packages are
>updated), building to use the system packages that will be available on that
>system may be a requirement imposed by IT.

It may be a requirement imposed by IT, but then it's not or only very hard fulfillable in the context of TG2.

Take the current ubuntu LTS (certainly a preferred choice for setting up a production system today). It contains zope.sqlalchemy, which is needed by TG2:

https://launchpad.net/ubuntu/lucid/+package/python-zope.sqlalchemy

The stable (LTS!!!!) version here is 0.4-7. Now let's take a look what a recent TG2 installation of a small project of mine:

tequila:home deets$ ll ~/.virtualenvs2.5/wordpuzzle/lib/python2.5/site-packages/ | grep zope.sqlalch
drwxr-xr-x   4 deets  staff     136 Oct 31 22:05 zope.sqlalchemy-0.6-py2.5.egg/

Bummer. Two minor version numbers higher.

So I can't go with the distro's version. I possibly event can't chose to not install it, as it might be a dependency of some other tool that needs in in that very version.

My solution here is to use a virtualenv, with --no-site-packages option to prevent sys.path pollution.

Which is yours?

>If you're developing an application that will be released for people in many
>different organizations to run (for example Drupal or mediawiki [Is it bad
>that all the apps that I can think of that fit this role off the top of my
>head are php? :-( ]) then you have to write code that can deal with a wide
>variety of different versions since the route most users will take to get
>this software will be what's packaged for their operating system.
>
>A little unrelated, putting all of the eggs you depend on into a single
>deb/rpm really is a recipe for disaster as it means that you need to manage
>the builds for all of the packages should you need to make changes to just
>one. It's much better to take the packages from your distro and modify them
>to suit your needs and then create a package repository to pull your packages
>from. That way you: 1) Gain the benefit of seeing how your system
>maintainers have built their packages to work around quirks that they've
>discovered in each of the packages. 2) When you discover something that you
>must fix, you're able to fix only that one package and issue an update
>rather than updating everything in the rpm, possibly introducing unexpected
>changes.

You see, all of this is fun & nice on a meta-level. But so far, whenever this discussion has crept up here or on c.l.py or wherever, it was just on exactly that level. I OTOH have to solve (and solved) the actual problem. I'm happy to be convinced that there is a better way that is more conformant with the OS dependency management.

So far, the only alternative I've been pointed to is using zc.buildout - which goes even *further* in not only providing eggs, but instead building whole libraries, including python itself.

So given the above example, how would you go about this?

- start creating a DEB package
- manage to install it without provoking a conflict with the system's version
- manage to have it picked up when starting the TG2-app which requires it

Bonus points for pointing out how you still benefit from the original zope.sqlalchemy package being under distro control, getting potential bug and security fixes (for the otherwise outdated version).

[Toshio Kuratomi]

On Tue, Jan 04, 2011 at 12:38:09AM +0100, Diez Roggisch wrote:
> -----Ursprüngliche Nachricht-----
> Von: "Toshio Kuratomi" <a.ba...@gmail.com>
> Gesendet: 03.01.2011 23:59:23
> An: turbo...@googlegroups.com
> Betreff: Re: [TurboGears] Re: TurboGears joins the PylonsProject
>
> >On Mon, Jan 03, 2011 at 09:36:22PM +0100, Diez Roggisch wrote:
> >> >Distribution is one of the axis considered for choosing a development
> >> >platform. We have heard from multiple consumers that egg files
> >> >deployment and management is not acceptable and they would consider
> >> >the product only if it is available in .rpm or .deb package format.
> >> >
> >> >Even from developers point of view, being able to get an environment
> >> >where they can start getting their hands wet is also important. (See
> >> >wiki updates on documents on the install page the number of updates
> >> >and workarounds for people to get TG2 stack installed)
> >>
> >> We have developers up and running (if they have a working linux) in under half an hour.
> >>
> >> And of course a DEB or RPM is nice, if that's the way you chose to deploy. But that has nothing to do with my point. You can certainly create a DEB containing a full versionset, with dependend eggs. The problem is to try & use the shipped dependencies of your distro. That's a desaster waiting to happen.
> >>
> >Generalizations are bad because they ignore that there's different needs
> >that different strategies are geared for and the different tools that
> >underly those strategies.
>
> This discussion is in the context of TG2 development and deployment.
>

Yes, and all of the examples I've given are ways to develop and deploy TG.
Because there are many different ways that you can deploy depending on the
circumstances that occur. Whereas your harping on the idea that deploying
on distro packages is unreasonable and a sure recipe for disaster is just
false.

> >
> >If you're creating a web application for use inside of an organization where
> >you're also the full-time sysadmin for the service, virtualenvs may be the
> >best way to manage that.
>
> I'm not the sysadmin. But I am of course working together with him. And he's quite happy with the solution so far :)
>

<nod> Yep, this is definitely a valid way to deploy for certain scenarios.

> >
> >If you're handing off to a dedicated IT department that deploys to a stable
> >system (one where API compatibility is ensured even when packages are
> >updated), building to use the system packages that will be available on that
> >system may be a requirement imposed by IT.
>
> It may be a requirement imposed by IT, but then it's not or only very hard fulfillable in the context of TG2.
>
> Take the current ubuntu LTS (certainly a preferred choice for setting up a production system today). It contains zope.sqlalchemy, which is needed by TG2:
>
> https://launchpad.net/ubuntu/lucid/+package/python-zope.sqlalchemy
>
> The stable (LTS!!!!) version here is 0.4-7. Now let's take a look what a recent TG2 installation of a small project of mine:
>
> tequila:home deets$ ll ~/.virtualenvs2.5/wordpuzzle/lib/python2.5/site-packages/ | grep zope.sqlalch
> drwxr-xr-x   4 deets  staff     136 Oct 31 22:05 zope.sqlalchemy-0.6-py2.5.egg/
>
> Bummer. Two minor version numbers higher.
>
> So I can't go with the distro's version. I possibly event can't chose to not install it, as it might be a dependency of some other tool that needs in in that very version.
>
> My solution here is to use a virtualenv, with --no-site-packages option to prevent sys.path pollution.
>
> Which is yours?
>

Actually, using the system packages is precisely what I do in my
deployments. The two targets that I have at the moment are RHEL5 with the
EPEL5 repositories and the current latest Fedora releases. Which just
proves that in some circumstances, it is possible to achieve just this.

>
> >If you're developing an application that will be released for people in many
> >different organizations to run (for example Drupal or mediawiki [Is it bad
> >that all the apps that I can think of that fit this role off the top of my
> >head are php? :-( ]) then you have to write code that can deal with a wide
> >variety of different versions since the route most users will take to get
> >this software will be what's packaged for their operating system.
> >
> >A little unrelated, putting all of the eggs you depend on into a single
> >deb/rpm really is a recipe for disaster as it means that you need to manage
> >the builds for all of the packages should you need to make changes to just
> >one. It's much better to take the packages from your distro and modify them
> >to suit your needs and then create a package repository to pull your packages
> >from. That way you: 1) Gain the benefit of seeing how your system
> >maintainers have built their packages to work around quirks that they've
> >discovered in each of the packages. 2) When you discover something that you
> >must fix, you're able to fix only that one package and issue an update
> >rather than updating everything in the rpm, possibly introducing unexpected
> >changes.
>
> You see, all of this is fun & nice on a meta-level. But so far, whenever this discussion has crept up here or on c.l.py or wherever, it was just on exactly that level. I OTOH have to solve (and solved) the actual problem. I'm happy to be convinced that there is a better way that is more conformant with the OS dependency management.
>

The OS dependency management is really not what I'm trying to be
illusttrative of here. I'm trying to show that deploying a single rpm of
all your dependencies is problematic compared to deploying with an rpm for
each of your dependencies. This is not a comparison of system package
managers vs virtualenvs.

>
> So far, the only alternative I've been pointed to is using zc.buildout - which goes even *further* in not only providing eggs, but instead building whole libraries, including python itself.
>

I haven't looked at zc.buildout, so I wouldn't know...

> So given the above example, how would you go about this?
>
> - start creating a DEB package

I work with rpm distros but the basics are the same. I'm going to write
down what I would do here based on the needs of my system administrators.
However, if your environment is not as strict, there's shortcuts that can be
taken in various places. I can talk about that a little as well.

Let's say that I'm developing for RHEL5 and the SQLAlchemy version there is
sqlalchemy-0.3 (in fact, that version comes from the EPEL repository but the
difference in policy there is minimal). Now the hot new app that I'm
developing is going to make use of features in SQLAlchemy-0.6 so what am
I going to do?

1) get the source of that package:
git clone git://pkgs.fedoraproject.org/python-sqlalchemy

2) Update the package to build SQLAlchemy 0.6 instead. In this case, I am
even lucky enough that the Fedora version of the package (which is
SQLAlchemy-0.6) is easily comparable to the EPEL-5 version so I can simply
merge many of the changes from the Fedora version to the EPEL-5 version.

3) Modify the spec to build a parallel installable version instead. In the
case of SQLAlchemy, the recipe would look something like this:

- Name: python-sqlalchemy
+ Name: python-sqlalchemy0.6
%build
- CFLAGS="%{optflags}" %{__python} setup.py --with-cextensions build
+ CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py--with-cextensions bdist_egg

%install
- %{__python} setup.py --with-cextensions install --skip-build --root %{buildroot}
+ easy_install -m --prefix %{buildroot}%{_usr} dist/*.egg

4) Build the package with something like::
rpmbuild -ba python-sqlalchemy0.6.spec

If I needed to build for RHEL5 but I wasn't on RHEL5, I'd get a free
account that can use the Fedora BuildSystem for scratch builds and use
that::

koji build --scratch dist-5E-epel python-sqlalchemy0.6-0.6.1-1.src.rpm

5) Here's one of hte points where shortcuts can be taken. I personally
would then get this package into EPEL so that others could benefit from
having this package available. But if you didn't want to do that and your
site policies allow it, you can also create a private repository where you
can host the package and pull it in via the system package managers.

> - manage to install it without provoking a conflict with the system's version

In either case, once Step 5 above is complete, installation is:

yum install -y python-sqlalchemy0.6

There will be no conflicts because of the way we installed took care to
use eggs to install in parallel.

> - manage to have it picked up when starting the TG2-app which requires it
>

If developing the app still or the deployment method is using paster::
1) make sure SQLAlchemy >= 0.6 is in setup.py
2) python setup.py egg_info
3) cp /usr/bin/paster .
4) edit ./paster so the __requires__ line looks like this:
__requires__ = ['PasteScript==1.7.3', 'appname']
5) ./paster development.ini

If deploying via mod_wsgi, edit the app.wsgi script so it has this at the
top:

import __main__
__main__.__requires__ = 'appname'
import pkg_resources

> Bonus points for pointing out how you still benefit from the original zope.sqlalchemy package being under distro control, getting potential bug and security fixes (for the otherwise outdated version).
>

With this setup, the original python-sqlalchemy package continues to be
under distro control, getting the updates that the distro deems necessary.

-Toshio

[Diez Roggisch]

>The OS dependency management is really not what I'm trying to be
>illusttrative of here. I'm trying to show that deploying a single rpm of
>all your dependencies is problematic compared to deploying with an rpm for
>each of your dependencies. This is not a comparison of system package
>managers vs virtualenvs.

You claim it is problematic. I see no attempt at proving that. But see below.

This is the interesting bit. You say they are installed in parallel. How so exactly? And how do I determine which version I get if I do

$ python
>>> import sqlalchemy

Especially, if I'm going to deploy/develop code for both respective versions?

That is the key. If you have an actual working solution for that, I'm all fine with using OS packages for deployment, including even the splitting up into several packages - because then there is a real benefit to meet other apps dependencies.

>With this setup, the original python-sqlalchemy package continues to be
>under distro control, getting the updates that the distro deems necessary.

The original - sure. Yours not, so all the burden of monitoring bugfixes + updates for 0.6, and thus there is no real benefit except from IT-requirements of course.

[Toshio Kuratomi]

On Tue, Jan 4, 2011 at 3:05 AM, Diez Roggisch <de...@web.de> wrote:
>>The OS dependency management is really not what I'm trying to be
>>illusttrative of here.  I'm trying to show that deploying a single rpm of
>>all your dependencies is problematic compared to deploying with an rpm for
>>each of your dependencies.  This is not a comparison of system package
>>managers vs virtualenvs.
>
> You claim it is problematic. I see no attempt at proving that. But see below.
>

I've already said that in the first email

"""
A little unrelated, putting all of the eggs you depend on into a single
deb/rpm really is a recipe for disaster as it means that you need to manage
the builds for all of the packages should you need to make changes to just
one.
"""

>>>  - manage to install it without provoking a conflict with the system's version

>>
>>In either case, once Step 5 above is complete, installation is:
>>
>>yum install -y python-sqlalchemy0.6
>>
>>There will be no conflicts because of the way we installed took care to
>>use eggs to install in parallel.
>
> This is the interesting bit. You say they are installed in parallel. How so exactly?

As written above, the following two steps:

CFLAGS="$RPM_OPT_FLAGS" %{__python} setup.py--with-cextensions bdist_egg

easy_install -m --prefix %{buildroot}%{_usr} dist/*.egg

That installs the package into the
/usr/lib64/python2.7/site-packages/SQLAlchemy-0.6.5-py2.7.egg-info/
directory and makes it available to the egg mechanism to find.

> And how do I determine which version I get if I do
>
> $ python
>>>> import sqlalchemy
>

Taking that invocation literally, you would get the version installed
by the OS vendor. (In my example, that's SQLAlchemy 0.3) If you add
the code that I mentioned in the previous email, the code that you
write will pick up version 0.6 of sqlalchemy when you do import
sqlalchemy. See the next section for more on that:

> Especially, if I'm going to deploy/develop code for both respective versions?
>

As I wrote before, to get the specific versions that you wish, you
need to get the dependency (indirectly is fine and what I consider
best practice) into __requires__ before you import pkg_resources. For
instance, if you wanted to import the 0.6 version of sqlalchemy in my
example from the python interpreter you could do this:

$ python
>>> __requires__ = ['SQLAlchemy >= 0.6']
>>> import pkg_resources
>>> import sqlalchemy
>>> sqlalchemy.__version__
'0.6.5'

In what I wrote before, I made the requirement for SQLAlchemy >= 0.6
part of the egg metadata for the application (listing it in setup.py)
and then set __requires__ to require the app. This indirectly
references the required version of SQLAlchemy and I consider it a
better practice in general as it means setup.py is the only place you
need to update your versions should things chang e(For instance if
you update your SQLAlchemy requirement or if you require a specific
version of Mako and Pylons as well).

See my previous message for where the places I've found necessary to
set __requires__ for deploying via paster vs deploying via mod_wsgi
are.

> That is the key. If you have an actual working solution for that, I'm all fine with using OS packages for deployment, including even the splitting up into several packages - because then there is a real benefit to meet other apps dependencies.
>

In some cases, virtualenv's do make more sense but in other cases,
using OS packages is the way to go (just as I tried to explain in my
original post). There's a whole set of costs and benefits to each
that need to be applied against the particular environment and
(usually human) resources that are available where you're deploying.

>>With this setup, the original python-sqlalchemy package continues to be
>>under distro control, getting the updates that the distro deems necessary.
>
> The original - sure. Yours not, so all the burden of monitoring bugfixes + updates for 0.6, and thus there is no real benefit except from IT-requirements of course.
>

Of course. I was going to write something about that but decided not
to because it seemed obvious that you, yourself, always carry this
burden once you decide not to use the vendor supplied package. The
burden of monitoring for security and bugfixes, incompatibility in
potential updates, etc becomes yours to carry whether you're building
a single rpm for the dependent modules, many rpms, or deploying in a
virtualenv. From this point of view, I'd say that using system
packages as much as possible is highly to your benefit (as that means
that you have to watch for changes in the fewest upstreams possible).
Of course, you need to settle on a Linux distribution that maintains
API compatibility when doing updates so that you can thoughtfully
build your requirements on top of it rather than fighting with it when
it upgrades to an incompatible version and you have to scramble to
either deploy an older compatible version or update your code.

-Toshio

[Diez B. Roggisch]

Sorry for being late on this, I had some email-troubles.

On Jan 4, 2011, at 7:53 PM, Toshio Kuratomi wrote:

On Tue, Jan 4, 2011 at 3:05 AM, Diez Roggisch <de...@web.de> wrote:

The OS dependency management is really not what I'm trying to be

illusttrative of here.  I'm trying to show that deploying a single rpm of

all your dependencies is problematic compared to deploying with an rpm for

each of your dependencies.  This is not a comparison of system package

managers vs virtualenvs.

You claim it is problematic. I see no attempt at proving that. But see below.

I've already said that in the first email
"""
A little unrelated, putting all of the eggs you depend on into a single
deb/rpm really is a recipe for disaster as it means that you need to manage
the builds for all of the packages should you need to make changes to just
one.
"""

Which is in what respect more disastrous or even tedious than having to manage all the builds for the *various* deb/rpm packages?

my scenario:

 - a virtualenv

 - my own software I develop

 - one way to wrap it up in a deb/rpm

 - in case of updated package, 

  1) "easy_install -U <the_package>",

  2)  create deb

  3) install on target machine

whereas in your scenario, you need 

 - checkouts for all packages you install

 - packaging scripts for all of them

 - in case of updated package

   1) update the working copy of the respective package, 

   2) package it as deb/rpm

   3) package your software, with updated dependencies the way you describe below

   4) install both packages

This is more work, and especially updating the meta-information is tedious and much more error-prone than simply to not having to do it at all.

So, again: where is the disaster looming?

Sorry, I somehow missed the __requires__, no idea why. It's interesting & good to know that debs/rpms can hook into the setuptools mechanism for pre-selecting versions, I didn't know that.

So I will refrain from thinking (and telling) that packages aren't up to the task, apparently they are.

But IMHO the whole procedure is rather elaborated, and and explicit repetition of implicit version dependencies I get if I "groom" my virtualenv. No need to create anything for the dependencies, no need to work with working-copies and so forth.

I'm sorry, but I'm still  not convinced that this is a road I would like to take. If the involved efford was for "the greater good", meaning that it's done with the intent to make the packages available for 3rd parties to use - fine.

But for my own system, especially for e.g. my own versions of standard packages (we run a patched version of ToscaWidgets), it's all the work without the gain to me.

Of course.  I was going to write something about that but decided not
to because it seemed obvious that you, yourself, always carry this
burden once you decide not to use the vendor supplied package.  The
burden of monitoring for security and bugfixes, incompatibility in
potential updates, etc becomes yours to carry whether you're building
a single rpm for the dependent modules, many rpms, or deploying in a
virtualenv.  From this point of view, I'd say that using system
packages as much as possible is highly to your benefit (as that means
that you have to watch for changes in the fewest upstreams possible).
Of course, you need to settle on a Linux distribution that maintains
API compatibility when doing updates so that you can thoughtfully
build your requirements on top of it rather than fighting with it when
it upgrades to an incompatible version and you have to scramble to
either deploy an older compatible version or update your code.

Of course I'm aware that this burden is shared in both approaches. It's obvious, as you say. But as long as one doesn't strive for the above mentioned  general availability of packages for other users as well (or in turn you use such packages), it just adds to the effort.

And also of course we use e.g. psycopg2 from the system's distro, gaining the benefit of any updates it might experience through the lifetime of the distribution's version in use.

But we also have been bitten numerous times by implicit installs of other python packages as system-packages that conflicted with our needs. It has not yet driven us to forego system packages fully by --no-site-packages isolated venvs. But again, it tips the scales.

In summary, I'm happy to hear that things work on a technical level, and if I'd suffer from a BOFHA that forced me to install all dependencies as single deb, it's good to know I can do it.

But I'm still thinking that a self-managed venv (potentially packaged up for distribution purposes) serves our and the most needs better, as it requires less work.

Diez

[shivaraj sajjan]

I want full information about turbogears

[Diez B. Roggisch]

Am 21.01.2011 um 05:23 schrieb shivaraj sajjan:

> I want full information about turbogears

I want cake!

Diez

[werner]

With coffee or tea:-)

Werner

[Timuçin Kızılay]

I'm now grinding some good coffee, so you start baking the cake. :-)

[Tamas Hegedus]

Dear Shivaraj,

I hope you understand the jokes and do not perform any suicide action
;-) The guys are nice here and very helpful. You just have to formulate
your question in a way that people understand. I guess you have some
problem with English. That also happened with me a few years ago - don't
give up :-)

So be specific.
What type of information do you need?
Why the online documentation is not enough for you?

I few days ago I also experienced some problem finding the online docs.
Today I noticed that first I found the 2.1 docs, but it was not so
helpful for me as the 2.0 docs. You can find 2.1 docs from the main
page, while the 2.0 here:
http://turbogears.org/2.0/docs/index.html#documentation-table-of-contents
http://turbogears.org/2.0/docs/toc.html

tamas

[Pavel Strashkin]

Guys,

You're rocks, but pls, dont flame :)

2011/1/21 Tamas Hegedus <biohe...@gmail.com>: