Packaging and development workflow for decoupled apps.
DaleB
i could use some advice for a good packaging workflow.
So far i used to put an "apps"-folder in my project root and i stored
all individual apps in a sub-folder.
This works, but i thought: "Okay, try to decouple things a bit more
and try to create some 'real' reusable apps."
So i followed Jacob's guide on buildout (http://jacobian.org/writing/
django-apps-with-buildout) and thanks to the clear and detailed
instructions i managed to build a nice package for one of my apps.
Okay, so i removed the according sub-folder from my app-dir, installed
my new package to the virtualenv containing my project and everything
worked ... Nice!
BUT ...
I see that this is really great for the distribution of an app.
But what about development and "continuous packaging"?
I mean, my project consists of more than five apps. One side of me
would rather turn them all into installable packages so i get a nice
decoupled environment.
But how do i work on my projects once things are separated?
I mean, in reality, though functional distinct, the apps work together
and interact to form a site as whole.
So, to get a complete feeling and an overview for what my app does (or
breaks) i need to see it embedded in a working site.
I got this, when all different apps where folders under the same root-
dir. I could easily edit things in one app and see the results
immediately on my site. Due to version control it was easy to rollback
changes that were problematic.
Do i simply miss an important point? Am i just spilled by the
monolithic workflow?
How do you implement new features? Do you just concentrate on a single
app (in a separated buildout dir), test everything, package it and
finally install the new version and
then check if everything fits still together? (And if not, go back to
your app, package it again, install it etc.)
I'd really love to hear some strategies how you cope with this.
Thanks,
Andreas
Bill Freeman
buildout offers a feature called development eggs. In addition to listing
your eggs in the "eggs" variable of the [buildout] section, you list the
path to the egg source (containing setup.py, etc.) in the "develop"
variable. This keeps buildout from trying to down load the egg and,
I believe, make a directory "develop-eggs" (a sibling of buildout.cfg)
containing ".link" files, each containing the path to where you can import
the package (typically the folder containing setup.py).
zope's recipe, I think, uses these to figure out what all to add to sys.path
in the various python based executable scripts that it makes in the bin
subdirectory, so that other code can just import the package by name.
I'm not sure whether the django deployment instructions you have will
use these, but if not you could certainly make a module for inclusion
by settings.py that figured out what to add to sys.path (if it's not already
there) by inspecting the files in the develop-eggs directory.
Bill
Mike Ramirez
On Thursday, March 10, 2011 09:57:29 am DaleB wrote:
> I see that this is really great for the distribution of an app.
> But what about development and "continuous packaging"?
> I mean, my project consists of more than five apps. One side of me
> would rather turn them all into installable packages so i get a nice
> decoupled environment.
>
> But how do i work on my projects once things are separated?
> I mean, in reality, though functional distinct, the apps work together
> and interact to form a site as whole.
> So, to get a complete feeling and an overview for what my app does (or
> breaks) i need to see it embedded in a working site.
> I got this, when all different apps where folders under the same root-
> dir. I could easily edit things in one app and see the results
> immediately on my site. Due to version control it was easy to rollback
> changes that were problematic.
>
> Do i simply miss an important point? Am i just spilled by the
> monolithic workflow?
> How do you implement new features? Do you just concentrate on a single
> app (in a separated buildout dir), test everything, package it and
> finally install the new version and
> then check if everything fits still together? (And if not, go back to
> your app, package it again, install it etc.)
>
Apps should do one thing and do it well. The trick in your situation is to avoid circular dependcies. If you have circular dependancies, refactor.
You want to build the apps like a pyramid, each block/app adds something, but only the upper levels should (if they need to) depend on an app at a lower level. but never the lower level depend on the upper level.
Test Driven Development really shines here. Write your tests firsts then write the code to pass those tests, not the other way around. With proper tests you shouldn't have to worry about how the code will break your projects. You'll know it won't, once they all pass. Tests also give good example code on how to use your app.
I think you are stuck in the monolithic point of view. I'm also not sure if you understand that django apps are python modules and the same rules apply.
The apps location within the PYTHONPATH is immaterial to a project, as long as the project can find the app. With this in mind, I tend to have a lib directory, that contains apps that are in development or checkouted out from other developers versioning system. This lib directory is added to the python path for each project that needs an app from this location. Please note this is a directory seperate from python${VERSION}/site-packages.
In the end how you do it is what fits your needs, but it's best to have one location for each app that is accessible to every project that needs it.
To get a sense of what I mean you should read this[1].
Mike
> I'd really love to hear some strategies how you cope with this.
>
> Thanks,
> Andreas
[1] http://docs.python.org/tutorial/modules.html#packages
--
QOTD:
"If I'm what I eat, I'm a chocolate chip cookie."
Tom Evans
It's all about packages really IMO. If two apps are coupled, then they
probably deserve to be in the same package, but still as separate
apps. An app that is only consumed by other apps, on the other hand,
would be in a package by itself.
Eg, one of my 'for fun' projects at the minute is a django/amqp
PVR/media player that records and plays back TV and organizes,
manages, collects my other media (photos, video clips, etc), and makes
all the media available either to control on the big screen, or
consume via the browser.
Some apps are tightly coupled together, eg the app that provides the
interactive remote and the app that provides transcoding support, so
they are both apps within the pvr package. The app that collects,
parses and displays TV listings data doesn't depend on anything else,
so it lives in its own package.
Eg, my INSTALLED_APPS looks like this:
INSTALLED_APPS = (
'pvr.transcoding',
'pvr.iremote',
'xmltvutils',
...
)
The apps in the pvr package are actually in my project source code,
since they are highly coupled, eg this structure:
project/
`- pvr/
`- transcoding/
whilst the xmltvutils is completely standalone and so is installed by
pip into the appropriate place in my virtualenv.
Some ideas to chew on :)
Cheers
Tom
DaleB
> I think you are stuck in the monolithic point of view. I'm also not sure if
> you understand that django apps are python modules and the same rules apply.
>
> The apps location within the PYTHONPATH is immaterial to a project, as long as
> the project can find the app. With this in mind, I tend to have a lib
> directory, that contains apps that are in development or checkouted out from
> other developers versioning system. This lib directory is added to the python
> path for each project that needs an app from this location. Please note this
> is a directory seperate from python${VERSION}/site-packages.
the fact that the package just needs no be on the path,
no matter where the files are actually stored. That really clears
things up for me.
And thanks for for your example Tom.
One last point.
Now, how do i deal best with different versions/branches of my app? Do
you just switch branches while developing or do you use pip in project
env to refer and switch to different tags/branches in your sourcecode.
The first approach feels more dynamic, the latter more 'stable'. In
the end i guess both work, but what do you do?
Greetings,
Andreas
Mike Ramirez
On Friday, March 11, 2011 12:41:52 am DaleB wrote:
> One last point.
> Now, how do i deal best with different versions/branches of my app? Do
> you just switch branches while developing or do you use pip in project
> env to refer and switch to different tags/branches in your sourcecode.
> The first approach feels more dynamic, the latter more 'stable'. In
> the end i guess both work, but what do you do?
>
> Greetings,
> Andreas
This is more of a policy situation. What that means is that it can be different. Depends on how you use them.
Tags, usually mark a completion point (or milestone).
Branches are usually a fork or a radical change or a fork..
In django's history, when they added unicode support through out it, they had the main branch with work being aimed at generally improving django.
The unicode branch was were all the changes for adding unicode was done. We as users of django had our choice on what to use, checking out the main tree or the branch with the unicode system.
Personally,. I check out branches to a different location than the main project as I consider them different applications.
Tags, I just use to mark important milestones for the code I'm working on.
Mike
--
I'm sorry if the following sounds combative and excessively personal,
but that's my general style. -- Ian Jackson
DaleB
use tags and when to use branches.
I am still looking for a smooth development workflow.
I created a libs-dir which i put on my python path an put my different
apps in it.
Okay, now i can refer to my apps from any project without copying the
files. Great and a lot better than before.
So now i start to create different branches for an app, i create a
development branch an change stuff.
As the app is on my path my projects initially refer to the state/
branch of an app that ist currently checked out, right?
That's what i refered to by "switching branches" (= changing the
branch of my app in my libs-dir)
Another (better?) idea i had looking at pip's docs was to take my libs-
dir off the python path, put them in a git-repo and use pip to install
my app.
Pip's option to refer to certain tags and branches (http://
pip.openplans.org/requirement-format.html#version-control) seems
pretty ideal in this situation.
The only drawback i see is that i have to install and uninstall quite
often if i want to test different branches/states of my app in a
project.
And now i am not sure which route is more promising or if there's even
an easier way i don't regard at the moment.
Hope that's more clear.
Thanks fpr you patience,
Andreas
Mike Ramirez
In this case, I'm sorry, I'm not sure. I don't use pip, given the arguments I've seen for it, I've never had a reason to really investigate it myself. Setuputils/distribute/easy_install are 'good enough' for me. A long time ago james bennet posted a big why you want to use pip on this list, I can't remember much more about it, but you might want to search for that posting.
I do things manually with checkouts. If I want to work on a branch, this is checked out to a new location (since it's a radical change/fork of a project), i.e. same lib dir, different package name to let me know it's a branch. Tags are there for historical reasons, really I just use them to mark completion points and they are considered part of the main tree (usually). It's just easier to know how far back to go when searching for a revision that introduced a bug (usually). Or if I completely messed up the current branch, I can just go get the last tag.
Mike.
--
mixed emotions:
Watching your mother-in-law back off a cliff...
in your brand new Mercedes.
Bill Freeman
> One last point.
> Now, how do i deal best with different versions/branches of my app? Do
> you just switch branches while developing or do you use pip in project
> env to refer and switch to different tags/branches in your sourcecode.
> The first approach feels more dynamic, the latter more 'stable'. In
> the end i guess both work, but what do you do?
>
> Greetings,
> Andreas
One possibility is to use a development egg with is a revision control
checkout. (The whole distutils/setuptools/distribute thing seems to be
specifically aware of subversion at some level, offering to use revision
information in version numbers. I haven't noticed support for other
VCSs, but there could be ways to use them by having script wrappers
on your shell path...). Then you can simply update to another revision
to test with it.
Bill
Tom Evans
Exactly, I use a pip requirements file, and if its an app that I'm
working on, it gets installed from svn. Eg:
-e svn+https://svn.nubtek.com/libraries/python/rc/trunk#egg=rc
this gets installed to venv_dir/src/rc, which is then a full svn
checkout, which I can manually switch branches, merge/test etc. When I
want to make it permanent, I can simply update my requirements.pip,
blow away my venv dir and reinstall (or remember the full invocation
for pip install --upgrade -e etc).
Cheers
Tom