new clean up branch

[James Casbon]

Hi,

I spent a little time working on this branch:
http://github.com/jamescasbon/codenode/tree/django-commands

It attempts to remove a lot of the magic going on in getting codenode
running to do with running servers, python paths, etc. I really
wanted to do this to try and make codenode installable in an existing
django setup with out too much hassle. To that end, I made the
following changes:
* removed sys.path changes
* remove os.environ changes
* all commands can be executed through the django management command
app (no need for a codenode-admin)
* fixed module importing to be absolute and not require path
manipulations (e.g. for compress, or twisted.plugins).
* the codenode starts do not spawn process with os.system but call
the twisted API instead

I think this helps with simplifying the installation a lot. You can
now do this to get codenode running:

1. install it into your environment 'pip install codenode'

2. create a django app with 'django-admin.py start project proj && cd proj'

3. pull in the default settings by replacing settings.py with a single
line file: 'from codenode.frontend._settings.import *'
(note that file should probably be called
codenode.frontend.settings.development')

4. initialize the database './manage.py syncdb --noinput' then
'./manage.py runscript --fixtures development'
(we can perhaps remove the dependency on django_extensions for
runscript here, but really the script is a lot nicer than a json dump)

5. run the dev server './manage.py codenode_desktop'

So you see that this is allows for a django user to deploy codnode in
their environment by customising the settings file (basically to add
to installed apps), and doesn't require a codenode directory with code
copied in. IMHO this is a lot cleaner. We could probably put a
manage.py in the source tree so that we can develop without the
separate codenode initialized directory.

With my sysadmin hat on, not having code copied into different dirs is
appealing for upgrades and maintenance, but really this lowers the
barrier to entry to django users to hook up their business objects to
codenode on their existing django site. Also, a pure

The one glitch is backend settings, which are currently directly
imported. We should probably solve this by expecting a
codenode_settings object to be present in the django settings, so that
you can do something like:
from codenode.backend.settings import development as codenode_settings
The service.py file could do with a tidy up after this.

James

[Alex Clemesha]

On Tue, Nov 24, 2009 at 1:34 PM, James Casbon <cas...@gmail.com> wrote:

Hi,

I spent a little time working on this branch:
http://github.com/jamescasbon/codenode/tree/django-commands

It attempts to remove a lot of the magic going on in getting codenode
running to do with running servers, python paths, etc.  I really
wanted to do this to try and make codenode installable in an existing
django setup with out too much hassle.  

*Wow*, awesome!!

I can't look at all this very carefully at the moment
( as I'm trying to finish up some other work right now)
but my first impression is "OMG, this rocks!".


We really need to all get on google Wave or IRC soon, and
organize our plan to move forward with this as efficiently as possible.

Related to this is the great backend stuff you and Matt Turk are working on,
that probably have a lot of stuff in common, we just need to figure out:
"what is in common" + "what needs to be done" + "what is done" + "who is doing what".
Maybe some wiki pages need to be created that list the above? Please
consider adding some here: http://wiki.github.com/codenode/codenode



thanks!
More to follow,
-Alex




 

--
http://groups.google.com/group/codenode-devel?hl=en
http://codenode.org

--
Alex Clemesha
clemesha.org

[Dorian Raymer]

Wow, great breadth of improvements!
I'm currently deep in the middle of another project, but I focused in on the database fixtures and database init script commits. I think these are very important, and this init script is way more appropriate than including a codenode.db file in the package/repository, as we currently do now. Very nice.

I'll be looking at the rest of the commits soon. This week is the end of a big effort on another project for me.

On Tue, Nov 24, 2009 at 1:59 PM, Alex Clemesha <clem...@gmail.com> wrote:

On Tue, Nov 24, 2009 at 1:34 PM, James Casbon <cas...@gmail.com> wrote:

Hi,

I spent a little time working on this branch:
http://github.com/jamescasbon/codenode/tree/django-commands

It attempts to remove a lot of the magic going on in getting codenode
running to do with running servers, python paths, etc.  I really
wanted to do this to try and make codenode installable in an existing
django setup with out too much hassle.  

*Wow*, awesome!!

I can't look at all this very carefully at the moment
( as I'm trying to finish up some other work right now)
but my first impression is "OMG, this rocks!".


We really need to all get on google Wave or IRC soon, and
organize our plan to move forward with this as efficiently as possible.

One possibility is to have a wave per specific area of effort or related group of efforts. This could a good way for those working/planning on specific things to coordinate and track progress in real time during development. 
 

Yes, since the backend is not (yet) an actual django project, this settings file is only half-baked. There are a few other things remaining to be cleaned up with the backend that I can address in the near future.

 

[Dorian Raymer]

Hi James,
I've been using this code (it has since been merged in to codenode/codenode master) and I have hit some bugs when following the instructions here

On Tue, Nov 24, 2009 at 1:34 PM, James Casbon <cas...@gmail.com> wrote:

When I do this to the settings file, and then run syncdb, i get this:

$ ./manage.py syncdb --noinput
Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_manager(settings)
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/__init__.py", line 362, in execute_manager
    utility.execute()
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/__init__.py", line 303, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/base.py", line 195, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/base.py", line 222, in execute
    output = self.handle(*args, **options)
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/base.py", line 351, in handle
    return self.handle_noargs(**options)
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/core/management/commands/syncdb.py", line 49, in handle_noargs
    cursor = connection.cursor()
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/db/backends/__init__.py", line 81, in cursor
    cursor = self._cursor()
  File "/home/dorian/code/github/deldotdr/test_codenode_dec09/testinstallcodenodetemp/lib/python2.5/site-packages/Django-1.1.1-py2.5.egg/django/db/backends/sqlite3/base.py", line 170, in _cursor
    self.connection = Database.connect(**kwargs)
sqlite3.OperationalError: unable to open database file

I also tried the variation _desktop_settings, and i get the same error.

I probably missed better instructions for how to use the new management commands somewhere... took me a while to even remember this email after being confused by all the new changes in the source for a while ;)

I also tried running codenode-admin codenode_desktop command, and that one succeeded in creating the db in ~/.codenode.
 

4. initialize the database  './manage.py syncdb --noinput' then
'./manage.py runscript --fixtures development'
(we can perhaps remove the dependency on django_extensions for
runscript here, but really the script is a lot nicer than a json dump)

5. run the dev server './manage.py codenode_desktop'

So you see that this is allows for a django user to deploy codnode in
their environment by customising the settings file (basically to add
to installed apps), and doesn't require a codenode directory with code
copied in.  IMHO this is a lot cleaner.  We could probably put a
manage.py in the source tree so that we can develop without the
separate codenode initialized directory.

With my sysadmin hat on, not having code copied into different dirs is
appealing for upgrades and maintenance, but really this lowers the
barrier to entry to django users to hook up their business objects to
codenode on their existing django site.  Also, a pure

I'm considering this carefully and I'm not totally positive how I feel about it yet. It certainly makes sense from the perspective you describe here, however, I also believe there is value in a codenode-admin that is its own thing, wrapping (lightly) both django-admin and twistd.
I think django-admin commands specific to the frontend are an excellent idea, just maybe not for codenode-admin.

I'm still making sure I fully get the implications of what you did (hopefully the answer to my bug question above will aid in that) but I'm wary of to much direct coupling to django. I liked how our outer most management functions used the Lamson code before because django doesn't necessarily need to be used/usable at that point in the start up sequence, which is important considering the backend doesn't actually even use django (yet).

Also, the codenode-environment (or codenode directory, as you called it) as it used to be has it's benefits (think Trac, if you happened to have used it): it is an explicit location in the file system representing a particular "codenode instance". The codenode-admin is the thing that operates on the codenode-environment.
As it is, the codenode-environment contains directories supporting django specific things, and directories supporting twisted specific things. codenode-admin knows how to call django-admin and twistd and use the respective directories in the codenode-environment accordingly. Codenode-admin takes care of steps you described in your list here (like syncdb). Eventually, codenode-admin would be the tool to add upgrade/migration functionality to; the user/deployer wouldn't need to know the inner details of upgrading just as they don't need to know the inner details of codenode-admin init.

The one glitch is backend settings, which are currently directly
imported.  We should probably solve this by expecting a
codenode_settings object to be present in the django settings, so that
you can do something like:
 from codenode.backend.settings import development as codenode_settings
The service.py file could do with a tidy up after this.

This is an important and nuanced point that has been around since the days of Knoboo:
unification of settings/configuration options.

It was funny to see your implementation of the run function -- that was how things used to work until we got better at using twistd service options and the twisted application framework! We are repeating a pattern here, but instead of learning how to make good twistd plugins, we are now talking about making django plugins!
Not sure if this is good or bad, but the common denominator that deserves attention is how best to declare and define configuration options for this multi-part system (the two parts of the frontend and the backend).
So, I see that as a very important implication of your work here, and maybe, in a new thread, we can hash out thoughts on how to centralize declaration of configuration options...currently declaration happens in the settings.py and in codenode/service.py and both those also define the values of those options...some options are ambiguously redundant, some un-used...It is working, but it's messy :)

Thanks,
Dorian

 

[James Casbon]

2010/1/4 Dorian Raymer <deld...@gmail.com>:

Hi Dorian,

I'm sorry you ran into a bug using the merged code. I am a little
suspicious of what happened in the merge as Alex ran into missing
imports which I'm sure were not there in my commits. I will see if I
can reproduce later this evening as I have a meeting starting in a few
minutes.

You are right to point out that the new documentation was missing -
that's mainly because I missed your new documentation on all those dev
scripts until after the merge. Apologies.

I'd like to step back and identify the main problem I was trying to
solve: I want to install codenode as a django app - and that is the
main part. The shipped DB seeemed a bit weird to me, as it was
preventing me doing this. I could now probably work around this using
django-admin but it still seems better to me that a bootstrap is done
in code rather than an sqlite database. During my work I stripped out
lamson configuration as I thought having two (django, twisted) was
better than having three (DRY and all that) - I should habe made this
change clearer.

As for having environment directories, I think this is a great idea
and nothing I committed prevents this - it simply chooses a sane
directory for the desktop version (as in end user version). Another
settings file

If this is blocking your work, I am totally happy with rolling back
until we have agreed some changes. I think the important thing here
is to carry on improving codenode - not descending into an argument
about which configuration style is better.

I will be back later with more info.

cheers,
James

[James Casbon]

On Jan 4, 11:05 am, Dorian Raymer <deldo...@gmail.com> wrote:
> Hi James,
> I've been using this code (it has since been

> merged<http://groups.google.com/group/codenode-devel/browse_thread/thread/0f...>in

OK I got it - you need a 'data' directory inside the directory you are
working in (the django project dir). Just try a 'mkdir data'.

That's because we are just fudging the settings file for the django
project - in a real instance you would have a database defined in a
proper location.

> I also tried the variation _desktop_settings, and i get the same error.
>
> I probably missed better instructions for how to use the new management
> commands somewhere... took me a while to even remember this email after
> being confused by all the new changes in the source for a while ;)

I will submit new documentation if we decide we are not going to roll
this back.

I had previously missed all the stuff in the 'devel' folder - with
this new system we could probably have a '_settings_development' to do
this style configuration.

OK all good points. Just as long as we can preserve a route without
using codenode admin I'm happy enough.

>
> The one glitch is backend settings, which are currently directly> imported.  We should probably solve this by expecting a
> > codenode_settings object to be present in the django settings, so that
> > you can do something like:
> >  from codenode.backend.settings import development as codenode_settings
> > The service.py file could do with a tidy up after this.
>
> > This is an important and nuanced point that has been around since the days
>
> of Knoboo:
> unification of settings/configuration options.
>
> It was funny to see your implementation of the run

> function<http://github.com/codenode/codenode/blob/master/codenode/management/c...>--

> that was how things used to work until we got better at using twistd
> service options and the twisted application framework! We are repeating a
> pattern here, but instead of learning how to make good twistd plugins, we
> are now talking about making django plugins!
> Not sure if this is good or bad, but the common denominator that deserves
> attention is how best to declare and define configuration options for this
> multi-part system (the two parts of the frontend and the backend).
> So, I see that as a very important implication of your work here, and maybe,
> in a new thread, we can hash out thoughts on how to centralize declaration
> of configuration options...currently declaration happens in the settings.py
> and in codenode/service.py and both those also define the values of those
> options...some options are ambiguously redundant, some un-used...It is
> working, but it's messy :)

Yes, this is always a difficult point. I've seen some projects go
crazy on this, so I think we should keep it simple.

It may be worth keeping this kind of thing in mind:
http://code.google.com/p/django-preferences/

Ideally the settings file should be simple. We already have
dependencies: i.e. the plot dir is the HOME_PATH + 'plot_images'. The
trouble is, if you change HOME_PATH after the plot dir has initialized
this won't change. Ideally you don't want to freeze this kind of
configuration dependency at all in the settings file, since it means
you cannot tweak the settings easy enough.

Cheers,
James

[James Casbon]

2010/1/4 Dorian Raymer <deld...@gmail.com>:

It looks like the code in the repo has the django_settings commented
out in INSTALLED_APPS. You will need it enabled to load the fixture
script.

How do you feel about adding this dependency? It seems pretty useful
(see attached model graph). However, we could probably work around
it.

[Dorian Raymer]

Right, so I think the partial solution is documenting the proper way to handle this (either with instructions for creating this dir, or something) for a django-centric setup, and then keeping the original condenode-admin around for those who don't want think about these details. 

 

> I also tried the variation _desktop_settings, and i get the same error.
>
> I probably missed better instructions for how to use the new management
> commands somewhere... took me a while to even remember this email after
> being confused by all the new changes in the source for a while ;)

I will submit new documentation if we decide we are not going to roll
this back.

And we'll get to this decision. I'm working a lot on two other things the next few days, so as soon as I get to a break point, I'll re-asses this again.

 

I had previously missed all the stuff in the 'devel' folder - with
this new system we could probably have a '_settings_development' to do
this style configuration.

Yeah, I'm interested in keeping this around, in some way shape or form. It doesn't have to be shel scripts, I just want to keep all the convenient benefits it provides -- it has proven to facilitate a good development work flow.

Right, I agree, everything ought to work via the standard django methods, and the codenode-admin tool would provide convenience functionality on top of that.

The first read through of this sounded very appropriate for some of our scenarios. Alex has created the usersettings app based on similar ideas, but I the thing I really like about the django-preferences is that, if I understand it correctly, preference options are declared in a regular file (they recommend a constants file), and not explicitly in a model, which sounds simpler and more flexible. 

Also, a take away practice is simply using a dedicated file for declaring preferences/settings for everything in the code, which could go along with the 'keep it simple" strategy.

 

Ideally the settings file should be simple.  We already have
dependencies: i.e. the plot dir is the HOME_PATH + 'plot_images'.  The
trouble is, if you change HOME_PATH after the plot dir has initialized
this won't change.  Ideally you don't want to freeze  this kind of
configuration dependency at all in the settings file, since it means
you cannot tweak the settings easy enough.

Good point. The plot dir may as well be hardcoded since it isn't actually configurable. I think this reflects that the way the notebook data is modeled and persisted still isn't just right yet. But, this is something that we will probably start addressing once the more real time collaborative features start getting implemented. 

I'll check back in when I have time to fire up my codenode devel environment and test the fix.

Thanks,

Dorian

 

Cheers,

[Dorian Raymer]

Cool graph! 

Clarification: did you mean the repo has the django_extenstions app commented out?

-Dorian

[James Casbon]

2010/1/6 Dorian Raymer <deld...@gmail.com>:

> Cool graph!
> Clarification: did you mean the repo has the django_extenstions app
> commented out?

Yes.

James