Announcing "focaccia" - a new configuration release from edX devops

[John Jarvis]

Key features:

* public AMI released for openedx ami-dbd0eeb2
* updated to include the latest edX server code
* removed most of the edX branding from the default front-page
* updated to the latest released ansible (1.4)
* devstack now includes the demo course and default users
* /edx/etc/server-vars.yml so that configuration is maintained between
runs of /edx/bin/update


It's never been easier to test-drive the edX platform, please
reference one of the links below:

- For developers - https://github.com/edx/configuration/wiki/edX-Developer-Stack
- Production configuration in vagrant -
https://github.com/edx/configuration/wiki/edX-Production-Stack
- Installing on a single server -
https://github.com/edx/configuration/wiki/edX-Ubuntu-12.04-Installation

If you run into problems please report them on our issue tracker -
https://github.com/edx/configuration/issues

-John

[Yarko Tymciurak]

Cool!

Will check out momentarily, test my latest personal patch which lets the theme feature be honored in the devstack bringup.

If all is well, I'll submit pull for this, and for base/*stacks which let you build/fork off of specific Open edX repo's & versions.

Base/fullstack should soon be in the mix, once I dig into cms's django-admin preprocess_assets - it's clearing out all the theme "includes" put in by other configuration scripts; right now that is fullstack creation for an Open edX based fork w/ theme.

The wiki instructions ("alternate env files") will be able to be simplified soon too.

Feanil offered helpful suggestion on how to move in this direction.  Thanks for the development path you guys have been paving - once you see it, it's pretty nice!

- Yarko

[Yarko Tymciurak]

John -

I have a lot of things to say about this release.

First off - this is a good direction.

Second - there are a few issues which affect Open edX based development at hand (and will positively impact edx.org too, so I hope everyone pays attention equally).

I'll start another post to talk more about the second point, and give you (all) a little teaser here.

Fist order of business:

• focaccia devstack came up just dandy!
• after poking around cms and lms briefly,  I pulled in https://github.com/edx/edx-platform/pull/2387 to test;  all fine too.

In fact, I've been trying to find a development idiom for "non edx.org" development so that I can start "getting serious", and

in the process Feanil pointed me to using configuration's  vagrant/base  scripts (in stead of vagrant/release, as is what this release is brought up under).

I've learned enough about that that I'll throw you a teaser:

In base/devstack/Vagrantfile,  I hunted around the sources, and created a json file for all the bringup settings I wanted to affect.

In particular I:

• want to be able to start from a particular version of a particular repo
• if it's github.com/edx  - I might prefer a "release" tag, rather than "bleeding edge"
• if someone asks for help, or an addition to their site (e.g. education.mongodb.com),  I'd want their repos, and version tags;
• want to be able to easily change development branches - to merge work from others, or keep others from my own bleeding edge work;
• want to be able to evolve a theme or skinning, so be able to specify a canonical theme, and change to an "in-development" version, change back-and-forth easily
• if I have to debug / triage something, I want to be able to isolate that particular repo+version to see if the issue continues in a VM context (or if it's perhaps to do with deployment provisioning)
• whatever I configure and do, I want neat, stair-steps from raw development env (thanks John!  I noticed the debug-toolbar shows w/o any drama to get it to come up!),  to supervisor'd and service'd operation, to test, to final destination deployment;
• it would also be nice to be able to hand a course developer a fullstack course development environment for their & their team's use.
• I also want to be able to actually test, which means I'd like everything that went into making a system tagged the same (not just edx-platform);
• and I'd like to be able to fork, and specify forks for any such items

That's a big shopping list - but there are a lot of moving parts here, so they deserve such a list.

I'll leave this to another post - a call for volunteer/developers  to push this vision, which I think the edx team has just beautifully layed the groundwork for.

The teaser:   My pull #2387 was ready just about the time John launched this announcement, so it behooved me to test my stuff on top of his new release.

The usual way worked, but I wanted to know if I could do it in a more orderly way (per my shopping list, above).

The usual way:

• `vagrant up` the initial focaccia bringup
• test / use;
• checkout my forked branch (this time, an edx-platform change my fork is https://github.com/yarko/edx-platform):
• git remote add yarko g...@github.com:yarko/edx-platform.git
• git fetch yarko   # on top of the edx-platform checkout this `vagrant up` created;
• git checkout --track yarko/yarko/fix-rake-env-assets  # "yarko/fix-rake-env-assets" is the branch name, so 2-yarko's;
• git diff master    # just a sanity check
• than in the VM, as edxapp - `rake devstack[lms]`  ran thru everything which used this pull;
• all is well;

In fact, you could do it this way also.

What I left out is that I rebased my branch against the current master, before pushing all this out to my fork (that was prep for the pull request).

Here is a less error prone, more repeatable way (hold on to your hats - this is a teaser of what is to come!):

• It turns out you can override variables set by various ansible playbooks, overriding things like (for example) theme and host name (good thing, too!).
• in base/devstack/Vagrant,  ansible (and the provisioning) runs on your host (not on the VM);
• this means that you have to install "the version" of ansible required, and all the various python libraries specified in configuration/requirements.txt
• that wouldn't be too bad, except vagrant's ansible provisioner does NOT honor python virtualenv, so anything you install becomes host-wide for you (potentially overriding versions you had with older or newer ones - blech!!!)
• release/devstack/Vagrant, in configuration/vagrant, instead sets up a shell script which will run the playbooks from within your VM
  
• yay!  your VM environment doesn't "pollute" your host environment.
• the only question is:  are they the same???

With  base/devstack   I used the vagrant ansible extra_args  In fact, there are so many options you will want to use that I used a filename for a JSON file.

It turns out that works in the shell script / release ansible also.

Here:  make this change to your Vagrantfile:

--- Vagrantfile+ 2014-01-30 23:34:29.000000000 -0600
+++ Vagrantfile 2014-01-30 22:36:09.000000000 -0600
@@ -16,7 +16,7 @@
 # this can cause problems (e.g. looking for templates that no longer exist).
 /edx/bin/update configuration master
 
-ansible-playbook -i localhost, -c local vagrant-devstack.yml
+ansible-playbook -i localhost, -c local vagrant-devstack.yml --extra-vars "@/vagrant/build_version.json"
 SCRIPT

Your bringup host directory is always mounted as /vagrant on your host.

You can read more about `--extra-vars` with file reference here: http://docs.ansible.com/playbooks_variables.html#passing-variables-on-the-command-line

Now, create a build_version.json  file, in the same directory with your Vagrantfile:  https://gist.github.com/yarko/8727129

COMMON_GIT_MIRROR is defined within the playbooks, defaults to github.com.

Now what?

• any of these will do:
• vagrant provision   # assumes you have a running instance
• vagrant up --provision   # if not; forces re-provisioning
• unfortunately, the canonical stanford theme fails on this release
• the last time I know it to work well is  `git log -n 1 --before="2014-01-20"`  (last commit of 1/20);
• alternately:  `git rev-list -n 1 --before="2014-01-20" master`
• you'd need to bring that commit up (specifically) with base/devstack/Vagrant - that's another post;
• getting us back to that point, and keeping us from straying - that is the call for volunteers;

Once you've re-provisioned, you can go into your VM, and bring up edx-lms and cms.

There's nothing earth shaking, except that when, in edx-platform (either host or VM is fine), if you do `git remote -v` you will see the repo this came from, 

and if you issue `git status` you will see the branch this is (it's not master! ;-)).

So - thank everyone on the edx staff profusely, and get to your pull requests - because now you can easily, without error, test!

Tomorrow, I'll write up a new post with what I think are some of the tasks to make this vagrant-level development & deployment configurable, and really working for everything.

The repos you can specify are full - everything you need to cover my laundry list.

Let the FUN begin!!!

Best,

- Yarko

[Juanan Pereira]

Hi!

On Thu, Jan 30, 2014 at 11:59 PM, John Jarvis <ja...@edx.org> wrote:

Key features:

* public AMI released for openedx ami-dbd0eeb2

this is awesome!  

I've added to the wiki one important comment for all European users out there: the AMI is only available in the N. Virginia AWS region. Any plans to mirror it to other regions? 

Keep up the good work!

--
Greetings,

Juanan Pereira

[Ed Zarecor]

Juanan,

I've made the AMI available in the following regions:

eu-west-1: ami-aab442dd

ap-southeast-1: ami-9c7a2cce

us-west-1: ami-704d7035

Please test it out and provide whatever feedback you have.  Also, a big thanks to Yarko for the incredibly detailed devstack manifesto.

Best,

e0d.

[John Jarvis]

Hi Yarko - see comments inline below:

> want to be able to start from a particular version of a particular repo
>
> if it's github.com/edx - I might prefer a "release" tag, rather than
> "bleeding edge"
> if someone asks for help, or an addition to their site (e.g.
> education.mongodb.com), I'd want their repos, and version tags;

I don't think you will need to use the vagrant/base/devstack directory
for what you are doing.
When you do a `vagrant up` in the
configuration/vagrant/release/devstack directory it will check out
the configuration repo and run the following command:

ansible-playbook -i localhost, -c local vagrant-devstack.yml

If you look at vagrant-devstack.yml file
(https://github.com/edx/configuration/blob/master/playbooks/vagrant-devstack.yml)
you can see where it is setting the ref for edx-platform -
edx_platform_version: 'master'.
If instead you want the edx-platform version to be set to "release"
modify the following line in the devstack Vagrantfile:

ansible-playbook -i localhost, -c local vagrant-devstack.yml

to:

ansible-playbook -i localhost, -c local vagrant-devstack.yml -e
"edx_platform_version=release"

You can override any variable set in vagrant-devstack.yml this way.
(now that I have read the rest of your response I think you already
are aware of this)

>
> want to be able to easily change development branches - to merge work from
> others, or keep others from my own bleeding edge work;
> want to be able to evolve a theme or skinning, so be able to specify a
> canonical theme, and change to an "in-development" version, change
> back-and-forth easily

Once you have the devstack running it should be relatively
straightforward to switch branches, forks,
git bisects, etc.

> it would also be nice to be able to hand a course developer a fullstack
> course development environment for their & their team's use.
> I also want to be able to actually test, which means I'd like everything
> that went into making a system tagged the same (not just edx-platform);
>
> and I'd like to be able to fork, and specify forks for any such items
>

If I'm understanding you correctly it sounds like you want to give a
vagrant image to
a course developer that has code from an edx-platform fork.
For this I suggest using the fullstack vagrant image and follow the
instructions to change
the edx-platform repo to use a fork -
https://github.com/edx/configuration/wiki/edX-Managing-the-Production-Stack#wiki-using-an-edx-platform-fork
Once you have a working VM you can issue a `vagrant package` to create
a new image and hand it to them.

>
> Here: make this change to your Vagrantfile:
>
>> --- Vagrantfile+ 2014-01-30 23:34:29.000000000 -0600
>> +++ Vagrantfile 2014-01-30 22:36:09.000000000 -0600
>> @@ -16,7 +16,7 @@
>> # this can cause problems (e.g. looking for templates that no longer
>> exist).
>> /edx/bin/update configuration master
>>
>> -ansible-playbook -i localhost, -c local vagrant-devstack.yml
>> +ansible-playbook -i localhost, -c local vagrant-devstack.yml --extra-vars
>> "@/vagrant/build_version.json"
>> SCRIPT
>
>

Good idea though I would make it a .yml file instead of a json file since yaml
is easier for humans to parse (and we can add comments).


I hope that answers all of your questions, let me know if there was
anything I missed!

-John

[Yarko Tymciurak]

Thanks John!

Good pointers;  I'm getting my sea-legs around this... slowly (still things to fix in the code-base to make this a smooth-ride, but mostly there).

So - based on vagrant and ansible docs, I used json;  didn't try yaml, though will (agree it would be easier - will confirm works w/ extra-args everywhere today);

RE: your comments on not using "base" - yeah, I like that... but...

it would be good to have a list of past "starting boxes" - for example, in a "README.boxlist.md" somewhere, with current box list, and dates (I'm thinking for "usual development and maintenance", you don't need to keep forever - it's just a convenience;  but maybe 2 or 3 boxes prior to current 'release' backlog would be nice;  that's an extra 10G-15G or so of storage).

For example:  starting w/ focaccia, and trying to go to the snapshot of a --before="2014-01-20"  commit I would expect to be problematic in hard to spot ways.   A list of past boxes, with their dates would help.   One can start from base.   That adds hours to the process.

Again - thanks for all this (ansible, settable variables) - clearly, you guys have been thinking about this, even if it is not widely used yet, I expect it will be.

- Yarko

[Yarko Tymciurak]

One more comment, John:

vagrant-devstack.yml  vars are a good pointer, but that is neither a sufficient nor necessary list to do "Open edX base product development".

Fortunately, extra-vars override things which are called in, so one of the things I want to ensure / watch is a dictionary of the vars which

are "to be paid attention to" and used in the playbooks (I don't think they all belong lumped in vagrant-devstack.yml).

For example, forum_source_repo is in roles/forums/defaults, as well as edx-east and edx-west playbooks.

This is an important thing to "freeze" a location and version for development on a fork (even if you're only developing edx-platform, it interacts w/ a particular forum - granted, less likely you'll be pinched by defining a snapshot in the short-run).

This is what I meant by "having a release snapshot everything that goes into it":  by definition, the focaccia box is "frozen",

but the release/devbox's base is _not_ frozen.   Indeed, this can fluctuate between bringups - a potentially significant source of developer frustration!

If the stock vagrant bringups used _all_ the elements which comprise the bringup, and explicitly cataloged them in a devstack yaml,

then it seems at least two things would result:

• what versions went into _this_ focaccia / this release would be explicit;
• if it wasn't hidden away from the developer in the configuration repo (I can hear you already!), then it would be quickly and immediately separate devops catalog + task items from development & baseline config items (by override - since devops needs those also).

While at one level it "makes sense" to leave the devops vars and configuration tasks of (say) forums to the forums repo, from a release-freeze perspective (and I think of vagrant-devstack as a form of a release), "more explicit" is "more better" I think.

And that's where I'm heading, to enhance clarity.

"These are the things in my gaggle of cooperating servers that is edx at this point?  Ok."

Hiding implementation details is good at one level.

Making a high level structure explicit is good at another level (and functional too - since your "vagrant-devstack.yml" as it is, doesn't have sufficient declarations to define "a release" - things are left "moving" in versions which vagrant-devstack depends on).

Let's keep talking.  Keeping it simple as possible (and no simpler!) is good.

Regards,

Yarko

[Yarko Tymciurak]

This is important enough, let me try to restate in a shorter (tl;dr) way:

In effect:

• focaccia.box isn't a Open edX developer's release;
• edx/configuration,  tag: release-focaccia is;
• and even that is not completely true;
• if any of the devops items have "head" or "master" as their version, and you did not also tag/freeze those repo's with "release-focaccia", and also capture that in some vagrant-devstack.yml, then you didn't really create a reproducible release yet.
• edx/configuration, version:  release-focaccia isn't what developers checkout to use vagrant-devstack;
• so development needs some easier way to keep in front of, be aware, change configuration (repo+version) of the moving pieces;
• that yaml developer's inventory needs to be a separate item from the configuration repo devops inventory of all the various playbooks

Make sense?

Hopefully that helps better explain what I'm seeing / my current perspective in this.

Kind regards,

- Yarko

[Yarko Tymciurak]

... continuing:

my current (developer's struggle) way of solving this is to put a "provisioning build_version.yml", and a template to put it front-center in front of vagrant developers what their big knobs & levers are.

I sort of see putting this in configuration/vagrant/release/devstack and .../release/fullstack  (and the base one's too).

THAT (a template, and a 'as released' build_version) would constitute a "focaccio" release, as anyone would be able to create the same starting focaccio on their own, and not depend on you to conveniently serve/host that box image.

Making sense?

I'm open to more insight, other ways to accomplish this.

I do think its important to separate development from devops (which is what the playbooks are is for - and indeed, I could see, for example, a fork by mongodb containing their own playbooks/mongodb/*  area next to edx-east and edx-west ... for deployment).

Indeed, you could say the classes of development in edx break into (sort of, roughly):

• blob 1:
• platform   (devstack)
• services (e.g. forums)  (? devstack)
• addons (e.g. xblock dev)  (not devstack)
• blob 2:
• deployment (all the tools, test, config - devops preparation)  (maybe fullstack)
• studio platform (working on the tools to facilitate instructional designers / course authors);  (devstack)
• courseware development  (fullstack)
• blob 3:  (not sure where else to place these)
• theming  (devstack)
• language (internationalization)  (devstack)
• user experience (client / js / etc. specialties)  (devstack or fullstack)
• marketing tools (i.e. jobs, about us, buying certificates, etc. - ancillary web site stuff)  (?)

[Yarko Tymciurak]

From these discussions, I am more convinced than ever that a devstack which actually specs components is key.

I have this on base, and w/ focaccio started to try it on the vagrant/release areas.

I'm going to move my end of this discussion (if there are no complaints) to a pull request - get feedback there.

That probably won't be until sometime beginning of next week.

I hope others will continue to provide their comments.

[Puntxo]

Thank you John,
Does this release include any changes in the databases from the previous empanada release? Could we update our servers and still use the same data without changes?

Regards
Javier

[John Jarvis]

Hi Puntxo - yes there is nothing in the ansible configuration scripts
that will remove old data. You will want to make sure that if you had
variable overrides on the last run (for example site name, ports, etc)
that you set them on the subsequent ansible run otherwise you will
revert to the default values.

-John

[Puntxo]

Thank you John, what I meant to say was if we could use the focaccia release with the old data without any change. We have a separated db server so I'm afraid we can't just point an updated edx platform server to the old db server.

I've noticed there are new records in tables like "auth_user_profile" or "django_content_type", for example. Where can I find the complete change list to apply them to our db?

Best regards

Javier

[John Jarvis]

Hi Puntxo - you can set the edxapp_version to the commit you are
currently running if you don't want to upgrade edx-platform to
matster. Migrations will be safe though and are always backwards
compatible. We use south to manage them and the migrations
directories are in l*/djangoapps/*/migrations.
-John

[Yarko Tymciurak]

A couple of quick updates to this:

• Thanks to Jason Bau's attention, the current master once again supports Stanford-Online/edx-theme as a USE_CUSTOM_THEM;
• with much gratitude to Jason, I've expressed that I'd like to make / keep the Stanford theme as the canonical theme from which to start development of "yours";
• thanks to John Jarvis's suggestion to use yaml, the ansible --extra-args approach works so nicely, is commented that I will adding a pull request to make this the default bringup tool-chain - you won't need to make any edits to Vagrantfile;
• this means that a release like focaccia will be able to conveniently include all the specs so you can re-build your own (it will be a release in that sense);
• you will be able to follow master, lock in at focaccia, or lock at whatever snapshot / branch your development calls for easily, readily, by updating one file;
• once pull #2387 on edx-platform is reviewed, passes tests, this will be useful (if not yet complete)
• it will still have rough edges, but it will be a usable starting point for an orderly development environment

FUN! :-)

- Yarko

[Siddhant singh]

Hi

    Was following https://github.com/edx/configuration/wiki/edX-Ubuntu-12.04-Installation for edx installation on aws. However could not find ami-9c7a2cce in southeast -1 region as mentioned. Has this moved?