Call for feedback on Upgrade Bundle technical requirements/design

[Chris Tweney]

Nakamurians,

In the spirit of open community and participation and all those good
things, I want to solicit input on requirements for an Upgrade Bundle.
This is the long-awaited tool that will provide a uniform, reliable,
decoupled framework for tweaking data structures from one release to the
next.

I've put down a sketchy technical requirements/design at [1].

Please have a look and offer your feedback on list!

Thanks,

-chris

[1]
https://confluence.sakaiproject.org/display/KERNDOC/KERN-2147+Upgrade+bundle

--
Chris Tweney, Senior Software Developer
Educational Technology Services
University of California, Berkeley
ch...@media.berkeley.edu

[Ray Davis]

The requirements sound like a very reasonable first guess, though I
wouldn't be shocked to seem them tinkered with after some experience.

Regarding the "Possible Requirements", and speaking from my local point
of view as a deployer:

"1. Don't accept web requests while any upgraders are running"

I don't think this would provide any extra value for us. Since we have
an Apache httpd front-end and a firewall, we can block access by normal
non-localhost users easily enough while the upgrade is being logged. And
we'll probably want to do a little behind-the-firewall checking after
upgraders are finished and before letting normal users back in.

What *would* be useful would be some way to block all "normal"
operations (scheduled jobs, message delivery, and so on) while an
upgrade is underway. In other words, provide something that can take the
upgrading role played by standard SQL admin tools in Sakai CLE. I
suspect that this goal is unrealistic given the interdependency of the
OAE, but mention it just in case I'm wrong.

"2. Upgrade should run automatically, without need for administrator
intervention"

Not needed by us.

"3. Support for a dry-run mode where data doesn't actually changed but
interesting log info is available"

I can see how bundle interdependency might make realistic reporting more
difficult, but this still gets a non-binding +1 from me.

Thanks,
Ray

[Chris Tweney]

Thanks for your feedback, Ray!

On 9/6/11 10:48 AM, Ray Davis wrote:
> "2. Upgrade should run automatically, without need for administrator
> intervention"
>
> Not needed by us.
>

I hope I can persuade you that it is needed. Consider: If upgraders get
run as soon as their bundle is activated, then nobody will ever have to
ask "did you run upgrade x.y.z in bundle so-and-so?" Basically, if it's
automatic, developers will always have the latest data schemas. If it's
manual, they'll always have to guess (or start from clean repos).

-chris

[Eli Cochran]

Adding the OAE Pilot Support Group and the front-end devs to this thread. 

This was one of the high priorities for the pilot support group. 

- Eli 

Begin forwarded message:

--
You received this message because you are subscribed to the Google Groups "Sakai Nakamura" group.
To post to this group, send email to sakai-...@googlegroups.com.
To unsubscribe from this group, send email to sakai-kernel...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/sakai-kernel?hl=en.

. . . . . . . . . . .  .  .   .    .      .         .              .                     .

Eli Cochran

project manager, CalCentral project

Educational Technology Services, U.C. Berkeley

"Software is hard" - Donald Knuth 

[Ray Davis]

Yes, I understand the benefits of auto-update to developers. But I was
speaking as a production deployer, and I don't think we currently want
to take the risk of unanticipated, unscrutinized large-scale updates to
production data. An automatic start-up check that the necessary updates
have been applied would be terrific. An automatic update, not so much.

In the Hibernate-based projects of Sakai CLE, auto-update is fairly
easily toggled. Something like that would be ideal, I think.

Make slightly more sense?

Thanks,
Ray

[Chris Tweney]

Seems reasonable. To summarize:

* Upgrader service has an "automatic" setting defaulting to "off"
* If automatic setting is off: On startup, Upgrader looks for registered
Upgraders that haven't yet run, and logs a warning (or error) if any are
found
* If automatic setting is on: On startup, Upgrader gets registered
Upgraders and runs 'em
* Upgrader also has a "dry run" setting, defaulting to "true". When "dry
run" is true Upgraders will log lots of fascinating data but not
actually do anything. This is orthogonal to the "automatic" setting.

Did I mangle anything?

-chris

[Ray Davis]

Looks beautiful from here on the other side of the desk.

Thanks again,
Ray

[Nate Angell]

For further thoughts: Drupal has an admin "status" page that does a
nice job of summarizing a variey of system statuses, including things
like you are talking about here. As an added feature, Drupal can
notify the system administrator via email automatically about pending
status issues. It's all tied to a scheduled job.

Let me know if a screenshot would help.

- Nate

[John Norman]

I'd like to see the problem structured a bit more into stages. It seems to me that there are some basic things that are needed immediately, but others that are potentially nice to have and apply to scenarios that we don't yet experience. Manual upgrades are a pain, but I am not sure how much return on investment this work would produce compared to other priorities.

J

_______________________________________________
production mailing list
produ...@collab.sakaiproject.org
http://collab.sakaiproject.org/mailman/listinfo/production

TO UNSUBSCRIBE: send email to production-...@collab.sakaiproject.org with a subject of "unsubscribe"

John Norman
Director - CARET
University of Cambridge
jo...@caret.cam.ac.uk
+44-1223-765367

[Eli Cochran]

John,

I agree that we should do this in stages. Could you elaborate a bit on which use cases you feel are "basic" and which you think are "nice to haves". I'm sure that would help Chris some up with a plan that properly scopes the project.

I know that upgrading is very critical to the pilot institutions. The way that OAE stores data makes it very easy to introduce seemingly minor changes to the data that result in user facing errors and problems. For example, a very small change to the UI to turn-off permissions on categories in personal profiles never surfaced for our users because it required a tiny data migration. Luckily we caught it in QA. (I'm not grumbling, we went to production before 1.0 so that's life.) Now that we are at 1.0 and a number of institutions are in production, we need a process where all changes like this and more serious ones can be captured, documented and migration code written. 

Thanks,

Eli 

User Interaction Developer &

Manager of User Experience Design

Educational Technology Services, UC Berkeley

"The opportunity lost by increasing the amount of blank space is gained back with enhanced attention to what remains."

    - John Maeda

[John Norman]

OK, I'm going to stop here. I am not challenging the desirability of a data upgrade service and I am certainly not challenging "...we need a process where all changes like this and more serious ones can be captured, documented and migration code written". When I looked at the specification work, I got the impression of a two people for 3 months sort of project and I wasn't sure that would be the biggest priority for that level of investment right now, given the resource available and the other things that need to get done. Thus the call for triage, to do the minimum for now so we can get other things done. I am not qualified to do the specification - that needs to come from the development team. Priority setting from the team leads and project manager.

I'll leave it there.

John

[Eli Cochran]

+1

Sent from my iPhone

Please excuse any spelling errors and the brevity of my reply. 

On Sep 10, 2011, at 12:57 PM, "Zach A. Thomas" <zach....@gmail.com> wrote:

Your point is well taken that we need to be disciplined about the scope of this (and indeed all) work. I think I can boil the essential requirement down to this: No institution shall need to devote a senior developer for sixty hours to a data conversion to move between version 1.x and 1.y

Luckily, the scope of the first pass of this capability should fall in the range of not more than a few weeks for one good dev.

Zach

[Eli Cochran]

Adding oae-pro...@collab.sakaiproject.org and oae...@collab.sakaiproject.org onto the thread. It's going to take us a while to remember these new lists. 

- Eli 

Begin forwarded message:

manager of user experience

user interaction developer

ETS, UC Berkeley

[Chris Tweney]

John--

Not to worry: this is more of a 1 person for 2-3 weeks type of project,
as specified (not including the "Future Work" section). As Zach said,
the basic core requirement is that it shouldn't require 60 hours of
senior dev time to write a data upgrade. We can get there with the 2-3
week investment and then build on it later as we have time.

-chris

>>>>> *From: *Chris Tweney <ch...@media.berkeley.edu
>>>>> <mailto:ch...@media.berkeley.edu>>
>>>>> *Date: *September 2, 2011 2:52:11 PM PDT
>>>>> *To: *Nakamura List <sakai-...@googlegroups.com
>>>>> <mailto:sakai-...@googlegroups.com>>
>>>>> *Subject: **[sakai-nakamura] Call for feedback on Upgrade Bundle
>>>>> technical requirements/design*
>>>>> *Reply-To: *sakai-...@googlegroups.com
>>>>> <mailto:sakai-...@googlegroups.com>

>>>>>
>>>>> Nakamurians,
>>>>>
>>>>> In the spirit of open community and participation and all those
>>>>> good things, I want to solicit input on requirements for an Upgrade
>>>>> Bundle. This is the long-awaited tool that will provide a uniform,
>>>>> reliable, decoupled framework for tweaking data structures from one
>>>>> release to the next.
>>>>>
>>>>> I've put down a sketchy technical requirements/design at [1].
>>>>>
>>>>> Please have a look and offer your feedback on list!
>>>>>
>>>>> Thanks,
>>>>>
>>>>> -chris
>>>>>
>>>>> [1]
>>>>> https://confluence.sakaiproject.org/display/KERNDOC/KERN-2147+Upgrade+bundle
>>>>>
>>>>> --
>>>>> Chris Tweney, Senior Software Developer
>>>>> Educational Technology Services
>>>>> University of California, Berkeley

>>>>> ch...@media.berkeley.edu <mailto:ch...@media.berkeley.edu>

>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Sakai Nakamura" group.
>>>>> To post to this group, send email to sakai-...@googlegroups.com

>>>>> <mailto:sakai-...@googlegroups.com>.

>>>>> To unsubscribe from this group, send email to
>>>>> sakai-kernel...@googlegroups.com

>>>>> <mailto:sakai-kernel...@googlegroups.com>.

>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/sakai-kernel?hl=en.
>>>>>
>>>>
>>>> . . . . . . . . . . . . . . . . . . .
>>>>
>>>> Eli Cochran

>>>> project manager, Cal*Central* project

>>>> Educational Technology Services, U.C. Berkeley
>>>>
>>>> "Software is hard" - Donald Knuth
>>>>
>>>> _______________________________________________
>>>> production mailing list
>>>> produ...@collab.sakaiproject.org

>>>> <mailto:produ...@collab.sakaiproject.org>

>>>> http://collab.sakaiproject.org/mailman/listinfo/production
>>>>
>>>> TO UNSUBSCRIBE: send email to
>>>> production-...@collab.sakaiproject.org

>>>> <mailto:production-...@collab.sakaiproject.org> with a

>>>> subject of "unsubscribe"
>>>
>>> John Norman
>>> Director - CARET
>>> University of Cambridge

>>> jo...@caret.cam.ac.uk <mailto:jo...@caret.cam.ac.uk>

>>> +44-1223-765367
>>>
>>
>> . . . . . . . . . . . . . . . . . . .
>>
>> Eli Cochran
>> User Interaction Developer &
>> Manager of User Experience Design
>> Educational Technology Services, UC Berkeley
>>
>> "The opportunity lost by increasing the amount of blank space is
>> gained back with enhanced attention to what remains."
>> - John Maeda
>>
>>
>>
>
> John Norman
> Director - CARET
> University of Cambridge

> jo...@caret.cam.ac.uk <mailto:jo...@caret.cam.ac.uk>
> +44-1223-765367
>
>
>
> _______________________________________________
> oae-production mailing list
> oae-pro...@collab.sakaiproject.org
> http://collab.sakaiproject.org/mailman/listinfo/oae-production