Dataflow development session - 2012-10-22

[Graham Klyne]

Development session held at Bodleian Library Digital Services, Osney mead, on 22
Oct 2012

Present:
- Anusha Ranganathan
- Bhavana Ananada
- Ben O'Steen
- Richard Jones
- Graham Klyne

(This summary is based upon notes made and agreed during the meeting, but has
been expanded in the drafting. The expanded version here has not been reviewed
by the meeting participants, so if there are any errors or misunderstandings,
please let me know.)

We made a slow start due to system disruptions triggered by weekend power outage
at the library services facility, affecting several systems including some of
the development facilities.

During the day, important progress was made on a number of fronts, and a number
of old and new problems that stand in the way of a final release were resolved,
but there remains work to do. Two significant outstanding issues are a
continuing problem with repeated access to the SWORD interface, which has been
identified as a resource leakage problem, and revising silo selection in the
dataset submission interface.

While there is ongoing work to develop a version of Databank based on the Django
web framework, which we believe will provide greater long-term stability, we are
still aiming for an initial release based on Pylons.

...

Reminder of next steps identified at the previous developer meeting on 13-Sep-2012:

(1) Configure and test the clean installation of latest Databank package on
Ubuntu 12.04
(2) Check for SWORD dataset submission server errors with latest version of
Databank
(3) Finish sorting out dataset submission interface
BLOCKER: need to resolve errors from Databank server when submitting.
(4) Recreate Jenkins CI test environment, and restart regular testing from
GitHub sources?
(5) Continue working through JIRA issues

Since that meeting, availability of some developers has been affected by
personal circumstances, and Ben O'Steen has been engaged to update the Databank
software to work with the Django web framework rather than the current Pylons
(Pylons development having ben discontinued).

...

At today's meeting:

(1) Anusha & Ben discussed move to Django for Databank, and in particular
clarified how Databank authentication/authorization is intended to work. The
intent is that a revised Databank will make more use of common framework
authentication/authorization facilities.

(2) Ben and Richard discussed Databank/Sword server integration points and options

(3) We identified and devised a fix for a configuration file problem with the
Sword server as deployed in Databank.

(4) A version of the current Databank software (Pylons based), installed from a
recent Debian package, was configured and basically functional, avoiding
problematic mySQL passwords. This was an important step as up to now we have
not had a clean install of Databank available for testing.

(5) An incompatible SWORD library used by Datastage was identified. The client
library used by Datastage was updated in GitHub.

(6) The submission test case from our previous development session was debugged
and passed muster against the new Databank installation.

(7) A dataset submission from Datastage to Databank using SWORD was completed
(though there may a lingering problem with access to the submitted data - this
needs to be checked).

(8) We confirmed and isolated a test case for failure of Databank with multiple
requests to access the SWORD service document; we have some indication of the
nature of the problem (Repose/auth?), but no fix yet.

(9) Bhavana and Graham discussed silo selection on the dataset submission form -
clarifying requirements and an implementation approach, but little actual
progress was made. (The blocker from the previous session has been resolved, at
least sufficient to allow progress to be made.)

...

Next steps:

* Ben will continue finalizing the Django version of Databank - continuing (1), (2).
* Richard will make small changes to the SWORD server package to facilitate
configuration file deployment - from (3)
* Anusha will roll SWORD configuration fixes into the Databank installation
package - from (3)
* Graham to update Databank installation notes and links in the Dataflow wiki -
from (4)
* Anusha will be looking into the mySQL password choice problem - meanwhile
avoid awkward characters like '/' in the mySQL password given when installing
Databank - from (4)
* A new Datatsage installation package needs to be created and tested - from
(5). This should probably be addressed after the submission interface has been
updated.
* Confirm that Datastage to Databank submission is actually working, by
retrieving and checking a submitted dataset - from (7).
* Anusha will continue investigating the multiple request problem with Databank
and SWORD access - from (8)
* Bhavana to proceed with a single-repository solution for Datastage - from (9)

Further steps carried forward from last meeting:

* Recreate Jenkins CI test environment, and restart regular testing from GitHub
sources?
* Continue working through JIRA issues

#g

[Katherine Fletcher]

Dear Graham,

Thank you for that clear and thorough update. It sounds like the hardware
at the Bodleian really is out to get us...

It looks like you have made some really promising advances (especially in
getting a clean install and getting the test case to go through
successfully). Thanks very much!

If you need any support at all for this, please ask -- otherwise I'll assume
you're all just getting on with it.

Thanks hugely,

Katherine

--