Progress update: Report from Datastage hackathon 16-17 Aug
14 views
Skip to first unread message
Katherine Fletcher
Aug 27, 2012, 5:35:54 PM8/27/12
to dataflo...@googlegroups.com, dataflo...@googlegroups.com
Hello from DataFlow!
Our lead developers are working on multiple projects this summer, which has slowed things down a bit -- but we're not giving up! The team met for a hackathon in mid-August to try to push us over the finish line.
Summary: we still do not have a ready-to-publish installation package, and some work is still needed to get there. Progress was further complicated by the major hardware failure at the Bodleian's service hosting environment, resulting in loss of several virtual machines, which were still in the process of being recovered as we conducted the hackathon.
What we did:
* Reviewed the current state of the software, based on Bhavana's development environment. Two immediate blockers were noted: a UI problem with silo selection, and failure of SWORD submission.
* A workaround (messy but workable) for the silo selection problem was implemented.
* The Sword submission problem was found to be due, in part, from errors coming from the Databank server when retrieving the service document. The error was repeatable independently of Datastage in Bhavana's test environment, but could only be repeated in some of Anusha's test systems.
* Installed datastage on Ubuntu 12.04, creating the installation package from github sources, and documented the process
(https://github.com/dataflow/DataStage/wiki/Datastage---create-debian-package-(rough-notes))
* A problem installing Databank on Ubuntu 12.04 was uncovered. Anusha is looking into this.
* Started looking into Django to try and understand if the silo list selection problem could be improved.
Key problems remain with:
* Obtaining SWORD service document with repeated requests
* Populating silos for selection in the submission interface
* Installing Databank on ubuntu 12.04 server
* Datastage accessing databank repositories outside the development environment
Next steps:
(1) Fix Databank install on Ubuntu 12.04 and/or create development installation on Ubuntu 11.10
(2) Review Sword access problems, isolate and fix (getting external help if needed)
(3) Review submission interface, review how this can work with Django, especially with respect to silo selection
(4) Recreate Jenkins CI test environment (blitzed in hardware failure), and restart regular testing from GitHub sources?
The developers have planned monthly hackathon sessions, focusing initially on the immediate problems and creating packages that install reliably on Ubuntu 12.04. It will probably take another two sessions to pull together a proper 1.0 release, meaning mid-October. We'll let you know how we're getting on -- and if anyone would like to lend a hand in the meantime, please feel free!
Best wishes,
Katherine
Our lead developers are working on multiple projects this summer, which has slowed things down a bit -- but we're not giving up! The team met for a hackathon in mid-August to try to push us over the finish line.
Summary: we still do not have a ready-to-publish installation package, and some work is still needed to get there. Progress was further complicated by the major hardware failure at the Bodleian's service hosting environment, resulting in loss of several virtual machines, which were still in the process of being recovered as we conducted the hackathon.
What we did:
* Reviewed the current state of the software, based on Bhavana's development environment. Two immediate blockers were noted: a UI problem with silo selection, and failure of SWORD submission.
* A workaround (messy but workable) for the silo selection problem was implemented.
* The Sword submission problem was found to be due, in part, from errors coming from the Databank server when retrieving the service document. The error was repeatable independently of Datastage in Bhavana's test environment, but could only be repeated in some of Anusha's test systems.
* Installed datastage on Ubuntu 12.04, creating the installation package from github sources, and documented the process
(https://github.com/dataflow/DataStage/wiki/Datastage---create-debian-package-(rough-notes))
* A problem installing Databank on Ubuntu 12.04 was uncovered. Anusha is looking into this.
* Started looking into Django to try and understand if the silo list selection problem could be improved.
Key problems remain with:
* Obtaining SWORD service document with repeated requests
* Populating silos for selection in the submission interface
* Installing Databank on ubuntu 12.04 server
* Datastage accessing databank repositories outside the development environment
Next steps:
(1) Fix Databank install on Ubuntu 12.04 and/or create development installation on Ubuntu 11.10
(2) Review Sword access problems, isolate and fix (getting external help if needed)
(3) Review submission interface, review how this can work with Django, especially with respect to silo selection
(4) Recreate Jenkins CI test environment (blitzed in hardware failure), and restart regular testing from GitHub sources?
The developers have planned monthly hackathon sessions, focusing initially on the immediate problems and creating packages that install reliably on Ubuntu 12.04. It will probably take another two sessions to pull together a proper 1.0 release, meaning mid-October. We'll let you know how we're getting on -- and if anyone would like to lend a hand in the meantime, please feel free!
Best wishes,
Katherine
Graham Klyne
Aug 28, 2012, 11:49:08 AM8/28/12
to dataflo...@googlegroups.com
Katherine,
Thanks for this... I'd been meaning to draft a public report, but didn't get
round to it.
#g
--
Thanks for this... I'd been meaning to draft a public report, but didn't get
round to it.
#g
--
Reply all
Reply to author
Forward
0 new messages