BioCloudCentral, new CloudBioLinux AMIs and updated usage example

[Brad Chapman]

Hi all;
There has been a ton of on-going work to simplify and improve
CloudBioLinux and CloudMan usage. Thanks to fabulous feedback from
people on this list, at meetings and via e-mail, we've gotten an idea of
some of the difficult points in deploying these tools and have been
working to make everything easier.

Towards those goals, here are some brand new exciting things in the
world of CloudBioLinux and CloudMan:

- BioCloudCentral (http://biocloudcentral.org) -- A web server designed
to easily launch CloudBioLinux/CloudMan instances. This takes away all
of the setup in the AWS console by automating security group setup and
user-data preparation. Source code for the site is on GitHub
(https://github.com/chapmanb/biocloudcentral) if you'd like to
contribute. It's a Django server hosted at Heroku.

- New CloudBioLinux AMIs -- There are two new AMIs, for Ubuntu 11.10 and
10.04 LTS with updated software: R 2.14, GATK 1.3, plus loads of other
updates.

- An updated usage example: http://j.mp/uNXZY6 This is a next-gen
sequencing analysis pipeline that provides all of the steps from signing up
for Amazon to running an analysis and viewing results, all in the
browser with BioCloudCentral, CloudMan and Galaxy. It's got videos and
hopefully will help with demoing what all of these tools are capable
of.

I'm excited to have this all out there and would like to thanks everyone
from the community who has contributed code and ideas. Enis is
especially due a round of thanks: he did all of the Amazon specific work
on BioCloudCentral and helped generalize CloudMan to make all of this
automation possible.

Please let us know if you try this out and have any questions,
Brad

[Diadema]

Hi Brad,

I'm hoping you can help me out here. I spent countless hours trying to access Cloud BioLinux via AWS EC2, but my NX client kept getting authentication failures. So I finally decided to try BioCloudCentral. When I enter the BCC fields into NX, I do get a connection, but it's a generic Ubuntu desktop, not the BioLinux desktop. Also I noticed that some bioinformatic apps are missing from the NX cloud connection (e.g., Act, Arb, Artemis, jemboss, and others). And when I try to do software updates, it keeps failing. So those are my three issues - no BioLinux desktop, missing bioinformatics apps, and unable to update software. Any help you can provide would be appreciated. Thanks.

John

[Brad Chapman]

John;
Thanks for getting in touch, and glad that you managed to get things
running via BioCloudCentral.

> but it's a generic Ubuntu desktop, not the BioLinux desktop.

The new version uses Ubuntu 12.04 which has a totally new window
manager. Tim and the Bio-Linux folks are working on porting over to the
new environment but the main thing you'll be missing is the default
background. You should still have a 'Bioinformatics' tool bar in your
menu drop down.

> Also I noticed that some bioinformatic apps are missing from the NX
> cloud connection (e.g., Act, Arb, Artemis, jemboss, and others).

This also has to do with the new version of Ubuntu: some software is not
yet ported. I checked and it looks like artemis arb and jemboss work
fine so I added these back in and they'll be in the next version. If
there is other specific software you're missing please let us know.

> And when I try to do software updates, it keeps failing.

I started up a new image and I can't reproduce this problem now. What
error messages are you seeing? I was able to update packages with 'sudo
apt-get update' and install, say, jemboss with 'sudo apt-get install
jemboss.' Are these working for you?

Thanks again for the feedback,
Brad

[Tony Travis]

On 11/08/12 17:45, Brad Chapman wrote:
>
> John;
> Thanks for getting in touch, and glad that you managed to get things
> running via BioCloudCentral.
>
>> but it's a generic Ubuntu desktop, not the BioLinux desktop.
>
> The new version uses Ubuntu 12.04 which has a totally new window
> manager. Tim and the Bio-Linux folks are working on porting over to the
> new environment but the main thing you'll be missing is the default
> background. You should still have a 'Bioinformatics' tool bar in your
> menu drop down.

Hi, Brad.

Actually, the FreeNX server provides a 'gnome-fallback' session under
Ubuntu 12.04 LTS, not a remote Unity session. I've tried x2go, which
does provide a remote Unity desktop. However, for the moment, we use
'gnome-fallback'. BTW, the Nomachimne NX 4.0 client does not suspend and
resume properly when connected to a FreeNX server. I tried using
Nomachine's NX 4.0 client on Mac's running MacOS 10.7+ because the NX
3.5.0-7 NX client for MacOS does not work under MacOS 10.7+. The OpenNX
client works correctly with a FreeNX server on Windows, Mac and Linux.

>> Also I noticed that some bioinformatic apps are missing from the NX
>> cloud connection (e.g., Act, Arb, Artemis, jemboss, and others).

These should be present on a "gnome-fallback" session, but I confess
that I've not installed the Bio-Linux packages on our Ubuntu 12.04 LTS
production server. However, the Debian-Med packages are on the menu as
you can see from the screenshot of my OpenNX -> FreeNX session attached.

BTW, I'm now working at the University of Edinburgh (UK) and I'm
interested in running Ubuntu 12.04 LTS CBL on our private cloud.

HTH,

Tony.

[Brad Chapman]

John and Tony;
John, glad that helped sort things out and thanks again for the
feedback. We need to work on getting that desktop image put in the right
place so the environment is a bit more familiar.

> Actually, the FreeNX server provides a 'gnome-fallback' session under
> Ubuntu 12.04 LTS, not a remote Unity session. I've tried x2go, which
> does provide a remote Unity desktop. However, for the moment, we use
> 'gnome-fallback'.

Tim and all, would the easiest path be to update the 'cloudbl-desktop'
package to put links and skeletons in a place that is compatible with
Ubuntu 12.04? I'm totally ignorant of what it would take to move over to
Unity but it sounds like sticking with FreeNX and gnome-fallback for now
makes the most sense.

Brad

[Tim Booth]

Hi Brad,

> Tim and all, would the easiest path be to update the 'cloudbl-desktop'
> package to put links and skeletons in a place that is compatible with
> Ubuntu 12.04? I'm totally ignorant of what it would take to move over to
> Unity but it sounds like sticking with FreeNX and gnome-fallback for now
> makes the most sense.

Sure - I'll update the cloudbl-desktop package at the same time I do the
customisation for the regular Bio-Linux.

Tony looks to have done the most testing with NX varieties so I'm going
to go with whatever he feels is the most stable option. I should have
time to test things on CBL once I get the BL7 Beta release out and I'll
push any necessary changes back into the GIT recipes. This includes
updating the list of packages to match what we have on BL - some are
being added, some retired or replaced.

Cheers,

TIM

> Brad
>

--
Tim Booth <tbo...@ceh.ac.uk>
NERC Environmental Bioinformatics Centre

Centre for Ecology and Hydrology
Maclean Bldg, Benson Lane
Crowmarsh Gifford
Wallingford, England
OX10 8BB

http://nebc.nerc.ac.uk
+44 1491 69 2705
--
This message (and any attachments) is for the recipient only. NERC
is subject to the Freedom of Information Act 2000 and the contents
of this email and any reply you make may be disclosed by NERC unless
it is exempt from release under the Act. Any material supplied to
NERC may be stored in an electronic records management system.

[Tony Travis]

On 13/08/12 15:05, Tim Booth wrote:
> [...]

> Tony looks to have done the most testing with NX varieties so I'm going
> to go with whatever he feels is the most stable option. I should have
> time to test things on CBL once I get the BL7 Beta release out and I'll
> push any necessary changes back into the GIT recipes. This includes
> updating the list of packages to match what we have on BL - some are
> being added, some retired or replaced.

Hi, Tim.

I'm using the PPA version of FreeNX for Ubuntu 12.04 LTS:

> deb http://ppa.launchpad.net/freenx-team/ppa/ubuntu precise main
> deb-src http://ppa.launchpad.net/freenx-team/ppa/ubuntu precise main

On Linux, Mac and Windows clients we are using OpenNX:

> http://opennx.net/download.html

On my laptop (Upgraded in-situ from BL6.0 to Ubuntu 12.04 LTS):

> deb http://download.opensuse.org/repositories/home:/felfert/xUbuntu_12.04 ./
> deb-src http://download.opensuse.org/repositories/home:/felfert/xUbuntu_12.04 ./

Please note that the Nomachine 4.0 NX 'Player' does not suspend/resume
correctly with a FreeNX server under MacOS 10.7+ and I do not recommend
using it. The OpenNX client works well across all three OS platforms.

I suggest we install the OpenNX client by default under BL/CBL7.0 so
that it can be used to connect to a FreeNX server. OpenNX is based on
the wxWidgets toolkit and can be freely distributed.

I'll continue to evaluate x2go, which seems to be Canonical's preferred
option for remote Unity desktops, but it is completely incompatible with
NX/FreeNX. In my hands, x2go is a lot less stable than FreeNX, but first
impressions of its functionality are very good indeed. I think it needs
a little more development before it's a realistic alternative...

Bye,

Tony.

[Tony Travis]

Hi, Brad.

I've just started a new instance of ami-46d4792f to start working on
Bio-Linux menus, and here are some small changes I suggest to make it
easier to do 'unattended' upgrades. Please note that you need to change
ORPHANOPTS to stop "deborphan" from uninstalling too many things(!).

HTH,

Tony.

> # start new instance - ami-46d4792f and login as "ubuntu"
>
> ./configure_freenx.sh
>
> # login again as "ubuntu" via NX client
> sudo -i
> aptitude update
>
> # install missing GPG key
> gpg --keyserver subkeys.pgp.net --recv-keys D018A4CE
> gpg -a --export D018A4CE | sudo apt-key add -
>
> # configure for unattended upgrades
> aptitude install upgrade-system
>
> cd /etc
> cp -ai upgrade-system.conf upgrade-system.conf.orig
> vim upgrade-system.conf
> # ORPHANOPTS="--guess-all --libdevel --priority=6"
>
> # update and check until no changes required
> upgrade-system
>
>
> 1) Updating package lists:
> W: Duplicate sources.list entry http://archive.canonical.com/ubuntu/ precise/partner amd64 Packages (/var/lib/apt/lists/archive.canonical.com_ubuntu_dists_precise_partner_binary-amd64_Packages)
> W: Duplicate sources.list entry http://archive.canonical.com/ubuntu/ precise/partner i386 Packages (/var/lib/apt/lists/archive.canonical.com_ubuntu_dists_precise_partner_binary-i386_Packages)
> W: You may want to run apt-get update to correct these problems
> ...
>
> # remove duplicate entry from list of repositories
> vim /etc/apt/sources.list.d/cloudbiolinux.list
> # deb http://archive.canonical.com/ubuntu precise partner
>
> # update and check until no changes required
> upgrade-system
> aptitude -f install
> ...
> upgrade-system
> aptitude -f install

[Brad Chapman]

Tony;

> I've just started a new instance of ami-46d4792f to start working on
> Bio-Linux menus, and here are some small changes I suggest to make it
> easier to do 'unattended' upgrades. Please note that you need to change
> ORPHANOPTS to stop "deborphan" from uninstalling too many things(!).

Thanks much for these pointers. I added in fixes for the missing GPG key
and duplicated repository lists, so hopefully those warnings will go
away with the next release.

The `upgrade-system` command kind of scares me, especially messing with
the deborphan configuration. Is this for handling upgrades between
releases? The nice thing about AWS images and the automated workflow is
that we can do clean builds with new releases and avoid this. Or am I
missing the point entirely and we need it in there?

Thanks again for these pointers,
Brad

[Tony Travis]

On 23/08/12 01:38, Brad Chapman wrote:
> [...]

> The `upgrade-system` command kind of scares me, especially messing with
> the deborphan configuration. Is this for handling upgrades between
> releases? The nice thing about AWS images and the automated workflow is
> that we can do clean builds with new releases and avoid this. Or am I
> missing the point entirely and we need it in there?

Hi, Brad.

Even the latest CBL image I could find on AWS is well out of sync with
the Ubuntu 12.04 repositories. I've used "upgrade-system" for years to
do unattended upgrades on lots of machines and, as long as you stop
"deborphan" from removing things by lowering the priority as I describe
it really does help to keep systems up-to-date with little effort.

There are always different points of view about upgrading systems but, I
believe, it is best to keep systems up-to-date. In fact,
"upgrade-system" was written to 'sanitise' Debian systems :-)

> Description: command for upgrading and sanitizing a Debian system
> Upgrade-system offers a convenient way to keep a Debian system up-to-date, yet
> free from accumulated cruft such as obsolete libraries.

I'm interested in long-running instances, rather than firing up a new
instance every time I want to do something: I just 'stop' my instances
when I'm not using them. As Tim knows, I'm interested in CBL/BL as a
bioinformatics terminal server, which is why I use it via FreeNX. You
might be interested to know that we are going to do a pilot study at
Edinburgh University running CBL on the ECDF OpenStack private cloud.

Bye,

Tony.