Fwd: [galaxy-iuc] Required Tool Dependencies and new packages

[Pratik Jagtap]

Communication with Ira and Bjoern.

- Pratik

---------- Forwarded message ----------
From: Pratik Jagtap <pja...@umn.edu>
Date: Fri, Mar 7, 2014 at 8:02 AM
Subject: Re: [galaxy-iuc] Required Tool Dependencies and new packages
To: Björn Grüning <bjoern....@gmail.com>
Cc: Ira Cooke <irac...@gmail.com>, Timothy Griffin <tgri...@umn.edu>, Bart Gottschalk <bgot...@umn.edu>

Hello Björn,

Great to hear from you ! I am copying Tim Griffin (the PI on Galaxy-P grant) and Bart Gottschalk (the project manager on Galaxy-P grant) on this mail.

Please see my replies:

> Great to hear that! I'm really confident we can create something really nice in the next month!

Thats great to hear. I have mailed Lennart Martens and his group - so that they are aware of the progress and might be able to coordinate with you regarding PeptideShaker integration.

> Sure, please add my gmail Address.

I will go ahead and add your name.

> You got an invitation for the trello board, please let me know if any I should add more interested persons.

I have copied Tim and Bart so that they can let you know of their interest and on any additional members that can be added.

> Trello is more a bugtracker than a mailinglist and the galaxy project is using it, so I thought it would be a good idea to > streamline that effort. Imho, mailinglists/groups are good to discuss topics and answer questions, the end result of 

> such a discussions or plans should go into the trello board. So yes: Groups for users, Trello for Developers.

This sounds like a good plan according to me. If we coordinate this well - we could achieve a lot. Tim and Bart might have some inputs as well.

Thanks and Regards,

Pratik

Pratik Jagtap,

Managing Director,

Center for Mass Spectrometry and Proteomics,

43 Gortner Laboratory
1479 Gortner Avenue
St. Paul, MN 55108

Phone: 612-624-9275

On Fri, Mar 7, 2014 at 7:41 AM, Björn Grüning <bjoern....@gmail.com> wrote:

Hi Pratik,

Am 07.03.2014 00:00, schrieb Pratik Jagtap:

Hello Björn and Ira,

Thanks for your mails.

As Ira must have mentioned we are very interested in installing some
interesting tools, getting them tested by users and improving on them as we
move forward.
We have identified MS-GF+ and PeptideShaker that we would like to start
with and have it tested in Galaxy environment.

Great to hear that! I'm really confident we can create something really nice in the next month!

Björn - would you be interested in joining the google group?

Sure, please add my gmail Address.

I will be

happy to share ASMS abstract that we have submitted for this.
There are many interesting prospects and we also anticipate some 5600+ data
at a latter point as well.

:)

Ira - I am planning to write a mail to Martens group on jump starting this
project. I will keep you and Björn posted regarding this.

I saw that Trello was mentioned in your thread. Can we discuss on
possibilities on how this can interface with the galaxyp google group? I am
anticipating the google group to be a user-centric forum - while
  Trello could be a developer-centric forum. Any thoughts / discussion will
be greatly appreciated !

You got an invitation for the trello board, please let me know if any I should add more interested persons.

Trello is more a bugtracker than a mailinglist and the galaxy project is using it, so I thought it would be a good idea to streamline that effort. Imho, mailinglists/groups are good to discuss topics and answer questions, the end result of such a discussions or plans should go into the trello board. So yes: Groups for users, Trello for Developers.

Thanks,
Bjoern

Cheers,

Pratik







Pratik Jagtap,
Managing Director,
Center for Mass Spectrometry and Proteomics,
43 Gortner Laboratory
1479 Gortner Avenue
St. Paul, MN 55108
Phone: 612-624-9275


On Thu, Mar 6, 2014 at 4:24 PM, Björn Grüning <bjoern....@gmail.com>wrote:

Hi Patrick, lets rock the Proteomics world :)



  Hi Bjoern,

Thanks for this.  I am cc'ing Pratik Jagtap who is from the GalaxyP
project.  He's particularly interested in progress on the msgfplus wrapper.

Pratik .. just so you know the context.  Bjoern is doing alot to enhance
proteomics support for tools in the toolshed, especially installation of
dependencies.  We have also made some progress on getting the msgfplus
wrapper working but there is more to do.  We're tracking progress using the
Trello board (as that is what the galaxy team use).  I will report broader
progress to your google group when we have something more substantial to
report.

Cheers
Ira



On 7 Mar 2014, at 12:19 am, Björn Grüning <bjoern....@gmail.com>
wrote:

  Hi Ira,

you should have a link somewhere in your postbox. I will inform the
galaxyp project, maybe they want to join.

Thanks,
Bjoern

Am 06.03.2014 10:16, schrieb Ira Cooke:

Hi Bjoern

No worries. I have lots of OS X machines but don't worry about then.
Most galaxy installs are Linux.

Feel free to setup a trellis board. Can you send me a link when you do.
The project is not galaxyp as that is probably specific to the Minnessota
group. Just call it proteomics to make it general.

Cheers
Ira


  On 6 Mar 2014, at 7:11 pm, Björn Grüning <bjoern....@gmail.com>

wrote:

Hi Ira,

  Hi Bjorn,

Thanks for trying ... but I think we should just assume libxml-dev is

installed.  I am also planning on removing as many dependencies from protk
as I possibly can.  I want it to be lightweight but it has become quite
bloated.

Also ... what do you think about leaving out the openssl and iconv

dependencies for ruby.  I have found that things work OK on my ubuntu setup
without those installed .. and on macosx it won't work either way ;).

Do you have a OS-X, I never had one and can not test on one. So I
don't know what is working there and what is not.

The openssl dependency is a good one to remove because it's quite

heavy ... as it also includes perl which takes forever.

What do you think?  Shall we just leave the package_ruby's as they

are for now? (i.e. ... the version with no dependency installs)

Yes, lets keep it simple for the moment. Maybe we can track our
progress and attempt into some bug-tracker? Do you mind if I set up a
GalaxyP trello board? Is your project also called GalaxyP?

  Cheers

Ira

  On 6 Mar 2014, at 7:53 am, Björn Grüning <bjoern....@gmail.com>

wrote:

Ok, I tried a lot but ruby will not pick up the library path from
libxml ... is that a ruby bug, or a libxml-ruby bug or do they use a
different ENV-var?

If you did not know, we simply assume that libxml-dev is installed
until I have patch galaxy to support --with-xml2-lib and sutch things ...

Am 05.03.2014 20:08, schrieb Ira Cooke:

Hi Bjoern,

Thanks but I'm guessing this hasn't solved my issue since I can't
see what could have changed (can't test for a few hours yet sorry. I am not
at work).  It seems like you only need libxml and not libxml-dev ... but
should we be testing with only the galaxy core packages which don't include
libxml right?

Cheers
Ira


  On 6 Mar 2014, at 6:01 am, Björn Grüning <

bjoern....@gmail.com> wrote:

So its installing for me :)

Can you also try. I havn't installed libxml2-dev but libxml is
installed. I also do not need any modification for protk, I only changed
package_ruby. If you confirm that its working I will add my changes to all
ruby_packages.

Cheers,
Bjoern

Am 05.03.2014 19:14, schrieb Ira Cooke:

Hi Bjoern,

iconv compiles for me ... but i think that was a distraction ...
it's the libxml stuff that is causing me the most worry.  If you can help
with that it would be really great.

If you want a test case for debugging simply try to install

http://testtoolshed.g2.bx.psu.edu/view/iracooke/package_
protk_1_2_6

On a system without libxml installed.  I have a vagrant virtual
machine I use for this ... happy to send my vagrantfile if you want it.

Cheers
Ira


  On 5 Mar 2014, at 6:35 pm, Björn Grüning <

bjoern....@gmail.com> wrote:


Hi Ira,

  Hi Bjoern,

Firstly thanks for your tip on hacking the installed xml ...

that's saving me lots of time :)

I've found that installing ruby ... and then subsequently

installing gems that need to compile packages is frought with issues unless

you have libxml2-dev installed using the package manager.  I
tried for ages to get this to work using the libxml2 package on the
toolshed but gave up.

Of course, without this ruby compiles just fine ... but when you

try to install the libxml-ruby gem (used by a lot of things) you run into
trouble.

So the final goal is to have ruby and libxml-ruby compiling,
right? I suspect that the problem right know is that libxml is not
exporting anything besides pkg-config-path. Have you tried to depend on
libxml during libxml-ruby installation? I will fix that for you and you can
care about more important stuff :)

  I have added a note to the read me that this needs to be

installed.

As part of my cleanup of protk I am hoping to reduce the

dependency load ... hopefully I can even eliminate the libxml-ruby gem

dependency as ruby does have a built-in (slow) xml library.

I think that will not be possible, I try to figure that out, for
the time being please assume its installed.

Thanks,
Bjoern

P.S. does iconv compiles for you?

  Cheers

Ira


  On 5 Mar 2014, at 10:58 am, Björn Grüning <

bjoern....@gmail.com> wrote:

Hi,

I'm abusing the testtoolshed. But once installed I change the
installed xml file and start debugging from that. So usually one revision
is enough to get in into my test-galaxy, debugging from the toolshed
installed version under shed_tools/../.../... and if all works I upload it
again and put it back in github.

Sleep well,
Bjoern

Am 05.03.2014 00:55, schrieb Ira Cooke:

Thanks for that.

By the way .. when you are testing do you run your own
toolshed?

I feel very bad making my testing changes on the testtoolshed

... but if I run my own toolshed then I will need to install all required

repositories on there to test stuff.

Just wondering how you do this?

Cheers
Ira


  On 5 Mar 2014, at 10:19 am, Björn Grüning <

bjoern....@gmail.com> wrote:

Hi Ira,

I do not think you need to unpack it by your own:

<action type="shell_command">tar -zxf
ruby-2.0.0-p451.tar.gz</action>
<action type="change_directory">./ruby-2.0.0-p451/</action>

That is done by Galaxy if the folder inside of the tarball
has the same name a the tarball. Otherwise you can use specify the download
name with 'target_filename=""' if you have target filename the same as the
folder contained in the tarball it will be extraced automatically.

But the really bad mistake was that you
"set_environment_for_install" before you downloaded anything ... For some
reason I never understood you always need to have a download a pull or
something like that before you do any other stuff ... so please see the
attached file, maybe it will work.

Thanks,
Bjoern

Am 04.03.2014 23:31, schrieb Ira Cooke:

Hi Bjoern,

I just checked out your galaxy tools package.

Don't worry about overwriting any changes I made yesterday.

  I did make changes to your ruby_2.0 package ...

but to be honest I'm still quite confused about why ruby
won't work properly.  I would like to build ruby with libxml2, iconv and
openssl

->  http://testtoolshed.g2.bx.psu.
edu/view/iuc/package_openssl_1_0

support ... but at present this doesn't work in the version I

have on the test toolshed.

Cheers
Ira





  On 5 Mar 2014, at 9:13 am, Björn Grüning <

bjoern....@gmail.com> wrote:

Hi Ira,

just a private message, can you please apply your changes
also to my github account. If you only update the toolshed, it can happen
that I just overwrite your changes if I upload them the next time. Because
I do not diff both version before updating ...

If you have a github account you can get direct commit
access.
Thanks,
Bjoern

Am 04.03.2014 23:09, schrieb Ira Cooke:

Hi Bjoern / devteam,

package_libxml2_2_9_1 by devteam

is an example of a package that doesn't define
LIBORTOOLNAME_ROOT_PATH

As and when I encounter these I'll just fix them myself
if they are iuc owned .. but if they are devteam I will send a message.

Ah yes .. as you say a pkg-config script is needed in
order to append to PKG_CONFIG_PATH.  In terms of downstream tools making
use of that I imagine that

<action type="autoconf"> will generally work because

configure scripts will generally use PKG_CONFIG_PATH ... but for tools

without a configure script, but which have a makefile one would need to

manually invoke pkg-config ... to set all the right environment variables.  I

must admit I wouldn't really know how to do that so I'd find myself falling
back to setting things manually if I knew the _ROOT_PATH for all the
packages I needed via <action type="shell_command"


Cheers
Ira




  On 5 Mar 2014, at 7:49 am, Björn Grüning <

bjoern....@gmail.com> wrote:

Hi Ira,

  Hi Bjoern,

I've looked at a few of the iuc, devteam and your
packages over the past few days and they seem to differ slightly in the way
they define environment variables.

I'd like to propose that all tools define an
environment variable LIBORTOOLNAME_ROOT_PATH as other developers can always
use

this as a last resort if something needs defining
explicitly (eg --with-XXX-dir passed to configure).

Yes I started it sometime ago because I used it as you
described. Feel free to point me to repositories where I missed it.

  Also, I've noticed that some tools add their path to

PKG_CONFIG_PATH, whereas others explicitly add to LD_LIBRARY_PATH etc.
It feels like this is something we should standardise,
but I'm not knowledgeable enough to know the right choice.

As far as I know the package needs a pkg-config script.
LD_LIBRARY_PATH should work without pkg-config. If I can add
PKG_CONFIG_PATH to any package please let me know.
Any has experience with that and would recommend a
standard-procedure?

Thanks Ira,
Bjoern


  Cheers

Ira


  On 4 Mar 2014, at 11:34 pm, Björn Grüning <

bjoern....@gmail.com> wrote:

Dear IUC members,

over the last couple of weeks we tried to define which
dependencies are recommended to have installed on a Galaxy server. The
outcome can be seen here:

https://wiki.galaxyproject.org/Admin/Config/
ToolDependenciesList

The list was originally collected here:

https://trello.com/c/7VTlX9rD

As you can see I have wrapped most of the dependencies
and the list is now much shorter. For the rest of dependencies, especially
for libatlas, liblapack and so, we should advertise to have some installed
(not as hard dependency, as soft-toolshed-dependency) and write something
like to following into every tool that depends on it: "required is libatlas
liblapack".
I will try to find time to wrap the missing ones.
Please comment on that wiki page and add new packages that needs to be
wrapped...

I think we are now in a good shape, since most of them
are passing the ToolShed Tests. What I haven't tested yet is if I exported
the correct/needed env variables, here any review would be appreciated.
Also it would be nice if we can go through all of
these package_* from IUC and devteam and rate them. I think it is trivial
in most cases and we can learn a lot from the other definitions :)
I would like to define a milestone to have all
package_* from IUC and devteam rated and reviews by GCC-2014, what do you
think?

All repositories can be found here:

https://github.com/bgruening/galaxytools

please integrate every change also into that github
account (I will provide you with access).

Any comments?
Thanks,
Bjoern

_______________________________________________
galaxy-iuc mailing list
galax...@lists.bx.psu.edu
http://lists.bx.psu.edu/listinfo/galaxy-iuc

<tool_dependencies.xml>