Where are anduril artifacts published?

[Anton Kulaga]

Hello all,

I started evaluating Anduril for my RNA pipelines. The first I started to do when creating a hello-world pipeline was creating build.sbt , not for running but for code highlighting and autosuggestions. Unfortunately, I did not find any Anduril artifacts neither in bintray nor in sonatype (maven central). I wonder, have you published them somewhere?

I also noticed that in Anduril source code many dependencies are just put as jars inside the lib folder. I wonder what are there reasons for that? If it is just a lack of time to organize dependency management properly I can probably contribute to fix this.

I also wonder, have you seen:

1) Ammonite ( http://www.lihaoyi.com/Ammonite/#ScalaScripts ) that seems to become a standard de facto for scala scripting. If yes, have you considered any ammonite components?

2) ADAM ( https://github.com/bigdatagenomics/adam ) that seems to become a useful part of many pipelines. Are there any plans for ADAM/SPARK components?

[Ville Rantanen]

Thanks for the interest!

I am not that well learned in the current world of Scala coding, but I think there are some clarifications I can give:

* Anduril core is still Java. JAR files in the /lib/ are there for historical reasons, which the introduction of scala for pipeline building didnt change. Anduril = java project.

* Scala is used _only_ to build the network of components. That is, which data files are provided to which processes, and when. Then, the Java core kicks in, and takes care of running a separate process for each analysis step instance. At the moment, no component uses Scala as computation language. All components are written in  Java, CRAN R, Matlab, Python, PERL, or they run external binaries with Bash. Some other languages are also used.  

* Currently, we have had no resources to build any autosuggestion data. In the previous java-only-version of Anduril, we built one for Eclipse, but i don't think many people used it. 

* Anduril dependency handling is a bit tricky IMHO. For many things we try to install everything in user space, but then, in many cases system wide installation is just easier to do. This problem pretty much always appears when you start to get tens and hundreds of different libraries and programs as dependencies. Since a perfect solution is hard to attain, we've built most of the components in a way, that components use whichever tools are already installed in the system. Anduril is an orchestrator of programs which you already know how to use, and have installed. Most of the components come with an installer for the tools, if you didn't have them already.

As to your questions, At the moment, none of the environments you mention have been discussed.

For most of the cases, new components are created when an interesting algorithm/library is published and we consider it useful in our work. Perhaps there is a new alignment algorithm, or an interesting deep learning method. Then, a new component is created with whatever environment that method requires to run. We never port code, we want to run the 'original implementation that was published (and hopefully tested)'. We never create components because of interest in the language or scripting environment.

Merry midwinter,  

Just ask if there's anything that i misunderstood or if you want more comments.  

[Anton Kulaga]

Dear Ville,

>* Anduril core is still Java. JAR files in the /lib/ are there for historical reasons, which the introduction of scala for pipeline building didnt change. Anduril = java project

My remark had nothing to do with the language. In fact, all normal JVM build systems (for JVM it can be: sbt, gradle, maven, cbt, pants etc.) support both Java and Scala projects and all support dependency management as well as artifact publishing. Keeping JARs (in case if they are published somewhere) in the source repository is considered a bad practice everywhere. Usually, people just specify names, versions, and repositories and the build system that they use downloads all artifacts for them during first compilation and changing versions means just changing numbers in config files.

I looked at anduril sources, most of your Java dependencies are outdated (some are 2-4 years old!), that means not only bugs but also dependency clashes for people who are building JVM anduril components. I think it could happen because changing dependencies for you implies downloading jars manually and you could not find time just to update the jars.

>* Currently, we have had no resources to build any autosuggestion data. In the previous java-only-version of Anduril, we built one for Eclipse, but i don't think many people used it. 

You probably misunderstood me. What I mean is that if you publish your JVM projects the users will be able to use whatever build system they want (sbt, cbt, maven, gradle) and get JVM code with code highlightning from IDE of their choice. Here is what I did with my hello-world pipeline https://github.com/antonkulaga/hello-anduril  I created a build.sbt file just to be able to edit my code in IDE of my choice (Intellij imports build.sbt natively, for Eclispe there is a command that generates the project). Unfortunately, as anduril and some of its dependencies are unpublished I added only several dependencies to build.sbt ( https://github.com/antonkulaga/hello-anduril/blob/master/build.sbt ) and put all others to jar files that I copy-pasted from Anduril build. Because of build.sbt for my Scala pipeline code I have type-suggestions from IDE in my project (all IDEs that work with Scala support importing sbt projects)

> Anduril dependency handling is a bit tricky IMHO. For many things we try to install everything in user space, but then, in many cases system wide installation is just easier to do.

Why cannot you simply use any normal build system  (sbt, cbt, maven, gradle, pants, etc.) instead of current weird Ant scripts? They do dependency management by default. And if you need something more advanced I recommend https://github.com/alexarchambault/coursier that integrates nicely to different JVM build-systems and also can be called directly from your code.

Happy holidays!

Best regards,
Anton Kulaga

[Ville Rantanen]

okay, thanks for the clarifications

the jar files are stored there mostly for historical reasons, the code using those has been untouched for a while. 

At the moment nobody is really developing any java components, maybe that's why the system is lacking more modern ways of handling java dependencies. I'd guess mostly they are from 2010 era.

Ant was the normal build system 10 years ago when anduril development started. Currently you can probably only find ant in the bundle roots next to bundle.xml. These scripts could've been replaced just by a shell script that finds the installers and runs them. I dont see how any other build systems would solve the dependency problem however..  

if you know a good build system with something that does at least these:

* sudo and without sudo installation of programs (userspace / systemwide)

* apt-get installations on debian based systems / yum installation on centos

* python packages (via pip and downloading tar balls)

* matlab module downloads

* cran-r / bioconductor package installation

* download and compile C to executable binaries

please tell me, and we could find a better way of dealing with the dependencies.