scoobi libjars: change default?

[Alex Cozzi]

A lot of my colleagues are bitten by problem causes by libjars: it seems that if an old version of the class is present in libjars the job starts and it killed with some obscure errors.

In fact we got in the habit of wiping libjars before starting our jobs on the hadoop cluster, which kind of defeat its purpose.

I was wondering whether it could be a good idea to NOT use libjars by default, at least until scoobi has a reliable way to check for up-to date versions, instead of defaulting to use libjars with the option to disable it with -- scoobi nolibjars

I think it would avoid confusion for new users.

Alex

 

[Jeff Zhang]

Hi Alex,

+1, This often happens when there's a library that is shared by multiple projects. It's very annoying.

在 2013年2月4日星期一UTC-8下午9时40分18秒，Alex Cozzi写道：

[Eric Torreborre]

Hi Alex,

I agree that uploading the lib jars should better not be enabled by default if it's not working and I can push that change in the next release. I would also like to make it work!

There is one issue I noticed. When you have a new version of a library with a new name, the old one will stay in the libjars directory. I wonder if this is what is causing the issue.

Cheers,

Eric.

[Alex Cozzi]

I think I might have an idea of what is the issue: most of our dependencies are maven snapshot releases, like 0.9-SNAPSHOT. While full releases in maven are immutable, -0.9, and thus once deployed in libjars they can be left alone, SNAPSHOT releases are basically nightly builds, and should be replaced every time. 

I think that the problem are that sometime I change the API in snapshot releases and then then there is a mismatch between client code and library when scoobi does not refresh the snapshot jar in libjars