Not able to import dependency

[Sander van Dijk]

Hey all,

I am trying to import a jar as dependency for us in a spark notebook, but I don't seem to be able to find the right way. I'm using spark notebook 0.5.2, spark 1.2.1, hadoop 2.6.0, with Mesos master. The jar is a fat jar built with sbt-assembly, I can use it with spark shell --jars flag, and with spark-kernel with the %addJar magic. The jar is on all instances at /mnt/foo.jar

What I have tried:

* Set 'spark.driver.extraClassPath' and 'spark.executor.extraClassPath' to '/mnt/foo.jar', both in the notebook metadata and using reset()

* Start spark-notebook with 'SPARK_CLASSPATH=/mnt/foo.jar ./bin/spark-notebook'

* Install the jar into local maven repository, with:

    mvn install:install-file -Dfile=/mnt/foo.jar -DgroupId=com.bar -DartifactId=foo -Dversion=1.0 -Dpackaging=jar

And set in the notebook's metadata:

    "customDeps" : [ "com.bar & foo & 1.0" ],

Also tried without this and running ':dp com.bar & foo & 1.0' in a cell instead.

None of this seems to work, tested by for instance running `import com.bar` (again, that works in spark shell).

With the 'customDeps' meta data I do get this additional output when running spark-notebook:

[info] :: resolving dependencies :: org.scala-lang#scala;2.10.4
[info]  confs: [compile, test, runtime]
[info] :: resolution report :: resolve 27ms :: artifacts dl 0ms
[info]  :: modules in use:
[info]  ---------------------------------------------------------------------
[info]  |                  |            modules            ||   artifacts   |
[info]  |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
[info]  ---------------------------------------------------------------------
[info]  |      compile     |   0   |   0   |   0   |   0   ||   0   |   0   |
[info]  |       test       |   0   |   0   |   0   |   0   ||   0   |   0   |
[info]  |      runtime     |   0   |   0   |   0   |   0   ||   0   |   0   |
[info]  ---------------------------------------------------------------------
Update report:
        Resolve time: 27 ms, Download time: 0 ms, Download size: 0 bytes
        compile:
        test:
        runtime:

I guess my jar is supposed to shop up there as well if it works?

What am I doing wrong? Should my jar have a certain property, does it just not work when it is installed in my local maven repo? The mvn install was done by the same user who runs ./bin/spark-notebook. Is there somewhere I may get more output to see what goes wrong?

Many thanks in advance for any insights!

Cheers,

Sander

[andy petrella]

Hello Sander,

Thank you for the detailed explanations, that will help us to find a potential solution.

For instance, when you have an uber jar you could use :cp that will inject it in the driver/repl/notebook classpath.

But if you use your local maven repo where sbt (in 2.10) can fetch it, then
1/ you need to tell where is the repo. It’s maven, then you can use the below options to configure it with the value [ "my-m2 % default % file:///home/<username>/.m2/repository % maven" ]

• :remote-repo in a notebook cell (→ no need for the square brackets here )
• customRepos in the notebook’s metadata
• manager.notebooks.custom.repos in the application.conf

2/ you’ll can use [ "com.bar % foo % 1.0" ] as value for

• :dp in a notebook cell (→ no need for the square brackets here )
• customDeps in the notebook’s metadata
• manager.notebooks.custom.deps in the application.conf

WARN You used  &  as separator in your deps, actually, you should have used % as above (point 2)

​

HTH

andy

--
You received this message because you are subscribed to the Google Groups "spark-notebook-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-...@googlegroups.com.
To post to this group, send email to spark-not...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/CAK3m2_LQZe1x03xiPKkVmQwhgp9yr_U58GdCdTe81_Y2iPVtuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Sander van Dijk]

Hi Andy,

Thanks! Firstly, of course you're right regarding & vs %, I don't know how I missed that. Changing that gives me output that it is actually trying to find the dependency (still fails, but enough to make it work). Secondly, I thought I tried :cp before, but if so I did it wrongly, because now it works.. Thanks for the great work!

Sander

To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/CAKn3j0tPFxOWuWy3yzkcMS_Sg_d06qPhEwu3C7psEnHGC_Y1pw%40mail.gmail.com.

[andy petrella]

\o/
#MeHappy 

[joao....@sessioncam.com]

Hi Andy,

I carefully read the previous posts but still cannot add a custom dependency,

I have scala code that I want to use with the spark-notebook. I published using "sbt pusblish", copied the resulting folder (/home/hadoop/repo) to a remote machine in which I have spark and notebook running.

In the notebook I defined the local repository with the line

:local-repo  /home/hadoop/repo

Then I added the dependency:

:dp "mygroupid % myartifactid % myversion"

However, I get the following warning:

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

[warn]  ::          UNRESOLVED DEPENDENCIES         ::

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

[warn]  ::mygroupid#myartifactid;myversion: not found

[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

sbt.ResolveException: unresolved dependency: mygroupid#myartifactid;myversion: not found

However, the output of sbt publish is in /home/hadoop/repo.

What am I missing?

Cheers,

Joao

\o/
#MeHappy 

To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-user+unsub...@googlegroups.com.

To post to this group, send email to spark-not...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/CAK3m2_LQZe1x03xiPKkVmQwhgp9yr_U58GdCdTe81_Y2iPVtuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "spark-notebook-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-user+unsub...@googlegroups.com.

To post to this group, send email to spark-not...@googlegroups.com.

[andy petrella]

Hello Joao,

oki, so normally before this report, it should also print the repository it used.
Can you paste this too?

Also, can you give me more information on what exactly contains this /home/hadoop/repo?
Like pasting here the results of ls -la /home/hadoop/repo/* should be fine

This will help me trying to understand what happens.

thanks a lot

​

To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-...@googlegroups.com.

To post to this group, send email to spark-not...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/CAK3m2_LQZe1x03xiPKkVmQwhgp9yr_U58GdCdTe81_Y2iPVtuQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "spark-notebook-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-...@googlegroups.com.

To post to this group, send email to spark-not...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/CAKn3j0tPFxOWuWy3yzkcMS_Sg_d06qPhEwu3C7psEnHGC_Y1pw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

This message is confidential and is intended to be read solely by the addressee. The contents should not be disclosed to any other person or copies taken unless authorised to do so. If you are not the intended recipient, please notify the sender and permanently delete this message. As Internet communications are not secure, the sender accepts neither legal responsibility for the contents of this message nor responsibility for any change made to this message after it was forwarded by the original author.

--
You received this message because you are subscribed to the Google Groups "spark-notebook-user" group.

To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-...@googlegroups.com.

To post to this group, send email to spark-not...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/765dc826-c60f-4dad-967b-1b01a3ec2704%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

andy