Spark distribution creating fails due to GitHub.

[Abhi Basu]

Folks:

Need your help. Following the instructions from here - http://spark.incubator.apache.org/docs/latest/running-on-mesos.html, to install Spark on Mesos to be able to access data on HDFS. Next step will be Shark after all of this works.

When it comes to creating  a Spark distrib (using make_distribution.sh), it fails due to github connection error. I know that does not directly related to what you do, but why the need to contact github again when the source is all available locally to build the distribution.I think Spark installation needs to be more simplified.

Adding the following does not help either:

git config –-global http.proxy http://proxy.xxxx.com:999

git config –-global https.proxy https://proxy.xxxx.com:999

git config --global url."https://".insteadOf git://

In short, need help to get over this!

Thanks.

[Abhi Basu]

So, any help? 

Most people work in environments with proxies on it. Can anyone provide a simpler build process for Spark or provide binaries required for Hadoop 1.0.3.

Thanks,.

Abhi

[Matei Zaharia]

Hi Abhi,

Can you run make_distribution.sh outside the firewall? It just builds a binary package that you can then distribute.

As far as I can see, the only place it would be talking to GitHub is in downloading a SBT plugin from there to build the project. But you can build the project on any machine and get the right binaries.

Matei

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Henrique Rodrigues]

Hi Abhi

You can try this: https://gist.github.com/sit/49288

However, if your network is blocking the git:// port as well (9418), you might need to do something else. In my case, I had to change the build scripts to fetch sources from somewhere else or use http instead of git.

Hope that helps,

--

Henrique Rodrigues

http://www.dcc.ufmg.br/~hsr

--
Henrique Rodrigues

http://www.dcc.ufmg.br/~hsr

[Abhi Basu]

Thanks folks, will give it a try.