How to build the project

[Erik Eklund]

Hello!
I am interested in trying to develop something but I cannot figure out how to build the project. And Intellij cannot figure out the sbt project structure either...
As per the README.md I do build/sbt assembly but I get this error:

[error] (*:update) sbt.ResolveException: unresolved dependency: org.eclipse.sisu#org.eclipse.sisu.plexus;0.3.0: org.sonatype.oss#oss-parent;9!oss-parent.pom(pom.original) origin location must be absolute: file:/Users/erikeklund/.m2/repository/org/sonatype/oss/oss-parent/9/oss-parent-9.pom

SPARK_HOME is set
code/graphframes - (master) > echo $SPARK_HOME
/Users/erikeklund/.sdkman/candidates/spark/current

Thanks for any help!
Erik

[Russell Jurney]

Hi Erik, the setup is a little finicky. 

Have you run "build/sbt assemble" from bash rather than IntelliJ? I have found that is the best way to develop. I don't know how to debug IntelliJ or Eclipse (which I use) but I can help with the CLI.

Thanks,

Russell

[Russell Jurney]

Oh, can you tell me what your $JAVA_HOME and $HADOOP_HOME variables are? Java in particular, I wonder if you are using your IDE's JVM, which can cause issues.

Russ

--
You received this message because you are subscribed to the Google Groups "GraphFrames" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graphframes...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/graphframes/2fccc7e9-90b9-488f-aad0-3ee6ee27d65fn%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ángel]

How strange! I'm using IntelliJ and haven't had problems so far running and debugging the unit tests.

I'm using Hadoop-3.4.0 and JDK corretto-11 (downloaded by IntelliJ).

I'm using Windows, so I also have to set the following java.library.path JVM property to the bin folder of the HADOOP_HOME.

To view this discussion visit https://groups.google.com/d/msgid/graphframes/CANSvDjpM%3DVsOvftZf8M6DjNfOP_DXo6-zEQzmJcY97DPsUekKQ%40mail.gmail.com.

[Erik Eklund]

Russell:

code/graphframes - (master) > sdk current java
Using java version 11.0.23-zulu
code/graphframes - (master) > sdk current hadoop
Using hadoop version 3.3.5

I have the problem on my Mac machine but I can get the project to build from CLI through WSL on my PC. So I have started to develop there :) However no intellij integration with WSL so I am using VSCode :(

Àngel:
Thanks, will try that if I continue have problems with WSL

[Russell Jurney]

I have tried falling back to Java 8 when I had build issues under 11 or 17. Obviously you shouldn't have to do that and we need to make the build more robust if people are struggling to build the project.

Ouch, developing Java / Scala in VSCode is rough compared to the "write the code for you" that JVM IDEs have long achieved...

Russell

[Ángel]

If some of you keep on having problems developing/debugging with the IDE maybe we could have a Zoom session or something like that. My speaking English is not as proficient as I'd like to be, but I guess it's enough to explain how I'm doing it, answer some doubts and fix some issues. What do you think?

PS: My experience with VS is mostly from ... 20 years ago? 😅🤣

https://winworldpc.com/product/microsoft-visual-stu/60

To view this discussion visit https://groups.google.com/d/msgid/graphframes/5d1a19f5-d15c-446e-8143-0325e73020e1n%40googlegroups.com.

[Russell Jurney]

Let's do that! In fact, let's use this opportunity and the renewed interest to decide on a date for the planned Hackathon! Angel, is there a Friday or Saturday where you can lead us in building GraphFrames? This will get everyone started on contributing efficiently and we can carry off our tickets and work together on the Zoom all day.

What do you think?

Russell

[Ángel]

Next Friday, February 7th, works for me. Does that work for everyone interested?

Just a heads-up: I don't consider myself an expert in anything, but I'm happy to offer any support I can!

To view this discussion visit https://groups.google.com/d/msgid/graphframes/baeb0c96-be39-4c49-a226-3b36f7180b03n%40googlegroups.com.

[Russell Jurney]

We need about three weeks of lead time to ensure attendance, are you free three Fridays out, on Feb 21? We can hold it from 8AM EST US, which will be 5AM PST for me… you could host the Zoom earlier than that for Europeans.

Russell

[Russell Jurney]

Angel, I’m not so hot at Scala but I’m learning. I have a lot of experience as a user of Hadoop and Spark but I’m not an expert either. We are inheritors of code, doing the best we can :)

Thanks,

Russell Jurney | rju...@graphlet.ai | graphlet.ai | Graphlet AI Blog | LinkedIn | BlueSky

On Sat, Feb 1, 2025 at 9:18 AM Ángel <angel.alva...@gmail.com> wrote:

To view this discussion visit https://groups.google.com/d/msgid/graphframes/CAGUyL8hx5Ek7W67rCJF0aWFEN%3DB7dyXDTe0SzX1oLdb0wn5f4g%40mail.gmail.com.

[Ángel]

Friday the 21st at that time works for me, but wouldn't it be too early for most of you? Are you sure about the AM part?

[Russell Jurney]

There is no way to accommodate everyone - and I am willing to have crazy hours that day. There are a number of Americans and Europeans that have expressed interest in a hackathon. I think we should have a staged start and let people join when their day starts or they can get away from work and just support them in getting started contributing. If you could be on the call on CET time to guide new contributors, I can help with that once it gets late for you. I will probably join at  8AM CET (11PM PST) and just work through the night, then get people started the next morning :)

Russ

[Russell Jurney]

Copying from the ticket, let's do the discussion here but I will update it for subscribers...

---------------

Yes, I think we should pin "get help with setting up environments" for a certain time in CET and US time, maybe 10AM CET (1AM PST) and 11AM EST (5PM CET)? If I see you run through some fixes I can do that if you're off by then.

So we would announce:

Hackathon!

* Get help setting up your environment with a presentation and help session held at one of two times!

    * 10AM CET - Angel Pascua will assist with project builds at the CLI and with IntelliJ

    * 11AM EST / 8AM PST - Russell Jurney will assist with project builds at the CLI and with IntelliJ

Come out and meet and get support... blah blah

What do you think?

Russell

[Ángel]

I know you're willing to put in crazy hours that day—so am I! But, my dear friend, not everyone shares our passion...

I'm an early bird and usually wake up at 5 AM (or even earlier when I'm excited about something). I believe we can make it work!

[Russell Jurney]

I think that's okay, and thanks for the devotion :) The build camps are scheduled conveniently for people in Europe and North/South America, choose either one. I will learn to build in IntelliJ from you (I need support here too) and use that knowledge in the US build camp.

People can join whenever they want and get support and have fun. I don't expect Americans to join before the scheduled build camp but they may. I don't expect people to all stay until the end but some may. I think people will drop in as they are able and that's perfectly okay.

Russell