Problems with sbt-spark-package

[Sem]

Hello!


I'm working on adding a Spark-Connect support (#447) and I need to add
an implementation of org.apache.sql.connect.plugin.RelationPlugin to
achieve that. To avoid dependency mess I'm trying to implement it as a
subproject but I'm constantly facing problems with a very old version of
sbt and with a spark-packages resolver. Is the sbt-spark-package project
alive? I see that the last commit was 8 years ago:
https://github.com/databricks/sbt-spark-package And I cannot update sbt
version because this plugin failed with a newer versions.


What do you think about:

- replacing sbt-spark-package with an explicit definition of dependencies

- updating sbt version (from 0.13.x to 1.9.x)

- rewriting build sbt using explicit definition of projects (lazy val
graphframes = (project in file(".))....)


I'm willing to open an issue, work on the implementation and open a PR
with changes.


Best regards,

Sem

[Russell Jurney]

I cc'd a couple of others that are more technical than I am to see what they think. I'm a GraphFrames user and volunteer community manager and not yet a GraphFrames expert. Soon! I'm actually thinking about a Spark / GraphFrames / FPGA solution for network motif mining... anyway:

1. Updating sbt version sounds awesome, it is an old version so this sounds like a great idea to me

2. I also noted how old the sbt-spark-package is. I would suggest making whatever changes you need to get your focused contribution landed, without trying to fix all issues of all things. I say update sbt-spark-package with an explicit definition of dependencies.

3. I've also had to make changes to our sbt build. I'm not 100% on what you're proposing, but I found it hard to get things done without monkeying with sbt. So go for it.

For some reason not all messages to this group land in my inbox... I'll try to respond same day by checking the group regularly.

Thanks,

Russell

--
You received this message because you are subscribed to the Google Groups "GraphFrames" group.
To unsubscribe from this group and stop receiving emails from it, send an email to graphframes...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/graphframes/2ac48277-dadb-4a5a-9fb7-c38277dd1dc5%40apache.org.
For more options, visit https://groups.google.com/d/optout.