Upgrade sbt to the latest version 1.4.9?

55 views
Skip to first unread message

Jacek Laskowski

unread,
Mar 30, 2021, 9:54:28 AM3/30/21
to Delta Lake Users and Developers
Hi,

I've noticed delta uses sbt 0.13.18 which (put it mildly) is very old and unsupported. Spark uses 1.4.9 [1].

This is to find out whether or not someone is working on this already? If not, I'd like to step in.

Should I file a task on https://github.com/delta-io/delta/issues ?



Tathagata Das

unread,
Mar 30, 2021, 10:14:30 AM3/30/21
to Jacek Laskowski, Delta Lake Users and Developers
I think back when we were building the project, Spark was using the same version. I had tested sbt 1.0+ but I vaguely remember (almost 2 years ago) that some plugin had compatibility issues. It definitely makes sense to try it again.

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CAB_G8ZuoL5CW33oRAm7x%3D8TrEcZv81dcpG%2B46tyO-ndX_m1XAw%40mail.gmail.com.

Jacek Laskowski

unread,
Mar 31, 2021, 8:13:43 AM3/31/21
to Tathagata Das, Delta Lake Users and Developers
Hi,

Thanks TD. I tried 1.4.9 yesterday and failed. Trying out again.

Jacek Laskowski

unread,
Apr 5, 2021, 5:54:59 PM4/5/21
to Tathagata Das, Delta Lake Users and Developers
Hi,

I've been working on upgrading sbt to 1.5.0 (released just today) and all seem to be working fine now except this one little thing with unidoc that errors out yet all sbt tasks seem to be OK. I can't explain why unidoc gives errors yet tasks report success.


I DMed Eugene Yokota hoping to engage him a bit as I've got no idea how to fix this.

Can you have a look at these errors and help if possible? Thank you.

Tathagata Das

unread,
Apr 5, 2021, 6:05:11 PM4/5/21
to Jacek Laskowski, Delta Lake Users and Developers
I took a quick look. It's possible that our class structure is throwing off the unidoc builder because we trying to hide a number of classes and packages from build documented.
I don't know exactly how to fix it, but I would check out those package exclusions in the SBT file - https://github.com/delta-io/delta/blob/331c2af50dc3a2746f72f58cb896a5459caebff1/build.sbt#L144-L150 
I can say that anything in `org.apache.spark.` should be hidden.

TD

Jacek Laskowski

unread,
Apr 5, 2021, 6:28:48 PM4/5/21
to Tathagata Das, Delta Lake Users and Developers
Thanks TD! Much appreciated to have found your email!

show Javaunidoc / unidoc / unidocAllSources gives the following:

Vector(/Users/jacek/dev/oss/delta/target/java/io/delta/tables/DeltaMergeBuilder.java, /Users/jacek/dev/oss/delta/target/java/io/delta/tables/DeltaMergeMatchedActionBuilder.java, /Users/jacek/dev/oss/delta/target/java/io/delta/tables/DeltaMergeNotMatchedActionBuilder.java, /Users/jacek/dev/oss/delta/target/java/io/delta/tables/DeltaTable.java)

which seems to be exactly what we want but since they reference hidden packages the build fails :(

Tathagata Das

unread,
Apr 5, 2021, 6:37:36 PM4/5/21
to Jacek Laskowski, Delta Lake Users and Developers
Thank you so much for taking the time to make this work. I think there has to be some way to make the unidoc plugin ignore references to hidden packages. This is unfortunate, and needs more digging :( I remember that I had similar issues in the past when I had tried (almost 2 years ago) and I had stopped investing time as we had other higher priority stuff. But now we so many improvements in sbt since then, it is definitely work investing that time. Let me know if you can uncover anything with more digging.

TD

Mushtaq Ahmed

unread,
Apr 6, 2021, 12:32:23 AM4/6/21
to Jacek Laskowski, Tathagata Das, Delta Lake Users and Developers
I notice that genjavadoc has a newer version (0.17) and delta build uses 0.16. Worth trying to upgrade and see? 


To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CAB_G8Zu3RAm33XbgcgKY5f-eCjpWoH0y7uPEYFU0eJHkCEgf4w%40mail.gmail.com.

Jacek Laskowski

unread,
Apr 6, 2021, 8:25:41 AM4/6/21
to Mushtaq Ahmed, Tathagata Das, Delta Lake Users and Developers
Hi,

Thanks Mushtaq! I think I've already tested all the versions - the default 0.15, 0.16 and the latest 0.17. Neither worked :(

https://github.com/delta-io/delta/pull/642/commits/067eaa15feeca5f43a2c5c7cc738cced1432977c

No idea how to make any progress except using inspect task in sbt and possibly reviewing the sources of sbt-unidoc. That's gonna take weeks. I think my next change is to clean up a bit and unwire mima and stylecheck from unidoc.

Any help much appreciated as I've got no hope with this atm!
Reply all
Reply to author
Forward
0 new messages