DBI

18 views
Skip to first unread message

Per Nyfelt

unread,
Aug 13, 2019, 4:20:14 AM8/13/19
to Renjin
Hi,

I found an issue with org.renjin.cran:DBI ver 1.0.0-b9 when using it with 3.5-beta64 (i created an issue for this here: https://github.com/bedatadriven/renjin/issues/474). I thought that I would try to fix this and create a pull request but I am confused about where the code actually is. Looking in https://github.com/bedatadriven/DBI it does not contain the 1.0.0-b9 code but some older version. Does anyone know where the latest code for DBI is?

Best regards,
Per

Bertram, Alexander

unread,
Aug 15, 2019, 8:41:13 AM8/15/19
to renji...@googlegroups.com
Hi Per,
The DBI package has been built directly from CRAN sources for sometime now:

It doesn't contain any native code, so I don't think rebuilding it against the latest beta will be neccessary, but let me know if you run into any issues.

Best,
Alex

--
You received this message because you are subscribed to the Google Groups "Renjin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to renjin-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/renjin-dev/efb42806-f907-4506-9bc7-9f5ce68ecb5d%40googlegroups.com.


--
Alexander Bertram
Technical Director
BeDataDriven BV

Web: http://bedatadriven.com
Email: al...@bedatadriven.com
Tel. Nederlands: +31(0)647205388
Skype: akbertram

Per Nyfelt

unread,
Aug 15, 2019, 9:22:34 AM8/15/19
to Renjin
Hi Alex,


You see things such as
<dependency>
    <groupId>org.renjin</groupId>
    <artifactId>grDevices</artifactId>
    <version>0.9.2709</version>
</dependency>

This means that if one is unlucky, the DBI dependencies become available on the classpath before the 3.5beta dependencies things and will fail miserably (complaining about missing class org/renjin/sexp/Environment$EmptyEnv).

This could easily be fixed by declaring the renjin dependencies in DBI to be scoped as provided. But that is not the whole story. I played with explicitly excluding all renjin dependencies and that works most of the time but not all of the time (e.g. when using DBI in a maven plugin) so I'd like to look at the sources and play around with dependencies etc. to see if I can figure out what the issue is.

So somewhere there must be some pom file defined for the cran build that makes it "renjin compatible" and available as a maven resource. But the pom in https://github.com/bedatadriven/DBI master branch is not this (it refers to renjin version 0.8.2240). I submitted an issue about this here https://github.com/bedatadriven/renjin/issues/474 but I thought it woule be nice to fix it myself and submit a PR, just dont know where the source is. When you say "it has been built from CRAN sources for sometime now", how/where is the pom file coming from? Sorry if I am not making sense, since I do not understand how the process to take something from cran.r-project.org and publish it in https://nexus.bedatadriven.com/content/groups/public/ under org.renjin.cran works I might be missing something obvious.

Regards,
Per

On Thursday, August 15, 2019 at 2:41:13 PM UTC+2, Alexander Bertram wrote:
Hi Per,
The DBI package has been built directly from CRAN sources for sometime now:

It doesn't contain any native code, so I don't think rebuilding it against the latest beta will be neccessary, but let me know if you run into any issues.

Best,
Alex

On Tue, Aug 13, 2019 at 10:20 AM Per Nyfelt <per....@gmail.com> wrote:
Hi,

I found an issue with org.renjin.cran:DBI ver 1.0.0-b9 when using it with 3.5-beta64 (i created an issue for this here: https://github.com/bedatadriven/renjin/issues/474). I thought that I would try to fix this and create a pull request but I am confused about where the code actually is. Looking in https://github.com/bedatadriven/DBI it does not contain the 1.0.0-b9 code but some older version. Does anyone know where the latest code for DBI is?

Best regards,
Per

--
You received this message because you are subscribed to the Google Groups "Renjin" group.
To unsubscribe from this group and stop receiving emails from it, send an email to renji...@googlegroups.com.

Bertram, Alexander

unread,
Aug 15, 2019, 12:34:07 PM8/15/19
to renji...@googlegroups.com
Hi Per,
As long as a newer (e.g. 3.5-beta-xx) version of grDevices is included in your project, Maven (and other JVM build tools) should choose the more recent version, regardless of which one is included first. If you're using Maven, you can see which version is resolved by running 

mvn dependency:list

or more details from:

mvn dependency:tree

Be sure to include org.renjin:renjin-script-engine:3.5-beta-xx in your dependencies, which includes all of the "default" packages. 

For all "normal" CRAN and BioConductor packages, our build system directly uses the sources published to CRAN/BioConductor and generates a pom.xml file automatically from the DESCRIPTION file on the fly. 

There are only a handful of "replaced" packages like stringi, or RPostgreSQL, which generally substitute an equivalent Java library for a native dependency. These packages do include their own pom.xml file and generally some extra Java code to replace the C code. You can identify these packages on packages.renjin.org because there is a banner that says

"A Renjin-specific version of this package is available"


But back to your main point, we could reconfigure the CI system to set the dependency scope of default packages like grDevices to "provided", but I'm still not sure that's neccessary if you include a new version of renjin-script-engine as a dependency.   

Could you share a public/private gist with the pom file of the project in question (or the output of dependency:tree) ?

Best,
Alex



To unsubscribe from this group and stop receiving emails from it, send an email to renjin-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/renjin-dev/a94e5944-4d74-4d96-92fa-cc757f84b4d5%40googlegroups.com.

Per Nyfelt

unread,
Aug 15, 2019, 3:14:22 PM8/15/19
to Renjin
Thanks Alex,

I agree that the the renjin version used should be evicted but in some cases it is not. 

I just created a rep in github which shows the issue: https://github.com/perNyfelt/renjinSamplesAndTests/tree/master/junit

Look at the SimpleRenjinWithDBITest for two tests that displays the issue (one of them is failing unless the renjin dependencies in DBI are excluded in the pom)

Below is the output of
mvn dependency:tree

[INFO] --- maven-dependency-plugin:2.8:tree (default-cli) @ samples-junit ---
[INFO] se.alipsa:samples-junit:jar:1.0-SNAPSHOT
[INFO] +- com.microsoft.sqlserver:mssql-jdbc:jar:7.4.1.jre8:test
[INFO] +- org.testcontainers:testcontainers:jar:1.12.0:test
[INFO] |  +- org.jetbrains:annotations:jar:17.0.0:test
[INFO] |  +- javax.annotation:javax.annotation-api:jar:1.3.2:test
[INFO] |  +- org.apache.commons:commons-compress:jar:1.18:compile
[INFO] |  +- javax.xml.bind:jaxb-api:jar:2.3.1:test
[INFO] |  |  \- javax.activation:javax.activation-api:jar:1.2.0:test
[INFO] |  +- org.rnorth.duct-tape:duct-tape:jar:1.0.8:test
[INFO] |  +- org.rnorth.visible-assertions:visible-assertions:jar:2.1.2:test
[INFO] |  |  \- net.java.dev.jna:jna:jar:5.2.0:test
[INFO] |  +- org.rnorth:tcp-unix-socket-proxy:jar:1.0.2:test
[INFO] |  |  +- com.kohlschutter.junixsocket:junixsocket-native-common:jar:2.0.4:test
[INFO] |  |  |  \- org.scijava:native-lib-loader:jar:2.0.2:test
[INFO] |  |  \- com.kohlschutter.junixsocket:junixsocket-common:jar:2.0.4:test
[INFO] |  \- net.java.dev.jna:jna-platform:jar:5.3.1:test
[INFO] +- org.testcontainers:mssqlserver:jar:1.12.0:test
[INFO] |  \- org.testcontainers:jdbc:jar:1.12.0:test
[INFO] |     \- org.testcontainers:database-commons:jar:1.12.0:test
[INFO] +- org.renjin.cran:DBI:jar:1.0.0-b9:compile
[INFO] |  +- org.renjin:tcltk:jar:0.9.2709:compile
[INFO] |  +- org.renjin:methods:jar:0.9.2709:compile
[INFO] |  +- org.renjin:graphics:jar:0.9.2709:compile
[INFO] |  +- org.renjin:datasets:jar:0.9.2709:compile
[INFO] |  +- org.renjin:splines:jar:0.9.2709:compile
[INFO] |  +- org.renjin:grDevices:jar:0.9.2709:compile
[INFO] |  |  +- org.renjin:renjin-gnur-runtime:jar:0.9.2709:compile
[INFO] |  |  |  \- org.renjin:gcc-runtime:jar:0.9.2709:compile
[INFO] |  |  \- org.jfree:jfreesvg:jar:3.3:compile
[INFO] |  +- org.renjin:tools:jar:0.9.2709:compile
[INFO] |  +- org.renjin:stats:jar:0.9.2709:compile
[INFO] |  |  +- org.renjin:renjin-nmath:jar:0.9.2709:compile
[INFO] |  |  |  \- org.renjin:renjin-math-common:jar:0.9.2709:compile
[INFO] |  |  \- org.renjin:renjin-appl:jar:0.9.2709:compile
[INFO] |  +- org.renjin:parallel:jar:0.9.2709:compile
[INFO] |  +- org.renjin:utils:jar:0.9.2709:compile
[INFO] |  +- org.renjin:grid:jar:0.9.2709:compile
[INFO] |  \- org.renjin:stats4:jar:0.9.2709:compile
[INFO] +- org.renjin:renjin-script-engine:jar:3.5-beta64:compile
[INFO] |  +- org.slf4j:slf4j-jdk14:jar:1.6.1:compile
[INFO] |  \- org.renjin:renjin-core:jar:3.5-beta64:compile
[INFO] |     +- org.renjin:renjin-blas:jar:3.5-beta64:compile
[INFO] |     |  \- org.renjin:renjin-common:jar:3.5-beta64:compile
[INFO] |     +- org.renjin:renjin-lapack:jar:3.5-beta64:compile
[INFO] |     +- org.renjin:gcc-bridge-runtime:jar:3.5-beta64:compile
[INFO] |     +- org.apache.commons:commons-math:jar:2.2:compile
[INFO] |     +- com.github.fommil.netlib:core:jar:1.1.2:compile
[INFO] |     |  \- net.sourceforge.f2j:arpack_combined_all:jar:0.1:compile
[INFO] |     +- org.apache.commons:commons-vfs2:jar:2.0:compile
[INFO] |     |  +- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] |     |  +- org.apache.maven.scm:maven-scm-api:jar:1.4:compile
[INFO] |     |  |  \- org.codehaus.plexus:plexus-utils:jar:1.5.6:compile
[INFO] |     |  \- org.apache.maven.scm:maven-scm-provider-svnexe:jar:1.4:compile
[INFO] |     |     +- org.apache.maven.scm:maven-scm-provider-svn-commons:jar:1.4:compile
[INFO] |     |     \- regexp:regexp:jar:1.3:compile
[INFO] |     +- org.tukaani:xz:jar:1.8:compile
[INFO] |     +- org.renjin:renjin-asm:jar:5.0.4b:compile
[INFO] |     +- org.renjin:renjin-guava:jar:28.0b:compile
[INFO] |     \- com.sun.codemodel:codemodel:jar:2.6:compile
[INFO] +- org.slf4j:slf4j-api:jar:1.7.27:compile
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.27:compile
[INFO] |  \- log4j:log4j:jar:1.2.17:compile
[INFO] +- junit:junit:jar:4.12:test
[INFO] |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] \- org.renjin:hamcrest:jar:3.5-beta64:test

If i explicitly exclude the renijin denendencies in DBI it all works. If i do not then mvn clean test results in 

org.renjin.eval.EvalException: Exception initializing compiled GNU R library class org.renjin.grDevices.grDevices
....
Caused by: java.lang.NoClassDefFoundError: org/renjin/sexp/Environment$EmptyEnv

Best regards,
Per

On Thursday, August 15, 2019 at 6:34:07 PM UTC+2, Alexander Bertram wrote:
Hi Per,

Bertram, Alexander

unread,
Aug 16, 2019, 8:00:30 AM8/16/19
to renji...@googlegroups.com
Hi Per,

It looks like I actually misunderstood how Maven resolves transitive dependency versions. I found this in the manual:


Dependency mediation - this determines what version of an artifact will be chosen when multiple versions are encountered as dependencies. Maven picks the "nearest definition". That is, it uses the version of the closest dependency to your project in the tree of dependencies. Note that if two dependency versions are at the same depth in the dependency tree, the first declaration wins.
  • "nearest definition" means that the version used will be the closest one to your project in the tree of dependencies. For example, if dependencies for A, B, and C are defined as A -> B -> C -> D 2.0 and A -> E -> D 1.0, then D 1.0 will be used when building A because the path from A to D through E is shorter. You could explicitly add a dependency to D 2.0 in A to force the use of D 2.0.

This explains the behavior in your sample project. If you add renjin-script-engine:3.5-beta64 to the dependency list in junit/pom.xml, then grDevices resolves to version 3.5-beta64. But if this is omitted, then the grDevices dependency included by DBI is considered to be "nearer" (DBI:1.0.0 -> grDevices:0.9.2709) than that included via parent -> renjin-script-engine:3.5-beta64 -> grDevices:3.5-beta64

This seems like a terrible idea to me, but it's apparently how Maven is designed to work.

Given this behavior, it does seem logical to set the scope of modules part of the Renjin release to "provided" rather than "compile".

I am unfortunately off on leave shortly, but when I am back in September I will update the build system.

In the meantime, adding renjin-script-engine as a direct dependency should solve the problem in most cases.

Best,
Alex

To unsubscribe from this group and stop receiving emails from it, send an email to renjin-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/renjin-dev/225299e1-c8e7-4cf1-95a1-26316b0d8ecd%40googlegroups.com.

Per Nyfelt

unread,
Aug 16, 2019, 8:32:07 AM8/16/19
to Renjin

Very good, thanks for figuring this out Alex! I could not imagine that a parent dependency would be considered further away than a transient one. Very strange indeed.

Looking forward to try the fix when you are back, happy holidays!

Best regards,
Per


On Friday, August 16, 2019 at 2:00:30 PM UTC+2, Alexander Bertram wrote:
Hi Per,

It looks like I actually misunderstood how Maven resolves transitive dependency versions. I found this in the manual:


Dependency mediation - this determines what version of an artifact will be chosen when multiple versions are encountered as dependencies. Maven picks the "nearest definition". That is, it uses the version of the closest dependency to your project in the tree of dependencies. Note that if two dependency versions are at the same depth in the dependency tree, the first declaration wins.
  • "nearest definition" means that the version used will be the closest one to your project in the tree of dependencies. For example, if dependencies for A, B, and C are defined as A -> B -> C -> D 2.0 and A -> E -> D 1.0, then D 1.0 will be used when building A because the path from A to D through E is shorter. You could explicitly add a dependency to D 2.0 in A to force the use of D 2.0.

This explains the behavior in your sample project. If you add renjin-script-engine:3.5-beta64 to the dependency list in junit/pom.xml, then grDevices resolves to version 3.5-beta64. But if this is omitted, then the grDevices dependency included by DBI is considered to be "nearer" (DBI:1.0.0 -> grDevices:0.9.2709) than that included via parent -> renjin-script-engine:3.5-beta64 -> grDevices:3.5-beta64

This seems like a terrible idea to me, but it's apparently how Maven is designed to work.

Given this behavior, it does seem logical to set the scope of modules part of the Renjin release to "provided" rather than "compile".

I am unfortunately off on leave shortly, but when I am back in September I will update the build system.

In the meantime, adding renjin-script-engine as a direct dependency should solve the problem in most cases.

Best,
Alex

Reply all
Reply to author
Forward
0 new messages