Uploading server dependencies to Maven

[Dylan Sanson]

Hi there,

I've had success uploading all of the necessary client libraries to a Nexus repo, such that I can now connect to Stardog instances hosted separately from our application.  I would like to use an embedded stardog instance for testing, however, and require some of the server libraries to be added to nexus to acomplish this.

I was wondering if anyone had any experience with this or is able to tell be which libraries exactly are needed?  I read in another answer that the contents of stardog-xxx/server/dbms were needed, however a handful of these jars don't have pom files.  Using gradle isn't really an option as the project I'm working on already depends heavily on maven.

Thank you,

Dylan

[Al Baker]

Hi Dylan,

You basically need all of the jars under the server folder tree.  Unfortunately, we do not have a script for uploading the server jars/dependencies to a private Nexus.  At some point in the future, we may publish POM’s for server side dependencies, but that’s not available today. 

Another option would be to use the maven exec plugin to execute the stardog start/stop commands from the CLI after the appropriate lifecycle phases in the Maven build.

Cheers,

Al

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[Dylan Sanson]

Hi Al,

Thank you for your reply! it's helped out quite a bit.  I've set up our nexus repo with the necessary libraries, and I'm attempting to take this a step further and integrate an embedded version with Spring.  Is it possible to connect a SnarlTemplate bean to an embedded instance of stardog?  My first attempts have yielded Connection Refused errors.  I suspect this may be because the embedded instance doesn't operate on a port?

Cheers,

Dylan

[Al Baker]

Hi Dylan,

Great to hear!  To use SnarlTemplate with the embedded database, you need to create a provider bean (your own implementation).   The reason for this is the embedded provider may have logic for your own application — in memory or disk, different options, whether you create/delete it on startup, etc.  Since there’s no one size fits all implementation for this, I opted to move this behind the Provider interface so you have full control over it.

Here is an example from the stardog-spring test suite:

https://github.com/clarkparsia/stardog-spring/blob/master/stardog-spring/src/test/java/com/complexible/stardog/ext/spring/EmbeddedProvider.java

With that bean in place, wire it in like this:

https://github.com/clarkparsia/stardog-spring/blob/master/stardog-spring/src/test/resources/test-applicationContext.xml

You’ll notice the DataSourceFactoryBean does not have any port/URL information in it, so you’ll want your bean definition to be similar.

Let me know if you hit any other issues.

Cheers,

Al

From: Dylan Sanson <dzr.s...@gmail.com>
Reply-To: <sta...@clarkparsia.com>
Date: Monday, January 12, 2015 at 11:26 AM
To: <sta...@clarkparsia.com>

--

[rick.m...@gmail.com]

Hi,

I'm currently trying to access the embedded mode sesame repository instances in Stardog, but I'm unclear on how to proceed.

Basically I'm trying to evaluate stardog as a backend to an existing program written against the Sesame API's in clojure.

In the Clojure community the build tool of choice is leiningen, which can access maven repository artifacts.  Unfortunately leiningen support for accessing free-standing jars and setting the classpath yourself is significantly more awkward because leiningen is very opinionated on the matter, and the recommended approach is to publish the artificats to a private maven repository. 

Sadly the sheer number of component jars for the server side stardog appears to make this incredibly awkward.

I tried to modify the client script to mavenise the server jars also, but I'm not sure which jars contain the artifacts I need, nor am I sure if all the jars contain the appropriate pom.xml's.

Basically I want to be able to use instances of the local repository/sail objects, but don't know where to find them or how to mavenise them.  Any help appreciated.

R.

[Al Baker]

Hi Rick,

For Stardog 2.2.x embedded server mode, you'll need all of the jars in the server folder.  Everything is in there, and while there are numerous jars, you can rest assured if you do add them all, the embedded server will work.  For a leiningen based project, this does require you to run a "mvn install" for all of those jars [1], and then add them all to your project.clj file.

Before going through that work to add all servers jars manually, you may want to try client/server to get a feel of Stardog and its features.  All of the client jars are available in your ~/.m2/repository after running the mavenInstall script you'll find in the bin folder [2]. With the client jars available, you can use them directly as normal Java classes, or use one of our language specific APIs, such as stardog-clj [3].  From an application development perspective, the difference between a client/server application and an embedded application is merely a change in the connection configuration, and optionally using some of the server side database management APIs.

We are looking at publishing POMs and enhancing our Maven support in an upcoming release.  If there are things you're looking for beyond just deploying the files in a local m2 folder, we'd love ot hear more about what type of use cases you're looking at.

Cheers,
Al

[1]: https://maven.apache.org/guides/mini/guide-3rd-party-jars-local.html
[2]: http://docs.stardog.com/#_support_for_maven
[3]: https://github.com/clarkparsia/stardog-clj

[Rick Moynihan]

Hi Al,

Thanks for the quick reply.

I could certainly try and access stardog through the client API's but
my fear is that the support for transactions and loading large amounts
of data won't be as good as in the in process case.

With sesame for example if you add a large amount of triples to a
SparqlRepository (sesame's client repository) it will simply create
one huge insert query. With a large dataset this can wreak havoc, as
what you want is for the results to be streamed into the store
efficiently... not batched into a query string twice (once in the
client and again on the server) potentially aggravating the GC and
eventually blowing up with an OutOfMemoryException.

I'm curious how stardog handles cases such as these in the client
mode... What are the big differences and trade offs between
client/embedded server? For example is it possible to use stardogs
client to create multi-request transactions; as exposed on your HTTP
API?

If so this might be viable.

Thanks for suggesting stardog-clj, the bindings look really good;
except our code is already using Sesame's API's so in this instance it
might not be useful for us.

Having documented maven packages and poms for the server would
certainly make this process easier, as would knowing what maven
packages include specific java packages and classes.

Thanks again,

R.

[Mike Grove]

On Fri, Jan 30, 2015 at 7:11 PM, Rick Moynihan <rick.m...@gmail.com> wrote:

Hi Al,

Thanks for the quick reply.

I could certainly try and access stardog through the client API's but
my fear is that the support for transactions and loading large amounts
of data won't be as good as in the in process case.

What's the process case?

 

With sesame for example if you add a large amount of triples to a
SparqlRepository (sesame's client repository) it will simply create
one huge insert query.  With a large dataset this can wreak havoc, as
what you want is for the results to be streamed into the store
efficiently... not batched into a query string twice (once in the
client and again on the server) potentially aggravating the GC and
eventually blowing up with an OutOfMemoryException.

Changes to Stardog are batched on the client, they're only sent to the server when you commit or you perform a read within the transaction.  They're also lazily processed; if you're adding data from a File, or stream, we don't read it until we're sending it to the server to minimize the memory footprint.

The payload(s) are streamed to the server over HTTP or the SNARL protocol depending on what the connection is using under the hood; it's not done via SPARQL update.

When they're received on the server they're processed in chunks as they come in and are flushed to disk as needed again to minimize the memory usage.

 

I'm curious how stardog handles cases such as these in the client
mode... What are the big differences and trade offs between
client/embedded server? 

The primary tradeoff would be the server is running in the process with your application, so it's sharing in consuming resources.  But with the embedded server you are not paying any overhead for communicating with the server.

The embedded server is really great for prototyping and for your test suite, but in most production deployments you'd probably want to use a client/server setup.

As Al mentioned, there's zero code change other than the connection configuration for switching between these two modes (as far as the client is concerned).

 

For example is it possible to use stardogs
client to create multi-request transactions; as exposed on your HTTP
API?

Yes, this is precisely how our HTTP client communicates with the server.

 

If so this might be viable.

Thanks for suggesting stardog-clj, the bindings look really good;
except our code is already using Sesame's API's so in this instance it
might not be useful for us.

Having documented maven packages and poms for the server would
certainly make this process easier, as would knowing what maven
packages include specific java packages and classes.

We'll improve that situation for the 3.0 release.

Cheers,

MIke