What's wrong with this code?

[Dale Visser]

I always get exception after about 6-9M triples inserted by this code:

    private static final int CHUNK_SIZE = 300_000;

    private static void addResultsToRepository(final ResultSet results,
            final Connection stardog_connection)
            throws SQLException {
        System.out.println("Processing RDBMS " + CHUNK_SIZE +
                " rows at a time.");
        IRI context = Values.iri(CONTEXT);
        IRI fname = Values.iri(NAMESPACE, "fname");
        IRI lname = Values.iri(NAMESPACE, "lname");
        IRI dob = Values.iri(NAMESPACE, "dob");
        int count = 0;
        stardog_connection.begin();
        try {
            Adder adder = stardog_connection.add();
            while (results.next()) {
                IRI person = Values.iri(NAMESPACE,
                        "Person" + results.getString(1));
                addToTransaction(adder, person, fname, results, 2, context);
                addToTransaction(adder, person, lname, results, 3, context);
                addToTransaction(adder, person, dob, results, 4, context);
                count++;
                if (0 == count % CHUNK_SIZE) {
                    String timeStamp = new SimpleDateFormat(
                            "yyyyMMdd_HHmmss").format(
                                    Calendar.getInstance().getTime());
                    System.out.print(count + " rows so far. " + timeStamp +
                            "...");
                    stardog_connection.commit();
                    System.out.println("Committed.");
                    stardog_connection.begin();
                    adder = stardog_connection.add();
                    System.out.println("Starting next batch.");
                }
            }
        } finally {
            stardog_connection.commit();
        }
        System.out.println("# of results processed: " + count);
    }

    private static void addToTransaction(Adder adder, IRI person, IRI predicate,
            final ResultSet results, int column, IRI context) throws SQLException {
        adder.statement(person, predicate,
                Values.literal(results.getString(column)), context);
    }

Here's the exception:

Exception in thread "main" com.complexible.stardog.StardogException: Unable to complete operation, a transaction has not been initiated.

at com.complexible.stardog.api.impl.AbstractConnection.assertInTransaction(AbstractConnection.java:266)

at com.complexible.stardog.api.impl.AbstractConnection.commit(AbstractConnection.java:197)

at ida.org.stardogtimer.Sql2Stardog.addResultsToRepository(Sql2Stardog.java:128)

at ida.org.stardogtimer.Sql2Stardog.run(Sql2Stardog.java:81)

at ida.org.stardogtimer.Sql2Stardog.main(Sql2Stardog.java:142)

Am I doing anything wrong with my handling of the Stardog connection or transactions?

Best regards,
Dale Visser

[Zachary Whitley]

I'm on my phone so it's a little tough to read but I'd guess that you're probably throwing a SQLException in the try block after you've closed the outstanding transaction but before you've opened a new transaction. The finally block executes and tries to close a nonexistent transaction and it throws the StardogException that you're seeing. You should check that you actually have an outstanding transaction in the finally block before attempting to close it then you should see what the original exception causing the problem is. 

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+u...@clarkparsia.com.

[Dale Visser]

Thank you. After reading your assessment, I made sure to catch and log SqlException, and for good measure catch and log StardogException, too. I also realized that the sample code seems to recommend "one call per adder", i.e., always call connection.add().statement(...) instead of repeated calls to the same Adder instance like I was doing. (For all I know, it was giving the same reference...I didn't check.) Long story short, the real issue was my Stardog server runs out of heap space around 7.2M triples. Right now I'm running with CHUNK_SIZE reduced to 300k to test if this helps.

Does anyone know if there is a Stardog native store size to Java heap requirements correlation, i.e., as my store grows beyond 7M statements, do I actually need to give the Stardog server more heap space beyond the default 2 GB?

[Michael Grove]

On Mon, Jan 16, 2017 at 3:48 PM, Dale Visser <dale....@gmail.com> wrote:

Thank you. After reading your assessment, I made sure to catch and log SqlException, and for good measure catch and log StardogException, too. I also realized that the sample code seems to recommend "one call per adder", i.e., always call connection.add().statement(...) instead of repeated calls to the same Adder instance like I was doing. (For all I know, it was giving the same reference...I didn't check.) Long story short, the real issue was my Stardog server runs out of heap space around 7.2M triples. Right now I'm running with CHUNK_SIZE reduced to 300k to test if this helps.

Does anyone know if there is a Stardog native store size to Java heap requirements correlation, i.e., as my store grows beyond 7M statements, do I actually need to give the Stardog server more heap space beyond the default 2 GB?

Capacity planning information is in [1]. 2G of heap will handle significantly more than 7.2M triples in a database. Are you accidentally using a memory database?

Cheers,

Mike

[1] http://docs.stardog.com/#_capacity_planning

 

stardog+unsubscribe@clarkparsia.com

[Zachary Whitley]

Glad to hear you found the source of your problem. I'm not sure what the actual correlation is but you can find Stardog memory recommendations at http://docs.stardog.com/#_memory_usage

stardog+unsubscribe@clarkparsia.com

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.

[Evren Sirin]

As Mike said your memory settings should be enough for what you
describe. You should definitely reuse the same adder or your code
would GC unnecessarily to get rid of Adder instances.

Also did you look at the virtual graph support [1] and specifically
the virtual graph imports [2] which is what you are doing here? Your
code pulls the data from the RDBMS, creates the triples on the client
side and pushes them to the server. Virtual graph import on the other
hand queries the RDBMS on the server side and inserts the triples
directly which is more efficient and you don't need to write write any
code. You only need to define a mapping file which should be very
straight-forward in your case especially if you use the Stardog
Mapping Syntax [3].

Best,
Evren

[1] http://docs.stardog.com/#_structured_data_aka_virtual_graphs
[2] http://docs.stardog.com/#_materializing_virtual_graphs
[3] http://docs.stardog.com/#_stardog_mapping_syntax

[Dale Visser]

I see this when I try to use the virtual graph feature:

$ stardog-admin virtual add lots_of_users.properties lots_of_users.ttl
Unknown or unsupported command type 'Add', cannot execute.

Is it not supported in Community Edition?

Best regards,
Dale

[Zachary Whitley]

I believe that virtual graphs are an enterprise feature and is available with the 30-day evaluation or subscription licenses.

stardog+unsubscribe@clarkparsia.com

For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.

[Dale Visser]

Getting a 30-day trial license key solved it. :-)  In my case, I had a 300 second timeout when I tried to "virtual import", due to the size of my table. I decided in the end to manually map, and use "bulk loading" to accomplish what I'm trying to do, since I know I don't have to worry about transaction timeouts when creating a DB.