Many Transactions - Server is not responding

[michael...@gmail.com]

Hello there,

I became a really big fan of stardog right now!

Our team is now trying to insert many data in short time.

Unfortunately it's not working, because the server seems to be too overloaded with so many requests. Or maybe we made something wrong ;-)

Here is our log: 

console.log(body) --> Transaction ID

console.log(response.statusCode)

5358d167-a3e5-4e48-9c70-47c9ca4d5b71
500
a676d57a-b436-491b-8a1c-a5b9773da4ae
500
2efd9935-bf2f-4c48-82d4-46987516e442
f3ec7ea8-175f-4d7d-b57e-5d5f85396788
05993b30-a772-4788-a060-bd4b993338fa
200
500
500

Is there some special workaround when making many inserts in a short time period?

Best Regards,

Michael

[Mike Grove]

It's useful to know what you consider "many" and a "short" time period.  

Also, without knowing what the error is that caused the 500 response it's difficult to make any suggestions.

Cheers,

Mike

 

Best Regards,

Michael

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+u...@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en

[michael...@gmail.com]

Hello,

here is a new Log with the error codes

Line Description:

1) Transaction ID

2) Status Code

  0c1c6da6-9ef4-4cc1-907f-5c3c6aeb843f

500

84894fe8-726c-4f15-81ae-be5dcea0ddf8

500

Error found!

{"code":"ECONNRESET"}

{ [Error: socket hang up] code: 'ECONNRESET' }

998079b3-6302-44c6-ac3e-13fb1a13e1f9

Error found!

{"code":"ECONNRESET"}

{ [Error: socket hang up] code: 'ECONNRESET' }

66a2b133-21ab-46df-8120-86942b430378

Error found!

{"code":"ECONNRESET"}

c3496feb-80e7-49f7-85b1-ba1a741528c0

Error found!

{"code":"ECONNRESET"}

9eea07bc-194c-48c0-ba4c-d547b420c3b0

Error found!

{"code":"ECONNRESET"}

{ [Error: socket hang up] code: 'ECONNRESET' }

df1ec183-c59e-46c4-9e00-4b06980c5da6

Error found!

{"code":"ECONNRESET"}

Error found!

{"code":"ECONNRESET"}

Error found!

{"code":"ECONNRESET"}

177b1d6f-9d64-4876-88fd-c6c59cb4d350

3fe89316-ae5b-462c-a6ba-1bb980878876

200

Error found!

{"code":"ECONNRESET"}

It's happening when trying to insert 10 triples in about 0.5 seconds.

[Mike Grove]

It's hard to say what might be causing the connection reset response without seeing actual log content from the server.

Cheers,

Mike

 

It's happening when trying to insert 10 triples in about 0.5 seconds.

Am Freitag, 19. September 2014 19:17:22 UTC+2 schrieb michael...@gmail.com:

Hello there,

I became a really big fan of stardog right now!

Our team is now trying to insert many data in short time.

Unfortunately it's not working, because the server seems to be too overloaded with so many requests. Or maybe we made something wrong ;-)

Here is our log: 

console.log(body) --> Transaction ID

console.log(response.statusCode)

5358d167-a3e5-4e48-9c70-47c9ca4d5b71
500
a676d57a-b436-491b-8a1c-a5b9773da4ae
500
2efd9935-bf2f-4c48-82d4-46987516e442
f3ec7ea8-175f-4d7d-b57e-5d5f85396788
05993b30-a772-4788-a060-bd4b993338fa
200
500
500

Is there some special workaround when making many inserts in a short time period?

Best Regards,

Michael

--

[michael...@gmail.com]

I think there is a bug. After the errors the database is not working anymore and the backend shows me Name: "undefined" instead of the database.

This error comes when trying to make new requests:

Database 'semantic' does not exist.

400

Am Freitag, 19. September 2014 19:17:22 UTC+2 schrieb michael...@gmail.com:

[Mike Grove]

On Tue, Sep 23, 2014 at 11:53 AM, <michael...@gmail.com> wrote:

I think there is a bug. After the errors the database is not working anymore and the backend shows me Name: "undefined" instead of the database.

This error comes when trying to make new requests:

Database 'semantic' does not exist.

400

That means the database is offline; the rendering of 'undefined' for the name is a bug and has been fixed for the next version.

Cheers,

Mike

Am Freitag, 19. September 2014 19:17:22 UTC+2 schrieb michael...@gmail.com:

Hello there,

I became a really big fan of stardog right now!

Our team is now trying to insert many data in short time.

Unfortunately it's not working, because the server seems to be too overloaded with so many requests. Or maybe we made something wrong ;-)

Here is our log: 

console.log(body) --> Transaction ID

console.log(response.statusCode)

5358d167-a3e5-4e48-9c70-47c9ca4d5b71
500
a676d57a-b436-491b-8a1c-a5b9773da4ae
500
2efd9935-bf2f-4c48-82d4-46987516e442
f3ec7ea8-175f-4d7d-b57e-5d5f85396788
05993b30-a772-4788-a060-bd4b993338fa
200
500
500

Is there some special workaround when making many inserts in a short time period?

Best Regards,

Michael

--

[michael...@gmail.com]

It seems like the DB cannot set ONLINE/OFFLINE now. I have to create a new database now in order to work with. This is a huge bug.

There was an error trying to set the database ONLINE

and

There was an error trying to set the database OFFLINE

[Edgar Rodriguez]

Are you able to set it ONLINE/OFFLINE via the CLI?

To set it online:

    $ stardog-admin db online <db>

to set if offline:

    $ stardog-admin db offline <db>

Cheers, 

Edgar

--

[Edgar Rodriguez]

Can you get any other debug information from the response object additional to the response.statusCode?

Cheers,
Edgar

​

On Tue, Sep 23, 2014 at 11:42 AM, <michael...@gmail.com> wrote:

--

[michael...@gmail.com]

No,

it says "Database 'xyz' does not exist."

[michael...@gmail.com]

Okay,

I have an status update regarding the problem definiton.

The problem arises when using parallel instead of serial usage of transaction.

I wrote a function for inserting test triples:

function action(callback) {

        conn.connection.onlineDB({ database: conn.database, strategy: "NO_WAIT" }, function () {

          conn.connection.begin({ database: conn.database }, function (body, response) {

            txId = body;

            conn.connection.addInTransaction(

              { database: conn.database, "txId": txId, "body": "<bob> <likes> <alice>", contentType: "text/turtle" },

              function (body2, response2) {

                conn.connection.commit({ database: conn.database, "txId": txId }, function (body3, response3) {

                  console.log("Response Code: " + response3.statusCode)

                  callback()

                });

              }

            );

          });

        });

      }

Using the async library I wrote two little tests, both calling the action() function. One serial, one parallel.

      async.serial([

        action,

        action,
        action,
        action,

        action

      ], done);

This works like charm

Status Code: 200

Status Code: 200

Status Code: 200

Status Code: 200

Status Code: 200

whereas the parallel call results in 404 responses:

      async.parallel([

        action,

        action,
        action,
        action,

        action

      ], done);

404...

Status Code: 200

Status Code: 404

Status Code: 404

Status Code: 404

Status Code: 404

Is there something to regard considering parallel calls to the stardog.js api in order to get parallel calls working?

Kind Regards,

Michael

[michael...@gmail.com]

I think the problem has to do with the "restler" library stardog.js is using.

The default value for maxSockets is 5, so after opening more than 5 sockets an error will be thrown.

onlineDB, begin, addInTransaction & commit are already opening 4 sockets. So it's logical that the second parallel call will fail because of the maxSockets restriction.

There should be something like a "tryAgain" function or something else when there are no more sockets available. Is it possible to make changes in the stardog.js and make a pull request?

[Kendall Clark]

On Tue, Sep 30, 2014 at 5:30 AM, <michael...@gmail.com> wrote:

There should be something like a "tryAgain" function or something else when there are no more sockets available. Is it possible to make changes in the stardog.js and make a pull request?

Absolutely. Go for it.

Cheers,

Kendall