Slowdown and slowly running out of memory

[Håvard Ottestad]

Hi,

I have just over 100 000 000 triples in Stardog with -Xmx3g -XX:MaxDirectMemorySize=4g.

I have around 5 000 000 entities in the database, and I'm running 1 query per entity to extract it as a document and index it into Elasticsearch.

At the beginning I manage around 7000 queries a minute, but this slows down to around 200 after 18 hours.

These are the errors we are seeing:

ERROR 2017-01-24 03:57:08,875 [Stardog.Executor-4277] com.complexible.common.protocols.server.rpc.ServerHandler:exceptionCaught(428): exceptionCaughtServerHandler

java.lang.OutOfMemoryError: GC overhead limit exceeded

ERROR 2017-01-24 03:57:08,876 [StardogServer.WorkerGroup-8] com.complexible.stardog.protocols.http.server.HttpMessageEncoder:createErrorResponse(293): The result encoder received an error message it could not encode, error was:

java.lang.OutOfMemoryError: GC overhead limit exceeded

I think there might be a memory leak of some kind, or issues with discarding old cache.

Cheers,

Håvard

[Håvard Ottestad]

After restarting, this is the server status:

bash-4.3# ./stardog-admin server status    

Backup Storage Directory : .backup

CPU Load                 : 0.0000 %

Connection Timeout       : 1h

Export Storage Directory : .exports

Memory Heap              : 900M (Allocated: 1.6G Max: 2.7G)

Memory Non-Heap          :  82M (Allocated:  88M Max: 4.0G)

Named Graph Security     : false

Platform Arch            : amd64

Platform OS              : Linux 4.4.0-45-generic, Java 1.8.0_92-internal

Query All Graphs         : false

Query Timeout            : 5m

Stardog Home             : /home/stardog/data

Stardog Version          : 4.2.2

Strict Parsing           : true

Uptime                   : 11 minutes 21 seconds

Watchdog Enabled         : true

Watchdog Port            : 5833

Watchdog Remote Access   : true

Databases                :

+------------+------------+-------------+-----------------+----------------------+-----------+

|    Name    | Conn. Open | Trans. Open | Running Queries | Query Avg. Time (ms) | Query/sec |

+------------+------------+-------------+-----------------+----------------------+-----------+

| brukere    |          0 |           0 |               0 |                  0.0 |       0.0 |

| testSesame |          3 |           0 |               0 |                  3.0 |     333.3 |

+------------+------------+-------------+-----------------+----------------------+-----------+

[Håvard Ottestad]

After an hour CPU use has risen a lot. Likely doing a lot of GC.

bash-4.3# ./stardog-admin server status

Backup Storage Directory : .backup

CPU Load                 : 53.5985 %

Connection Timeout       : 1h

Export Storage Directory : .exports

Memory Heap              : 2.5G (Allocated: 2.9G Max: 2.9G)

Memory Non-Heap          :  89M (Allocated:  92M Max: 4.0G)

Named Graph Security     : false

Platform Arch            : amd64

Platform OS              : Linux 4.4.0-45-generic, Java 1.8.0_92-internal

Query All Graphs         : false

Query Timeout            : 5m

Stardog Home             : /home/stardog/data

Stardog Version          : 4.2.2

Strict Parsing           : true

Uptime                   : 1 hour 14 minutes 51 seconds

Watchdog Enabled         : true

Watchdog Port            : 5833

Watchdog Remote Access   : true

Databases                :

+------------+------------+-------------+-----------------+----------------------+-----------+

|    Name    | Conn. Open | Trans. Open | Running Queries | Query Avg. Time (ms) | Query/sec |

+------------+------------+-------------+-----------------+----------------------+-----------+

| brukere    |          0 |           0 |               0 |                  0.0 |       0.0 |

| testSesame |          3 |           0 |               0 |                  4.0 |     250.0 |

+------------+------------+-------------+-----------------+----------------------+-----------+

[Håvard Ottestad]

After two hours the number of queries that it's managing is around 3000. So less than half of what it manages on a fresh boot.

Here is the server status.

bash-4.3# ./stardog-admin server status

Backup Storage Directory : .backup

CPU Load                 : 62.6033 %

Connection Timeout       : 1h

Export Storage Directory : .exports

Memory Heap              : 2.7G (Allocated: 2.9G Max: 2.9G)

Memory Non-Heap          :  90M (Allocated:  92M Max: 4.0G)

Named Graph Security     : false

Platform Arch            : amd64

Platform OS              : Linux 4.4.0-45-generic, Java 1.8.0_92-internal

Query All Graphs         : false

Query Timeout            : 5m

Stardog Home             : /home/stardog/data

Stardog Version          : 4.2.2

Strict Parsing           : true

Uptime                   : 2 hours 0 minutes 57 seconds

Watchdog Enabled         : true

Watchdog Port            : 5833

Watchdog Remote Access   : true

Databases                :

+------------+------------+-------------+-----------------+----------------------+-----------+

|    Name    | Conn. Open | Trans. Open | Running Queries | Query Avg. Time (ms) | Query/sec |

+------------+------------+-------------+-----------------+----------------------+-----------+

| brukere    |          0 |           0 |               0 |                  0.0 |       0.0 |

| testSesame |          2 |           0 |               0 |                  6.0 |     166.7 |

+------------+------------+-------------+-----------------+----------------------+-----------+

[Zachary Whitley]

Are you executing your queries from the CLI or API? If it's the API there is a chance that you're not freeing some resource. Possibly opening multiple connections and not closing them or not closing your result sets. Can you share your code?

--
-- --
You received this message because you are subscribed to the C&P "Stardog" group.
To post to this group, send email to sta...@clarkparsia.com
To unsubscribe from this group, send email to
stardog+unsubscribe@clarkparsia.com
For more options, visit this group at
http://groups.google.com/a/clarkparsia.com/group/stardog?hl=en
---
You received this message because you are subscribed to the Google Groups "Stardog" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stardog+unsubscribe@clarkparsia.com.

[Håvard Ottestad]

Java api.

I have a connection pool, all connections are released.

The query is a graph query, and I use Iterations.stream on the results. Iterations.stream automatically closes the graph result set.

[Håvard Ottestad]

I have found one location in my code where a result set is being closed, but not by a try-with-resource or finally block. So if there were to be an exception before the close was called it would potentially leave resources open. I doubt it was what was causing the issues, but might be.

[Håvard Ottestad]

Is there any way to check what resources are still open by a connection for debugging?