TBD shared graph db. Where are the graphs stored? [Dockerized]

[Stefan Verweij]

Hi,

I've deployed a docker service with TopBraid EDG 6.3 as a running instance. Deployment works as it should be and after license registration and setting the configuration parameters (choosing TDB: Store graphs in the general shared database) my TopBraid instance is up and running. I chose to work with the "TBD general shared database", since all graphs will be stored at the root of the workspace in the /_DATA directory (according to https://doc.topquadrant.com/6.3/server-administration/ - EDG Configuration Parameters part). This way I can create a bind volume to /_DATA so all the graphs are stored at my host as well (in case the container fails and all my graphs are lost).

However, when I shell into my container using

docker exec -it [container_id] /bin/bash

to look if my newly added graphs (such as an ontology) are indeed stored in the /_DATA directory, the directory is empty. 

Now my question is where does TopBraid store the graphs? I've added an ontology so it must be located somewhere right?

(This question also applies to license storage and configuration storage, so I don't have to manually register the license and change the EDG configuration parameters every time I start the Docker instance).

Kind regards,

Stefan

[Richard Cyganiak]

Try /_Data under the workspace. Capitalization matters on most operating systems and the documentation has it wrong. We'll fix it.

You might want to consider binding the root of the workspace to a host directory instead, because data ends up in a number of workspace locations:

- /Repositories/, where the system creates connector files for all graphs

- /server.topbraidlive.org/dynamic/, where configuration files and the license file are stored (and sometimes temp files)

- /Corpora/, if you use the Corpus asset collection type with file upload

- /logs/, where the TB log is stored

- Uploaded projects

The rest of the workspace is static files that are not modified after initial installation (unless I'm forgetting something).

Hope that helps,

Richard

--
You received this message because you are subscribed to the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/dab1c5ed-c91e-4d8a-baf7-f65c57ff9c48%40googlegroups.com.

[Stefan Verweij]

Thanks for your fast reply Richard, it is much appreciated.

I cannot find any other `_DATA` or `_data` or `_Data` in my container besides the one at the root level `/`. Also I can only find the logs of the paths you described in the workspace (which is `/usr/local/tomcat/` if I am not mistaken?)

Would love to bind the root of the workspace, but Docker does not allow this - or at least I have not found a way yet to bind `/`. 

Looking forward to your reply.

Regards,

Stefan 

Op dinsdag 2 juni 2020 14:51:05 UTC+2 schreef Richard Cyganiak:

To unsubscribe from this group and stop receiving emails from it, send an email to topbrai...@googlegroups.com.

[Richard Cyganiak]

Stefan,

When I say “workspace” I mean the EDG workspace, which is a directory you configure during installation and it gets stored in the web.xml file. This is usually a directory created for EDG. Our docs say: “Example locations include /var/lib/topbraid/ontologies for Linux and C:/Users/TQ/Workspace for Windows.”

/usr/local/tomcat is the Tomcat installation directory and is something different.

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/cd16445d-9785-42df-bfbf-262a20029939%40googlegroups.com.

[Stefan Verweij]

Thanks Richard, this has been very helpful! Found the path indeed now. It indeed seems logical to just bind to the root of this workspace indeed instead of creating a volume for every subdir.

Though I still need a volume to the installation directory for the server.xml and the web.xml right?

Stefan

Op dinsdag 2 juni 2020 16:12:58 UTC+2 schreef Richard Cyganiak:

To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/cd16445d-9785-42df-bfbf-262a20029939%40googlegroups.com.

[Richard Cyganiak]

Stefan,

On 2 Jun 2020, at 15:45, Stefan Verweij <stefan....@dupont.com> wrote:

Thanks Richard, this has been very helpful! Found the path indeed now. It indeed seems logical to just bind to the root of this workspace indeed instead of creating a volume for every subdir.

Though I still need a volume to the installation directory for the server.xml and the web.xml right?

That depends on how the Docker image is built. It might be possible to put these files into the image as they don't change after installation.

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/fb6d3add-dad7-488f-96ee-c83c98ae3e63%40googlegroups.com.

[Stefan Verweij]

Richard, everything seems to work like you mentioned. Thanks for your help!!

Just FYI about the structure of the Docker image: I currently copy the server.xml and the edg.war to their locations in the TopBraid installation path, I also copy the web.xml to the image (but not yet it's right location). For this, I use a wrapper script which first starts the Tomcat service in the background using catalina.sh start. Copies the web.xml to its right location (though I am not sure if this is /usr/local/tomcat/conf/web.xml or /usr/local/tomcat/webapps/edg/WEB-INF/web.xml). Anyway, then I restart the Tomcat service in the foreground using shutdown.sh and catalina.sh run. 

Have a lovely day,

Stefan

Op dinsdag 2 juni 2020 17:11:33 UTC+2 schreef Richard Cyganiak:

Stefan,

To unsubscribe from this group and stop receiving emails from it, send an email to topbrai...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/fb6d3add-dad7-488f-96ee-c83c98ae3e63%40googlegroups.com.

[Richard Cyganiak]

Hi Stefan,

On 3 Jun 2020, at 09:02, Stefan Verweij <stefan....@dupont.com> wrote:

Richard, everything seems to work like you mentioned. Thanks for your help!!

Just FYI about the structure of the Docker image: I currently copy the server.xml and the edg.war to their locations in the TopBraid installation path, I also copy the web.xml to the image (but not yet it's right location). For this, I use a wrapper script which first starts the Tomcat service in the background using catalina.sh start. Copies the web.xml to its right location (though I am not sure if this is /usr/local/tomcat/conf/web.xml or /usr/local/tomcat/webapps/edg/WEB-INF/web.xml). Anyway, then I restart the Tomcat service in the foreground using shutdown.sh and catalina.sh run. 

The web.xml should go into webapps/edg/WEB-INF/web.xml within the Tomcat directory.

One thing you also could try: edg.war is really just a zip file, and if you put web.xml into the zip file at WEB-INF/web.xml, then it will end up in the right place after edg.war is deployed, and this may work without a Tomcat restart.

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/01dfb927-7337-4cd5-b339-76baa133f262%40googlegroups.com.

[Stefan Verweij]

Thanks Richard, this seems to work as well.

I have a new question, partially related to this question. I am running two docker services, one with a TopBraid instance with the dev license and the other with the production license. Each service listens to its unique port on the same host. One of the services runs without problems, the other exits with error code one stating 

Error occurred during initialization of VM
Too Small initial heap

I guess this has something to do with the JAVA_OPTS env variable (https://www.topquadrant.com/docs/tbl/TBLInstallGuide/index.html), which I declare once in each Docker service as follows:

environment:
  - JAVA_OPTS=-Xmx4g

ShouldI take another value for the JAVA_OPTS? Or how can I solve this error message otherwise?

Op vrijdag 5 juni 2020 16:02:17 UTC+2 schreef Richard Cyganiak:

Hi Stefan,

To unsubscribe from this group and stop receiving emails from it, send an email to topbrai...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/01dfb927-7337-4cd5-b339-76baa133f262%40googlegroups.com.

[Richard Cyganiak]

Stefan, please double-check both values for the -Xmx setting. This error can occur when accidentally setting something like -Xmx4 instead of -Xmx4g.

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/06fd3b1f-2bad-4d79-9731-ca28596eea8fo%40googlegroups.com.

[Stefan Verweij]

Your feedback and answers are so on point it's almost scary :)

Was indeed a type and all TopBraid services seem to work fine now indeed!

Have a nice day,

Stefan

Op donderdag 11 juni 2020 14:59:39 UTC+2 schreef Richard Cyganiak:

Stefan, please double-check both values for the -Xmx setting. This error can occur when accidentally setting something like -Xmx4 instead of -Xmx4g.

Richard

To unsubscribe from this group and stop receiving emails from it, send an email to topbrai...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/06fd3b1f-2bad-4d79-9731-ca28596eea8fo%40googlegroups.com.