Data is not visible on frontend after data import

[go.save...@gmail.com]

Hello,

I have a question regarding data import.

I recently installed SKOSMOS with Docker-compose on a remote server.

Frontend port 9090 and backend Fuseki backend port 9030 are basically up and running, following the installation instructions. Pre-installed empty vocabularies are visible on 9090 (STW Thesaurus for Economics and UNESCO Thesaurus). Then, the following two commands returned no error and I can see this data on 9090.

# load UNESCO vocabulary data

curl -L -o unescothes.ttl http://skos.um.es/unescothes/unescothes.ttl

curl -I -X POST -H Content-Type:text/turtle -T unescothes.ttl -G http://localhost:3030/skosmos/data --data-urlencode graph=http://skos.um.es/unescothes/

I think everything is fine by this point.

But, I try the same for my datasets, I can see data on 9030 (when making SPARQL query or seeing info in SKOSMOS data set), but data cannnot be seen on 9090.

sudo curl -I -X POST -H Content-Type:text/turtle -T UT_test.ttl -G http://130.37.53.108:9030/skosmos/data --data-urlencode graph=https://vocabs.acdh.oeaw.ac.at/unit_of_time/

Also, when I add new dataset and import it with Fuseki GUI on 9030, I can see data on 9030 (when making SPARQL query or seeing info in SKOSMOS data set), but data cannnot be seen on 9090.

I do not know where to start to solve this problem.

Do you have any idea what the problem would be?

One additional question is where the imported data reside? I see some configuration files

in ttl files in /dockerfiles and/or /ansible/roles/skosmos/files/config.ttl, as well as

docker-compose.yml in /dockerfiles, but I am not sure what is the right way to configure and where the data is stored.

Many thanks!

Go Sugimoto

[go.save...@gmail.com]

Sorry URIs are not written properly in the previous email.

So, please ignore URIs mistakes below, they are just examples. In practice, the correct URIs are used

(9030 are used in those cases becaus we use docker-compose).

http://localhost:3030/skosmos/data >> http://CORRECT_URI_HERE:9030/skosmos/data

http://130.37.53.108:9030/skosmos/data >> http://CORRECT_URI_HERE:9030/skosmos/data

[Bruno P. Kinoshita]

Hi Go,

>Also, when I add new dataset and import it with Fuseki GUI on 9030, I can see data on 9030 (when making SPARQL query or seeing info in SKOSMOS data set), but data cannnot be seen on 9090.

From what I understood, you can upload and query data in Jena Fuseki. So that part of your installation appears to be OK.

As you cannot see the same data in SKOSMOS, in the port 9090, I suspect your configuration file for SKOSMOS or vocabulary needs to be corrected. When using Docker Compose, the service names are used as host names. So Jena Fuseki will be fuseki:9030, and SKOSMOS will be skosmos:9090. The Docker Compose installation includes a cache, mapped as fuseki-cache:80.

Depending on how your services were created, you may have to change some of these settings:

- https://github.com/NatLibFi/Skosmos/blob/ffc1d62c9ae982fde76244857cf52abdaed7788a/dockerfiles/config/config-docker-compose.ttl#L21

- https://github.com/NatLibFi/Skosmos/blob/ffc1d62c9ae982fde76244857cf52abdaed7788a/dockerfiles/config/config-docker-compose.ttl#L107

(please note that it is really hard for us to troubleshoot your environment settings, even with clear explanations, we might miss something that would be clearer to you or somebody else with access to your environment.)

>One additional question is where the imported data reside? I see some configuration files in ttl files in /dockerfiles and/or /ansible/roles/skosmos/files/config.ttl, as well as docker-compose.yml in /dockerfiles, but I am not sure what is the right way to configure and where the data is stored.

The vocabulary data resides in Jena Fuseki, after you uploaded to it. It will be inside your Jena Fuseki container, unless you have mounted a volume. It is recommended to use a Docker volume with your Jena Fuseki, so that restarts/re-installations don't delete your data - See https://docs.docker.com/storage/

The SKOSMOS .ttl configuration file used by Docker Compose is ./config/config-docker-compose.ttl (ref https://github.com/NatLibFi/Skosmos/blob/ffc1d62c9ae982fde76244857cf52abdaed7788a/dockerfiles/docker-compose.yml#L44). It will be bound as a volume when the Docker Compose services are created. The ansible files are used only if you install SKOSMOS via Ansible, they are not used when you use Docker & Docker Compose directly.

You can customize the docker-compose.yml via an override file - https://docs.docker.com/compose/extends/#understanding-multiple-compose-files

Hope that helps,

Bruno

--
You received this message because you are subscribed to the Google Groups "Skosmos Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skosmos-user...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/skosmos-users/331def53-7f11-4e19-8bfb-6fee4a2197f4n%40googlegroups.com

.

[go.save...@gmail.com]

Dear Bruno,

Thank you so much for your prompt support!

I manged to import data with CURL by adding vocabulary info in the configuration file (dockerfiles/config/config-docker-compose.ttl)

It is nearly there, but would you mind if I ask three related questions?:

1) Unfortunately I am not all familar with Docker-compose, and, if I stop and restart Docker-compose, then

on the frontend (port 9090), only home page (metadata page) remains and imported data is lost. How to keep the imported data, after restart? (Is it possible?)

2) As said above, I managed to import data via CURL, but I don't understand how to import data via the Jena backend (port 9030). On 9030 , I created a new dataset and imported data by uploading files with GUI. But, the uploaded dataset is only visible on 9030. On 9090, I do not see anything on the home page (i.e. list of vocabularies). What configuration is needed to create a new vocabulary there and see imported data?

3) The vocabulary imported via CURL can be seen in the default skosmos dataset on 9030 GUI, but how to specify another dataset within CURL command to import it, so that I can see it it in that dataset on 9030 GUI?

 

Many thanks in advance!

Go

[Bruno P. Kinoshita]

Hi Go,

On 1): You must create a volume in your docker-compose.yml configuration. We use the jena-docker image which is configured [1] to use the /fuseki directory for the Fuseki data. There is a comment in the Skosmos' docker-compose.yml about it, that can be used as reference to configure it: https://github.com/NatLibFi/Skosmos/blob/31f430b65be209d55209d53729c6beb7c92df984/dockerfiles/docker-compose.yml#L13-L16

Doing that, the container data might be destroyed, but not the volume data. So when you restart it, the container will bind the /fuseki directory to the existing volume the next time you start it.

Now on 2) & 3) I think you are using curl as in these instructions [2]. If you uploaded your data to Jena Fuseki in 9030, and queried and the results are what you expect, the next step is to configure Skosmos.

I think what you must be missing is just an entry for your vocabulary (e.g. [3])? If you have the entry, and the vocabulary is displayed when you browse the Skosmos GUI, then you must review the configuration as Skosmos might be failing to access Jena Fuseki.

https://github.com/NatLibFi/Skosmos/blob/master/dockerfiles/README.md#running-with-docker-compose

Maybe you could share your configuration in a gist or repository, so that myself or others could take a look if there's anything wrong?

Cheers

Bruno

[1] https://github.com/stain/jena-docker/tree/master/jena-fuseki#data-persistence

[2] https://github.com/NatLibFi/Skosmos/blob/master/dockerfiles/README.md#loading-vocabulary-data

[3] https://github.com/NatLibFi/Skosmos/blob/31f430b65be209d55209d53729c6beb7c92df984/dockerfiles/config/config-docker-compose.ttl#L100-L108

https://groups.google.com/d/msgid/skosmos-users/2ef20816-ec43-47f0-8b34-ebdef38c611bn%40googlegroups.com

.

[go.save...@gmail.com]

Dear Bruno,

Thank you so much for your feedback. Good support!

Basically problems are not solved yet :( I explain again my situation with config files in Gist.

1) I did what you said (uncomment 2 lines in docker-compose.yml), and I stoped and restarted, docker-compose, but

all data imported is gone. Here is nothing special yml file: https://gist.github.com/GO5IT/2f4ae21adc9c1108d3d52f8f80ada371

I do not know what to do with the following instruction you gave me, because I use Docker-compose not Docker. Should I still follow what is written in it?: https://github.com/stain/jena-docker/tree/master/jena-fuseki#data-persistence

2 and 3) By adding a vocabulary in the config file (config-docker-compose.ttl): https://gist.github.com/GO5IT/019ccbb7fbd152e49480c1a8c03a7001, I can see the empty voc on the home page of 9090 SKOSMOS GUI. Then, I can import vocabulary by CURL, thus, I can see data on 9090. However, as I asked in the previous message, I do not know how to specify a dataset name in the CURL command. Currently I cannot see CURL imported data on Fuseki inferface on 9030 (e.g. Data Manage page http://xxx.xx.xx.xxx:9030/manage.html), after an import success message by CURL. My CURL looks like:

sudo curl -I -X POST -H Content-Type:text/turtle -T metadataOnly.ttl -G http://xxx.xx.xx.xxx:9031/skosmos/data --data-urlencode graph=https://go.com/

Basically, I do not understand the mechanism to make connections between the CURL command and data management interface on 9030.

In addition, as said in the previous message, when I upload skos files in Fuseki 9030 interface, it doe not become visible on SKOSMOS GUI on 9090 (i.e. create a dataset, selecting the option "Persistent" , and specify https://go.com/ as graph name).

Cheers,

Go

[Bruno P. Kinoshita]

Hi Go,

Thanks a lot for the patience. It's hard (for both parties) to troubleshoot configuration issues like this. But I think I found the issue after using your gist.

On 1)  I created a pull request for Skosmos to update the comment in the Docker Compose file. Your gist has two `volume` keys under the Fuseki service. Turns out the example in the comment cannot be simply uncommented. I updated the text to make it clearer. See the pull request [1] for an example of what it should look like, but it's basically like this:

    volumes:
      - type: bind
        source: ./config/skosmos.ttl
        target: /fuseki/configuration/skosmos.
      - ${PWD}/fuseki:/fuseki

It's important to have items aligned in YAML for the Docker compose (I'm using my Yahoo mail, the lines may not be well formatted above, if that happens, see the pull request [1] for a better version.)

On 2 & 3) I've added some steps in the Skosmos Wiki for Docker [2]. I re-created my dev environment with your gist, but then realized Varnish was exiting with status 2 after it was started by Docker Compose. The error showed that the container couldn't bind its own port `80`, probably due to the user used in the Varnish container.

Some web searching led me to this issue [3] that seems to indicate that they recently-ish broke the image, making the default settings unusable with the image (i.e. users must use Varnish+Docker on a different port.) Since it's not straightforward for me to configure Varnish to use a different port, in the pull request [1] I simply pinned on version 6.x of Varnish, instead of latest/7.x.

Take a look at the pull request, and try applying those changes locally (the part with the version after the image name, after the `:` colon symbol.)

It'd be great if you could try it, and maybe report in that pull request if that solved your issue.

Thank you

Bruno

[1] https://github.com/NatLibFi/Skosmos/pull/1294

[2] https://github.com/NatLibFi/Skosmos/wiki/Install-Skosmos-with-Fuseki-in-Docker#troubleshooting

[3] https://github.com/varnish/docker-varnish/issues/42

https://groups.google.com/d/msgid/skosmos-users/30dc85ac-1e51-4d35-8d7b-369ce42993bfn%40googlegroups.com

.

[go.save...@gmail.com]

Dear Bruno,

Sorry for a bit of silence. Holidays and travel season started and it is hard to get back to normal ;-)

So, 1) is tested and it worked! Thank you! So, for readers, the new YML file should be used for docker-compose and

uncomment  - ${PWD}/fuseki:/fuseki

See the used file here: https://gist.github.com/GO5IT/90731d0c1e0d50426d2696ac948c9671

2) and 3),

With the new YML file for 1), I tried but the situation is the same. Data import on the backend at 9090 say successful, but data does not appear at all on frontend at 9030. CURL also show success, but data does not appear at 9030. If data cannot be imported properly, we cannot use SKOSMOS, although it is running. I do not know where to find a problem...