SKOS import in demo site

[jesus san miguel]

I was having issues importing SKOS taxonomies in my site, and just to be sure went to the demo site, but I am getting a 500 Internal server error just by clicking on the menu.

Just in case someone has been successful in importing it:

http://tesauros.mecd.es/tesauros/geografico/1208851.rdf

Best wishes for all this Christmas!

Jesus 

[Dan Gillean]

Hi Jesus, 

Imports are disabled in the public demo site for security purposes, so the 500 error there is expected. We don't have a configurable way to disable imports via the settings, so the current solution does lead to a 500 error message, unfortunately. 

SKOS is a very flexible standard, and not all SKOS files will import directly into AtoM as expected, as AtoM only supports some SKOS fields and configurations. I recommend that you create some test terms in your AtoM installation (e.g. a few terms in a taxonomy that use a number of fields, and have both hierarchical and associative relations) and then export the parent term, so you can look at how AtoM renders the SKOS and compare this to your import file. It may be possible to make minor changes to your SKOS file for it to import as expected into AtoM.

Additionally, if the file is large, you may want to import it via the command-line to avoid timeout issues. If you encounter local 500 errors, be sure to check the webserver error logs for further information: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/logging/#web-server-logs 

Good luck!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/38a31f70-8fb3-4489-b786-f7fb9afd4da0n%40googlegroups.com.

[jesus san miguel]

Hi Dan,

We just managed to tweak the SKOS file so that AtoM likes it, but now it won't import from the command line (perhaps related to https://projects.artefactual.com/issues/12364):

php symfony import:bulk --verbose --taxonomy="42" /root/funciona-6.rdf

Importing 1 files from /root/funciona-6.rdf (indexing is DISABLED) ...

Importing: /root/funciona-6.rdf

Imported 0 XML/CSV files in 2.19 s. 9346664 bytes used.

The same file imports fine through the user interface:

[info] [2022-02-14 07:48:23] Job 157846 "arFileImportJob": Job started.
[info] [2022-02-14 07:48:23] Job 157846 "arFileImportJob": Importing SKOS file: funciona-1.rdf.
[info] [2022-02-14 07:48:23] Job 157846 "arFileImportJob": Type of scheme: file
[info] [2022-02-14 07:48:23] Job 157846 "arFileImportJob": Taxonomy: Places
[info] [2022-02-14 07:48:23] Job 157846 "arFileImportJob": Term ID: 1
[info] [2022-02-14 07:48:26] Job 157846 "arFileImportJob": The graph contains 6804 concepts.
[info] [2022-02-14 07:48:50] Job 157846 "arFileImportJob": A total of 100 concepts have been processed so far.
[info] [2022-02-14 07:49:12] Job 157846 "arFileImportJob": A total of 200 concepts have been processed so far.
[info] [2022-02-14 07:49:34] Job 157846 "arFileImportJob": A total of 300 concepts have been processed so far.

...

 but when submitted the full repository (a mere 150k places, in a 25MB rdf file) it aborts with a no memory exception:

[info] [2022-02-11 13:36:02] Job 806 "arFileImportJob": Job started.
[info] [2022-02-11 13:36:02] Job 806 "arFileImportJob": Importing SKOS file: viernes.rdf.
[info] [2022-02-11 13:36:02] Job 806 "arFileImportJob": Type of scheme: file
[info] [2022-02-11 13:36:02] Job 806 "arFileImportJob": Taxonomy: Places
[info] [2022-02-11 13:36:02] Job 806 "arFileImportJob": Term ID: 1
[info] [2022-02-11 13:36:18] Job 806 "arFileImportJob": Exception: XML error: "No memory" on line 558035, column 1

Despite our settings with no memory limit in php-fpm daemon:

 /usr/bin/php -d memory_limit=-1 -d error_reporting=E_ALL symfony jobs:worker

What am I missing here?

Jesus

[Dan Gillean]

Hi Jesus, 

I'm not sure - the error message looks different than the issue ticket you've found. Would you be willing to share the tweaked file, so I can try importing it locally and see if I can reproduce the issue?

Also, please remind me about your installation environment - full version number, any changes made from the recommended documentation, etc. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e1cd07a8-4223-41ff-86c5-c737e5a3d504n%40googlegroups.com.

[jesus san miguel]

Sure Dan, please find attached the partial (funciona-1.rdf) and full (viernes.rdf) geographical data.

I tested on:

2.4.0 - 156

2.6.0 - 184 (docker)

2.6.4 - 184
No customization whatsoever... I did a barebones 2.6.4 install to test this.

Best,

Jesus

[Dan Gillean]

Hi again Jesus, 

Regarding the Viernes RDF file, I'm beginning to suspect that this is in fact a memory issue. 

While you have set your atom-worker to have no memory limit, the import can still exhaust all the available memory on your server. When I tried importing the funciona-1 file in my test Vagrant box, it took about 25 minutes just to create the first ~2000 terms. Since the viernes file is more than 5x the size, I'm not surprised that it may fail if your system doesn't have a lot of memory allocated - since much of the SKOS file is just references from one concept to another, I think that AtoM may try to load the whole file into memory for processing. This would explain why it fails before even starting with the Viernes RDF file. 

I'm not sure exactly why the command-line import attempt aborts with 0 concepts imported - I'll ask our developers if they have any ideas. 

Another factor that may be complicating things - these files are not only using SKOS RDF, but also seem to wrap every concept in OWL annotation tags or some other schema, which AtoM may not understand. My test import has still not yet completed, but it looks like most of the new concepts may be coming in untitled (even when viewed in Spanish). I also don't see skos:Concept elements used, which AtoM expects, which is likely why many terms are not importing as expected. Here's a sample from the funciona-1 file: 

    <NamedIndividual rdf:about="http://tesauros.mecd.es/tesauros/geografico/1108062">
        <rdf:type rdf:resource="http://www.w3.org/2004/02/skos/core#Concept"/>
        <skos:broader rdf:resource="http://tesauros.mecd.es/tesauros/geografico/1108063"/>
        <skos:inScheme rdf:resource="http://tesauros.mecd.es/tesauros/geografico"/>
        <skos:prefLabel xml:lang="es">Cavandi</skos:prefLabel>
    </NamedIndividual>

Notice that it is wrapped in the <NamedIndividual> element (which is not a part of SKOS), and uses the <rdf:type> element instead of AtoM's expected <skos:Concept> element. Compare that to a sample Place term exported from AtoM:

<skos:Concept rdf:about="http://10.10.10.10/british-columbia">
<skos:prefLabel xml:lang="en">British Columbia</skos:prefLabel>
<skos:prefLabel xml:lang="fr">Colombie-Britannique</skos:prefLabel>
<skos:inScheme rdf:resource="http://10.10.10.10/places"/>
<skos:narrower rdf:resource="http://10.10.10.10/new-westminster"/>
</skos:Concept>

One of our analysts did document a process for modifying Library of Congress Subject Heading files to be importable into AtoM - as you'll see, it was not an automatic process, but required several manual steps to work: 

• https://projects.artefactual.com/issues/13444#note-1
  

So, I suspect there may be a number of factors at play here, and further troubleshooting may be beyond the level of support we can provide via the forum. 

Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/1565abf2-72ec-4c3a-9776-71f85e074663n%40googlegroups.com.

[jesus san miguel]

Just for your reference, here I have attached the Spain rdf converted to the exact Atom favored syntax (skos:Concept elements used) that produces Exception: XML error: "No memory" on line 218333, column 1 but imports fine if cut in several pieces, like the also attached Mallorca rdf file. 

I also checked memory exhaustion is not the issue here: no significant memory is consumed upon the import exception. I would bet on a short vs long variable being used in the import routine.

Best,

Jesus