SKOS Places taxonomy import problems: BT/NT relationships dropped

[Anne-Marie]

Hello, team,

While attempting to import an expanded Places taxonomy that we developed as SKOS RDF XML, AtoM (2.1) displayed the following error message, "An object cannot be a descendant of itself." In reviewing the results, it appears that all 417 terms from our file imported, but about halfway through the file the broader term (BT) and narrower term (NT) relationships between records stopped being recognized, such that everything after a certain term ("Lythrangomi") appear as orphan terms. I have attached a copy of the file in question.

Note that there are child terms displaying for select records after "Lythrangomi", specifically for MD, MA, Paris, Princes Islands, Sicily, Slovakia, Spain, Syria, Turkey, UK, US and Washington DC. This appears to be related to where the terms appear in the XML:  that is all the countries appear as topConcepts at the beginning of the file and all the NT's for the terms in this list that are NOT countries, all begin with a letter before "L" (so before Lythrangomi in the file).

We don't believe this to be a problem of duplicative preferred terms, as we addressed these within the file using qualifiers (e.g. - "Cambridge, MA" to differentiate from "Cambridge" in the UK) after we identified the problematic slug issue which we previously posted about (https://groups.google.com/d/msg/ica-atom-users/PrI5WiMVmX8/kioo_fTT00sJ).

Background: We used the following taxonomy freeware, TemaTres, to produce the SKOS file and successfully imported a smaller test file. View our taxonomy here, http://koronis.at/tematres/vocab/index.php.

Any suggestions on how to resolve without having to manually enter all missing relationships post-import?

Thanks,

Anne-Marie H. Viola

Metadata & Cataloguing Specialist,

Image Collections and Fieldwork Archives (ICFA)

Dumbarton Oaks Research Library and Collection

1703 32nd Street, NW Washington, DC 20007

http://www.doaks.org/library-archives/icfa

violaa[at]doaks.org

[Dan Gillean]

Hi Anne-Marie,

Are you performing this import via the user interface, or the command line? If the former, it seems possible that the script is timing out partially through the import, and that is why the relationships are not being created?

In 2.1 there is a --taxonomy option for the command-line XML import, that will allow you to target a specific taxonomy (e.g. Places) for import. You may be running 2.0 at your institution, I can't recall, but if it's possible to update, I would recommend trying this and seeing if it resolves the issue.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "ICA-AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at http://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d6774700-fc9d-4a5e-9120-61de66b441c8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[prathmes...@gmail.com]

Hi Dan,

   Me and Anne-Marie here at Dumbarton Oaks have been importing this using the CLI.

I even used the -v (verbose) option to see if I get an more info about the Error, but no luck. And even the output option produces an empty csv file, which I hoped would produce or give more insight on this error.

php symfony import:bulk --taxonomy="42" --output="/path/to/output.csv" --v /path/to/mySKOSfiles

This script runs for sometime and produces the above mentioned error on the CLI "An object cannot be a descendant of itself."

Best,
Prathmesh Mengane,
Database and CMS Developer,
Dumbarton Oaks.

[Dan Gillean]

Hi Prathmesh,

I will try to find some time to test your import locally.

In the meantime, I want to make sure you are in fact importing on a 2.1 instance - otherwise, the --taxonomy command won't work. Note as well that the --verbose option was changed and fixed in 2.1 - in 2.0.1, it requires a parameter (e.g. --v=1 ), while in 2.1 it was changed to --verbose, and no longer requires a parameter. I'm not sure why the --output option is not working for you, but I do know it was improved in 2.1. 

In any case, I will try to set aside some time soon to try running the import locally, and see if the file fails for me in the same way. Have you tried to validate the SKOS file in any way to ensure that there is not a problem in the hierarchy, and to make sure that there are no instances where a child term is created before its parent?

Cheers,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/6abd3da7-3656-49bf-979f-f71dc9dd4e67%40googlegroups.com.

[prathmes...@gmail.com]

Hi Dan,

   Yes, we are importing this in 2.1 using the CLI.

 And surprisingly even the,--verbose command doesn't provide me any detailed step by step procedure of the executing script. What is it designed to display within the CLI when the script is running ?

Thanks, for taking time to test it on your instance. Let us know what you discover.

Best,
Prathmesh

[Anne-Marie]

Dan, I think we finally figured out the problem -- thanks to you! Lythrankomi is where the problem began and is a child of Northern Cyprus, which given that all non-top concept terms appear in alpha order, this parent-child combo are in reverse order.

We are reviewing the file for any other possible instances of this problem and will have a revised version ready for testing within a day or so, and will be sure to let you know the results!

Thanks again,

Anne-Marie (& Fani)

[Anne-Marie]

Update:  We revised the SKOS RDF XML file for our Places taxonomy based on the theory that parent-child terms need to appear in hierarchical order within the XML file -- and that appeared to solve the problem! We imported the file yesterday and all relationships now appear correctly formed for all terms. (Note: Immediately following import we noticed a number of 500 Internal Server errors and a few missing relationships, which Prathmesh was able to resolve by rebuilding the MYSQL nest.)

To elaborate for future reference, we identified about a dozen instances in which a parent term appeared after a child term in the file because of alpha order. This didn't seem to be a problem for any second-level term as those terms' parents were top concepts. But for third-level terms that appeared before their second-level parent, I moved the parent up in the list to appear before the child term. If a the relationship was between a third- and a fourth-level term, I chose to move the fourth-level term down to appear after its parent to ensure I didn't disturb the order in which the the second- and third-level terms appear.

I also want to note that while this worked, it is more of a "fix" than a real solution. Should we produce a larger file, we would need to do similar analysis and restructuring because of the alpha-order of the output. As such, we are relaying our feedback to the developers of the taxonomy freeware that we used to let them know that it would be useful if terms appeared in hierarchical order (i.e. - that all similar "level" terms appear in groups, beginning with top concepts, then first-level, etc.).

Thanks again for helping us identify the problem!

Best,

Anne-Marie, Fani & Prathmesh

Image Collections and Fieldwork Archives (ICFA)

Dumbarton Oaks Research Library and Collection

[Dan Gillean]

Anne-Marie,

That's great news! Yesterday afternoon I had gotten as far as setting up a local 2.1 instance and attempting to import the file - I had reproduced the same error you were encountering, but had not yet had a chance to analyze the SKOS file for irregularities.

I'm glad to hear you've sorted things out - thanks so much for posting back an update.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Product Manager / Systems Analyst,
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/5b49d7fa-817c-4dac-a72b-9a152d070f85%40googlegroups.com.