AAT Query for Adlib-windows

[Rolf Blijleven]

As posted a while ago here, I'm working on an interface between GVP and Adlib for Windows. The user enters (the left part of) a term in some Adlib field. This term is then queried at the GVP AAT sparql endpoint. The response is tranformed into the Adlib XML dialect, and presented to the user. The user can then select a term, after which the following is copied into the user's own Adlib thesaurus:  

- the term itself, of course

- its GVP ID (plus link); an important reason for having the interface in the first place, is that it enables cross-collection matching by term ID. 

- preferred broader term and GVP ID (plus link)

- scope note

- comment

- start and end date. This is seldom used, but there is space for it, so why not. 

This query does the job. The search term 'crystol*' and the language ('nl') will normally be variables. For now, I'm planning on Dutch and English.  

    SELECT * WHERE {  

      ?subject luc:term "crystol*" .  

      OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?term]}  

      OPTIONAL {?subject gvp:broaderPreferred ?broader .  

                ?broader xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broaderterm]}  

      OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note_nl]}  

      OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl; rdfs:comment ?comment]}  

      OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl; gvp:estStart ?estStart]}  

      OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl; gvp:estEnd ?estEnd]}  

      filter regex(?term, '^crystol', 'i')  

      }  

    ORDER BY ?term  

    LIMIT 12

The filter removes hits dissimilar to the original search string, just to mimic Adlibby behaviour. If performance is significantly better without the filter, I'll probably remove it. The whole thing does need to be nicely responsive. 

I would appreciate any comments on this. Can it be improved? Did I miss anything? The use case is manual term-by-term searches: there'll be at least minutes, if not hours, between one query and the next from one particular user. I'm not expecting huge popularity either. 

[Kevin Ford]

Hi Rolf,

The query looks fine from my perspective, for whatever that is worth,
and I won't be offended if my opinion is given little stock.

I do have three comments.

1) If you always know you only want results from AAT, you can stipulate
the skos:ConceptScheme:
?subject skos:inScheme <http://vocab.getty.edu/aat/> .

That will further reduce the result set.

2) The REGEX FILTER is interesting. Adding the above condition
(scheme=AAT) winnows the results to one, and it is the one in which you
are interested. That makes the FILTER seem unnecessary but its presence
ensures that the searched word is found in the prefLabel of the chosen
language. Compare this query to yours (note the slight difference in
spelling between the term being searched and the term being filtered).

SELECT * WHERE {
?subject luc:term "crystal*" .
?subject skos:inScheme <http://vocab.getty.edu/aat/> .

OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
xl:literalForm ?term]}
OPTIONAL {?subject gvp:broaderPreferred ?broader .
?broader xl:prefLabel [dct:language gvp_lang:nl;
xl:literalForm ?broaderterm]}
OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl;
rdf:value ?scope_note_nl]}
OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
rdfs:comment ?comment]}
OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
gvp:estStart ?estStart]}
OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
gvp:estEnd ?estEnd]}
filter regex(?term, '^crystol', 'i')
}
ORDER BY ?term
LIMIT 12

I'd keep the FILTER for now. It is applied (or should be) after the
base results have been collected so the performance hit should be
negligible, assuming the number of hits is small enough. Moreover, the
FILTER seems paramount to what you hope to achieve - it not only ensures
the searched term is in the prefLabel of the chosen language but it also
performs a left-anchor match on the prefLabel, which the luc:term search
does not guarantee (for the field).


3) At this time, I do not see a reason why the skosxl:prefLabel
condition should be OPTIONAL at all. The presence of the FILTER in your
query means that if the prefLabel does not contain the searched string,
then no results/rows will be returned.

While on the topic of labels, your query also does not search alternate
labels. What if the term being searched matches on an alternative label?

The below query adds such a feature; it looks also at the altLabel
values for a possible match with the searched term. I had to make the
query return a specific set of variables and added the DISTINCT term to
remove duplicate hits.


SELECT DISTINCT ?subject ?pref_term ?broader ?broaderterm ?scope_note_nl
?comment ?estStart ?estEnd

WHERE {
?subject luc:term "crystol*" .

?subject skos:inScheme <http://vocab.getty.edu/aat/> .

?subject xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm

?pref_term] .
OPTIONAL {?subject xl:altLabel [dct:language gvp_lang:nl;
xl:literalForm ?alt_term]}
FILTER ( REGEX(?pref_term, '^crystol', 'i') || REGEX(?alt_term,
'^crystol', 'i') ) .

OPTIONAL {?subject gvp:broaderPreferred ?broader .
?broader xl:prefLabel [dct:language gvp_lang:nl;
xl:literalForm ?broaderterm]}
OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl;
rdf:value ?scope_note_nl]}
OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
rdfs:comment ?comment]}
OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
gvp:estStart ?estStart]}
OPTIONAL {?subject xl:prefLabel [dct:language gvp_lang:nl;
gvp:estEnd ?estEnd]}
}

ORDER BY ?pref_term

LIMIT 12


HTH,
Kevin

> --
> You received this message because you are subscribed to the Google
> Groups "Getty Vocabularies as Linked Open Data" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to gettyvocablo...@googlegroups.com
> <mailto:gettyvocablo...@googlegroups.com>.
> To post to this group, send email to gettyv...@googlegroups.com
> <mailto:gettyv...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/gettyvocablod/dec3ef1e-9304-47c5-b255-67bc6567e8b9%40googlegroups.com
> <https://groups.google.com/d/msgid/gettyvocablod/dec3ef1e-9304-47c5-b255-67bc6567e8b9%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Rolf Blijleven]

Hi Kevin, 

many thanks for your comments, I'll do some more homework & report back. That may take a while though. 

Meanwhile, I remain open to suggestions. 

Cheers,

Rolf

Op maandag 26 februari 2018 22:19:17 UTC+1 schreef Kevin Ford:

> <mailto:gettyvocablod+unsub...@googlegroups.com>.

[vladimir...@ontotext.com]

Hi Rolf!

1. As Kevin suggested, use "skos:inScheme aat: " to be more precise.

2. Please read http://vocab.getty.edu/doc/queries/#Full_Text_Search_Query and the following sections. Eg if the user enters "arts and crafts" you should remove "and".

3. Your query corresponds to http://vocab.getty.edu/doc/queries/#Combination_Full-Text_and_Exact_String_Match. I don't see why the FILTER should be slow, if the FTS returns only a small result set.

4. But also consider http://vocab.getty.edu/doc/queries/#Exact-Match_Full_Text_Search_Query which uses FTS "phrase" syntax for more precisision (eg '  "crystol*" '). Note: you can't anchor to the beginning of the string in the FTS query.

5. Your query traverses xl:prefLabel several times. This is a bit inefficient, but more importantly it assumes there's one NL prefLabel (else it'll mix the characteristics of the different prefLabels together). SKOS recommends one prefLabel per language but that's in fact not true in AAT: there are 46 cases (en, de, fr):

select * {
  ?x skos:inScheme aat: ; xl:prefLabel ?l1, ?l2.
  ?l1 dct:language ?lang.
  ?l2 dct:language ?lang.
  filter(str(?l1) < str(?l2))
}

[Rolf Blijleven]

Hi Kevin, 

Many thanks, your points 1, 2 and 3 are accepted & adopted. 

As for your other suggestion, you said 

While on the topic of labels, your query also does not search alternate
labels.  What if the term being searched matches on an alternative label?  

Good point. Thinking out loud here: Adlib doesn't support the distinction between preferred and alternative, but it's easy to modify it so it does. However, each thesaurus term has its own record. So when the user chooses a term, one term record will be created (plus one record for its broader). So let's say this is part of one line in the result set: 

pref_term alt_term broader

==============  =============   =============

crystoleums@nl crystoleum@nl aat:300137294

We'll have to present that to the user as: 

term            term_type       broader

==============  =============   =============

crystoleums@nl  PrefLabel       aat:300137294

crystoleum@nl   AltLabel        aat:300137294

And then the user can select one. (If there are more Alts to one particular Pref, it should show the one Pref and all its Alts below it). 

This is certainly feasible, although it does require more processing in the interface. So for now, I think I'll stick to Prefs in version 0.1, let some people try that out, and add Alts to version 0.2 (and modify their Adlib) if they want. 

One remark though: Shouldn't DISTINCT be avoided if possible? I think it was Vladimir who said that and I believe it's good practice to never DISTINCT "just to be safe" (from point 4 here).

much obliged, 

Rolf

On to Vladimir's comments

[Rolf Blijleven]

Hi Vladimir, 

many thanks for your comments, see below please. 

Op dinsdag 27 februari 2018 12:41:49 UTC+1 schreef vladimir...@ontotext.com:

Hi Rolf!

1. As Kevin suggested, use "skos:inScheme aat: " to be more precise.

Done! 

 

2. Please read http://vocab.getty.edu/doc/queries/#Full_Text_Search_Query and the following sections. Eg if the user enters "arts and crafts" you should remove "and". 

I see what you mean, however at this point, I'm assuming the user already has some idea of the term they're looking for, e.g. by having an AAT browser at hand. That is, within Adlib, I'm not expecting any elaborate FTS being formulated to go into the query. Yet. Will bear that in mind though. So thanks!  

  

3. Your query corresponds to http://vocab.getty.edu/doc/queries/#Combination_Full-Text_and_Exact_String_Match. I don't see why the FILTER should be slow, if the FTS returns only a small result set. 

This was purely theoretical. Any filter (coffee filter, audio filter, software filter...) will always introduce some lag. If it's totally negligable in this case, all the better!  

4. But also consider http://vocab.getty.edu/doc/queries/#Exact-Match_Full_Text_Search_Query which uses FTS "phrase" syntax for more precisision (eg '  "crystol*" '). Note: you can't anchor to the beginning of the string in the FTS query. 

Adlib users are used to right-truncated searches, ie left-anchoring to a string. And see point 2 also.  

 

5. Your query traverses xl:prefLabel several times. This is a bit inefficient,

Is there a way to traverse fewer times and still get the same result? I don't know sparql that well yet, so if there is, please elaborate. 

 

but more importantly it assumes there's one NL prefLabel (else it'll mix the characteristics of the different prefLabels together). SKOS recommends one prefLabel per language but that's in fact not true in AAT: there are 46 cases (en, de, fr):

select * {
  ?x skos:inScheme aat: ; xl:prefLabel ?l1, ?l2.
  ?l1 dct:language ?lang.
  ?l2 dct:language ?lang.
  filter(str(?l1) < str(?l2))
}

I did indeed assume there's only one prefLabel per language, however, do I understand correctly:
(1) there are only 46 exceptions to the rule? 

(2) this applies to languages only? For example Vanikoro and Vano are two English prefLabels for one and the same language? 

I don't think this is a problem. E.g. search terms "Vano" and "Vani" give different results, but I guess the users will have to live with that. 

Anyway, many thanks again. 

All the best, 

Rolf

 

[vladimir...@ontotext.com]

DISTINCT is a problem for big resultsets, because it requires to sort the resultset in memory. On the other hand, if you can write your query to avoid duplicates, distinct is not needed, right?

Here's how to avoid traversing several times:

    SELECT * WHERE {  

      ?subject luc:term "crystol*"; xl:prefLabel ?label.

      ?label dct:language gvp_lang:nl; xl:literalForm ?term.

      filter regex(?term, '^crystol', 'i')  

      OPTIONAL {?subject gvp:broaderPreferred ?broader .  

                ?broader xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broaderterm]}  

      OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note_nl]}  

      OPTIONAL {?label rdfs:comment ?comment]}  

      OPTIONAL {?label gvp:estStart ?estStart]}  

      OPTIONAL {?label gvp:estEnd ?estEnd]}  

[Rolf Blijleven]

One small update (for anyone interested): in Adlib, the distinction pref/alt fits better in the field term_status. Not term_type. 

Op dinsdag 27 februari 2018 21:37:38 UTC+1 schreef Rolf Blijleven:

[Rolf Blijleven]

Hi Vladimir, 

Neat! Advanced SPARQL, if you ask me :-) You create a help variable label, specify the language for it, and then use that to fill comment, estStart and estEnd in the correct language. Right? 

Here's the latest version, with vocabulary spec:

SELECT * WHERE {  

      ?subject luc:term "crystol*"; xl:prefLabel ?label. 

      ?subject skos:inScheme <http://vocab.getty.edu/aat/> . 

      ?label dct:language gvp_lang:nl; xl:literalForm ?term.

      filter regex(?term, '^crystol', 'i')  

      OPTIONAL {?subject gvp:broaderPreferred ?broader .  

                ?broader xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broaderterm]}  

      OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note]}  

      OPTIONAL {?label rdfs:comment ?comment}  

      OPTIONAL {?label gvp:estStart ?estStart}  

      OPTIONAL {?label gvp:estEnd ?estEnd}  

    } ORDER BY ?term LIMIT 12

Same query in action. (As many spaces removed as possible.) 

And just for fun, the same query for fiets, which is Dutch for bicycle.  These show that it'll probably be more user-friendly to include altLabels in the result set. I'll come back to that. 

So this is it for now. 

Many thanks again for your help! 

Cheers,

Rolf

Op woensdag 28 februari 2018 10:43:54 UTC+1 schreef vladimir...@ontotext.com:

[Rolf Blijleven]

Hi everyone, 

To wrap things up, I decided on two different queries, one queries xl:prefLabel, the other queries xl:altLabel. The user may actively select which term type should be queried. In Adlib, you configure two separate tabs ('advanced external sources' in Adlib-speak), one for prefs, the other for alt. 

I chose this approach because 

(a) the user should be aware of the distinction, 

(b) in Adlib, the window to present hits (that the user must choose from) is rather narrow: you can't show that many properties per term, and 

(c) it's much faster than one query that returns both prefs and alts each with their own set of additionals. And you don't have to unravel and re-format its response. 

- Language and search term (here 'nl' and 'fiets') will be user-specified variables. 

- The query for prefs does not return term type as it only yields Descriptors (right?) 

    SELECT * WHERE { 

        ?subject skos:inScheme <http://vocab.getty.edu/aat/> . 

        ?subject luc:term "fiets*"; xl:prefLabel ?pref_GID.  

        ?pref_GID dct:language gvp_lang:nl; xl:literalForm ?pref_term

        FILTER regex(?pref_term, '^fiets', 'i')

        OPTIONAL {?subject gvp:broaderPreferred ?broader_GID . 

                 ?broader_GID xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broader_term]} 

        OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note]}  

        OPTIONAL {?pref_GID rdfs:comment ?p_comment}  

        OPTIONAL {?pref_GID gvp:estStart ?p_est_start}  

        OPTIONAL {?pref_GID gvp:estEnd ?p_est_end}

    } ORDER BY ?pref_term LIMIT 20

- The query for alts is almost identical, but it does return term type, as AlternateDescriptor and UsedForTerm may map to different term_statuses in Adlib. 

    SELECT * WHERE { 

        ?subject skos:inScheme <http://vocab.getty.edu/aat/> . 

        ?subject luc:term "fiets*"; xl:altLabel ?alt_GID.  

        ?alt_GID dct:language gvp_lang:nl; xl:literalForm ?alt_term; gvp:termType ?alt_term_type

        FILTER regex(?alt_term, '^fiets', 'i')

        OPTIONAL {?subject gvp:broaderPreferred ?broader_GID . 

                 ?broader_GID xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broader_term]} 

        OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note]}  

        OPTIONAL {?alt_GID rdfs:comment ?a_comment}  

        OPTIONAL {?alt_GID gvp:estStart ?a_est_start}  

        OPTIONAL {?alt_GID gvp:estEnd ?a_est_end}

    } ORDER BY ?pref_term LIMIT 20

Lastly, while I intend to continue my own journey with this, Axiell have meanwhile declared they are willing to set-up a web service that provides the necessary functionality, whereby (XML?) response to the SPARQL query is transformed into AdlibXML by means of a (user-modifyable?) stylesheet. This message is the specification. 

Again, please feel free to comment. 

cheers, 

Rolf

Op woensdag 28 februari 2018 12:43:38 UTC+1 schreef Rolf Blijleven:

[Reem Weda]

Hi Rolf, Thank you, and Kevin and Vladimir, for the good work on this. It will make it possible (Dutch) users of the Adlib system, and there are quite a few, to acces the live data from AAT from within their systems!
I'm looking forward to publish this news to the users of the Dutch AAT. Let me know!
Cheers, Reem

[Vladimir Alexiev]

> You create a help variable ?label, specify the language for it, and then use that to fill comment, estStart and estEnd

Precisely. But I'd say this is basic stuff.

It really helps to draw yourself a graph when writing queries (even if only mentally).

• In my query there's a path to ?label then branching off in two directions.
  
• In your query there are two separate paths that you HOPE will follow a common route to "label". But if you want that, better write them to be explicitly the SAME until the node ?label, to also save the effort of reaching that node twice.
  

[Rolf Blijleven]

Hi Vladimir, 

I'm sorry but I'm confused. What do you mean by : 

In your query there are two separate paths that you HOPE will follow a common route to "label". 

My latest version is pretty much copied from yours, the one to avoid several traversals. This is it, slightly revised. In {{ }} will be user variables and ?label renamed to ?GID:   

SELECT * WHERE {  

    ?subject skos:inScheme <http://vocab.getty.edu/aat/> .  

    ?subject luc:term "{{search_term}}*"; xl:{{pref|alt}}Label ?GID.  

    ?GID dct:language gvp_lang:nl; xl:literalForm ?term; gvp:termType ?gvp_term_type

    FILTER regex(?term, '^{{search_term}}', 'i')  

    OPTIONAL {?subject gvp:broaderPreferred ?broader_GID .  

              ?broader_GID xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broader_term]}  

    OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note]}  

    OPTIONAL {?GID rdfs:comment ?comment}  

    OPTIONAL {?GID gvp:estStart ?est_start}  

    OPTIONAL {?GID gvp:estEnd ?est_end}  

} ORDER BY ?term LIMIT 12

path 1 is to subject leading to broader, broader_GID and scope note; they relate to the concept.  

path 2 is to GID connecting to all the other variables, they relate to the alt|prefLabel. 

Or do we mean the same? I hope so :-) 

By the way, may I ask what timezone your are in? 

thanks again, 

Rolf

Op vrijdag 16 maart 2018 10:27:56 UTC+1 schreef Vladimir Alexiev:

[Vladimir Alexiev]

>> In your query there are two separate paths that you HOPE will follow a common route to "label".

> I'm confused. My latest version is pretty much copied from yours

I’m sorry, I meant the previous version of your query.
Cheers!

[Rolf Blijleven]

Reassuring. Cheers to you too!

Op vrijdag 16 maart 2018 13:12:39 UTC+1 schreef Vladimir Alexiev:

[Rolf Blijleven]

New insights: search for both prefLabels and altLabels in one query. Which will then become: 

SELECT * WHERE {

    ?subject skos:inScheme <http://vocab.getty.edu/aat/> .

    ?subject luc:term "fiets*"; xl:prefLabel|xl:altLabel ?Getty_ID.

    ?Getty_ID dct:language gvp_lang:nl; xl:literalForm ?term; gvp:termType ?term_type

    FILTER regex(?term, '^fiets', 'i')

    OPTIONAL {?subject gvp:broaderPreferred ?broader_GID .

             ?broader_GID xl:prefLabel [dct:language gvp_lang:nl; xl:literalForm ?broader_term]}

    OPTIONAL {?subject skos:scopeNote [dct:language gvp_lang:nl; rdf:value ?scope_note]}

    OPTIONAL {?Getty_ID rdfs:comment ?comment}

    OPTIONAL {?Getty_ID gvp:estStart ?est_start}

    OPTIONAL {?Getty_ID gvp:estEnd ?est_end}

} ORDER BY ?term LIMIT 20    

Just in case anyone might be interested. 

[Vladimir Alexiev]

I suddenly wondered whether every term has gvp:termType. Checked with this query, turned out that not:

select * {
  ?x a xl:Label
  filter not exists {?x gvp:termType []}
} limit 100

But reading the description at http://vocab.getty.edu/doc/#Term_Characteristics, I think this is a data bug, will post to the Getty.

https://jira.getty.edu/browse/ITSLOD-540, https://jira.getty.edu/browse/ITSLOD-541.

For the time being, you may want to enclose that part in OPTIONAL as well.

[Rolf Blijleven]

Thanks for the heads up. I'd be surprised if there would be Dutch terms without a termType, but I haven't checked. 

Op donderdag 29 maart 2018 11:46:00 UTC+2 schreef Vladimir Alexiev:

[Vladimir Alexiev]

 

ITSLOD-541 concerns malformed term URLs ending in "-": it is fixed and should appear on next data refresh (within 2 weeks).

ITSLOD-540 (a data issue) affects only 3 terms and will be fixed by the editorial team in due time.