OpenRefine fetching URLs from ULAN very slow

s.o.b...@gmail.com

unread,

Jan 27, 2021, 6:51:49 PM1/27/21

to Getty Vocabularies as Linked Open Data

Hi all,

I know questions about processing speed aren't always so useful to the community, but I want to know if I'm causing the problem I'm experiencing.

I have a project with 11k rows. I've filtered to target 8k for which I'm adding a column by fetching URLs from ULAN for nationality based on ULAN ids. I've done this before without a problem using GREL:

'http://vocab.getty.edu/sparql.json?query=select+(str(?nat)+as+?x)+{ulan:'+value+'+foaf:focus/gvp:nationalityPreferred/gvp:prefLabelGVP/xl:literalForm+?nat}'

However, I've been trying to run this same function for two days and can't get it to complete. The current attempt has been running for 9 hours and is only at 32%. Is there an error in the GREL? Do I need to allocate more memory for this process? Should I use Chrome instead of Firefox?

Thanks!

Sarah

Gregg Garcia

unread,

Jan 27, 2021, 8:32:41 PM1/27/21

to Getty Vocabularies as Linked Open Data

Hi, Sarah.

The query is simple (and correct) and should not take long to execute. You might want to look at the value for "Thottle delay" on the input screen when fetching URLs. The default is 5000 milliseconds which means there is a 5 second delay between each request to the endpoint.

Gregg Garcia

Software Architect

Getty Digital

s.o.b...@gmail.com

unread,

Jan 28, 2021, 12:06:27 PM1/28/21

to Getty Vocabularies as Linked Open Data

Gregg,

Thanks for your reply! I had the throttle delay set to 1 second, as you had mentioned to me before. Today I've switched to running this on my work PC (instead of my home Mac book) and it is performing much quicker, 25% complete within 30 minutes. So it appears it was just a performance issue after all. But I'm glad to know that I'm not running in circles with my own error.