ROR API affiliation search issue

[John Tsai]

Hi,

We are trying to integrate ROR API affiliation search with our customer affiliation matching processes. One issue we are currently running into is getting inconsistant result as the UI search. For example when searching "Eureka Aerospace Corporation" on https://ror.org/, "Eureka Aerospace (United States)" (https://ror.org/00q9x4d93) is returned as the top organization. However, when searching the same organization against ROR API (https://api.ror.org/v2/organizations?affiliation=Eureka%20Aerospace%20Corporation), "The Aerospace Corporation" (https://ror.org/01ar9e455) is returned as the top match with `"chosen": true`. We tried falling back to the default "multi_search" by removing "single_search" from the request parameters. But still the same org https://ror.org/01ar9e455 was returned at the top, with none of the organizations returned being marked as `"chosen": true`. 
It would be greatly appretiated if anyone has insight to share on why the difference in behavior between UI search and API search, and how we can leverge the API to get better match result.

Thanks,

John  pace Corporation.

[Amanda French]

Hi John,

Thanks for writing in! Good to hear that you're incorporating ROR. The ROR UI search uses the ?query parameter of the ROR API, not the ?affiliation parameter. See these pages in our documentation to learn more about the difference: 

- https://ror.readme.io/docs/web-search 
- https://ror.readme.io/docs/api-query 

- https://ror.readme.io/docs/api-affiliation 

Basically, the ?query parameter, which runs the ROR web search UI, expects input that is a simple search for the name of an organization, while the ?affiliation parameter expects long and messy input strings that include extra information like locations, street addresses, punctuation, and organization subunits. If most of the affiliations you're getting from your customers are organization names like "Eureka Aerospace Corporation", you might get better results by using the ?query parameter of the ROR API.  

Does that help? 

Cheers,

Amanda

[Amanda French]

I should add that using the ?query parameter does require a human in the loop who can choose the desired result, so that might not suit your purposes if you're trying to do automated matching of text strings to ROR IDs. Is that what you're building? But in any case, I hope it's clear why the UI search produces a different list of results than the affiliation matching service does. 

[John Tsai]

Hi Amanda,

Thanks for the clarification. Yes, we are trying to automate the matching process and using the ?query parameter won't work for us. 

Do you think it would be possible for ROR to looking into the matching algorithm used by the ?affiliation single search and perhaps find ways to improve the result? I could see why the ?affiliation search return "The Aerospace Corporation" when searching  "Eureka Aerospace Corporation". But it seems the search returns "The Tech Interactive" https://ror.org/009rbsv7 whever the search term contains the word "Tech" with score:1 and chosen:true. For example 
https://api.ror.org/v2/organizations?affiliation=Something%20Tech&single_search
https://api.ror.org/v2/organizations?affiliation=Tech%20Anything&single_searchThe Aerospace Corporation

[Adam Buttrick]

Hi John,

In this case, an overly broad (but valid) alias "The Tech" on https://ror.org/009rbsv73 is causing false positive matches. Since this variant name appears to have no meaningful affiliation usage or mention in funding statements, we will drop it from the ROR record in our next release. Similarly, for https://ror.org/00q9x4d93, we'll add the aliases you mentioned, which should resolve the matching issue. 

In most cases, these kinds of affiliation matching issues indicate that the matched false positive record requires an update. If you encounter this, please submit a curation request (https://curation-request.ror.org) to correct or reach out to us at sup...@ror.org if it is unclear what should be changed.

Thanks,

Adam

--
To email the members of this group, email ror-...@ror.org.
ROR Code of Conduct: https://ror.org/community/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "ROR Technical Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ror-tech+u...@ror.org.
To view this discussion visit https://groups.google.com/a/ror.org/d/msgid/ror-tech/f1d08acd-605d-4db1-8f2c-b7e37719ea76n%40ror.org.

--

Adam Buttrick

Product Manager, ROR

ad...@ror.org | https://ror.org | @ResearchOrgs

[Tsai, John]

Hi Adam,

This for your explanation and helpful instructions. I will use those channels to reach out in the future.

 

One more question, using Eureka Aerospace as example.

 

If search using “Eureka Aerospace”, the result listed 7 organizations, with “BlackSky Global (United States)” at the top. “Eureka Aerospace (United States)” does not show up in the search result at all. I see that BlackSky Global has an alias “BlackSky Aerospace” so perhaps that is why it’s returned. The rest of the result matches on either “Eureka” or “Aerospace” in name or aliases. So that begs the question why “Eureka Aerospace (United States)” isn’t returned at all? I get that maybe this too will be resolved after you add “Eureka Aerospace Corporation” to the alias. I’m really just trying to understand the search behavior a bit more, to help us assess if this integration will work for us.

 

Thanks,

John

[Adam Buttrick]

Hi John,

Adding a shorter alias without the country suffix (i.e. just "Eureka Aerospace") clears a length cutoff used in the single search strategy. Specifically, its logic includes a length filter that skips any candidate name where len(name) >= len(input) + 4 before match scoring. Your example input "Eureka Aerospace" is 16 characters and the only name on record is "Eureka Aerospace (United States)" at 32 characters, so the "(United States) suffix pushes it past this cutoff and the name is dropped without being compared. The length filter itself is a reasonable choice in all of our testing, because without it a much-longer candidate name would trivially contain a short input as a substring, producing inflated similarity scores for non-specific matches. Here, it just interacts badly with the (Country) disambiguation convention on the long tail of sparse company records. We can look at some form of bulk update to these records to mitigate.

Otherwise, the matching code is open source in the ROR API repo if you want to review directly. In terms of evaluation, this matching strategy achieves 97% precision and 73% recall against a manually-annotated, randomly sampled dataset derived from all affiliation strings in Crossref. If you'd like more details on how we design and use these evals to assess whether a given matching strategy is fit for purpose, you can see our blog posts here:

https://ror.org/blog/2024-08-28-the-myth-of-perfect-metadata-matching/

https://ror.org/blog/2024-11-06-how-good-is-your-matching/

Thanks,
Adam

[Tsai, John]

Wow. Thank you for the very helpful insight and explanation of the matching logic.