confusion re: jobs url endpoints wrt bibliometrics and funding, missing funding section

[sn...@bsd.uchicago.edu]

We have an older implementation of ProfilesRNS (2.9) that we have upgraded to 3.1.  Our DBA spent some time working on the jobs and is confused about how the variables in the packages are being used.

 

According to the DBA, the GET PMIDS step (Get API Post Data) of the PRNSWebService package uses the URL for the job as specified in [Profile.Import].[PRNSWebservice.Options].  And the Call PRNS Webservice step uses the variable PRNSWebService.  The variable Job was defined in the script that created the ExporterDisambiguation_GetFunding job, but Call PRNS Webservice was not overridden.  The original values for the package were for the ProfilesRNS_GetBibliometrics job.  In the package, PRNSWebService was hard coded to http://profiles.catalyst.harvard.edu/services/GetPMIDs/GetBibliometricData2.asp and HMSPubService was hard-coded to http://profiles.catalyst.harvard.edu/services/GetPMIDs/GetBibliometricData.asp. The PRNSWebService URL (with GetBibliometricData2) did not see to have an API behind it.  The DBA has set the PRNSWebService in the package as show below for the jobs.  That is, the variable PRNSWebService  was set to the same value found in the table [Profile.Import].[PRNSWebservice.Options]. 

 

job

url

bibliometrics

http://profiles.catalyst.harvard.edu/services/GetPMIDs/GetBibliometricData.asp

Funding

http://profiles.catalyst.harvard.edu/services/NIHGrants/FindFunding.asp

geocode

https://maps.googleapis.com/maps/api/geocode/xml?address=

GetPubMedXML

http://profiles.catalyst.harvard.edu/services/GetPMIDs/GetPubMedXML.asp

GetPubMedXML_All

http://profiles.catalyst.harvard.edu/services/GetPMIDs/GetPubMedXML.asp

PubMedDisambiguation_GetPubs

http://profiles.catalyst.harvard.edu/services/GetPMIDs/default.asp

 

The DBA is reporting that the jobs are functioning in the sense that they are not erroring and some data is getting inserted.  But the DBA is a bit apprehensive for the GetFunding and Bibliometrics job.  (The other jobs were defined previously and the DBA thinks their values are likely correct.)

 

Also, we do not understand why records have been inserted into [Profile.Data].[Funding.DisambiguationResults] but nothing appears for the  Research Activities and Funding section of a profile that has records in the table. 

 

There could be something wrong with the configuration of our staging profiles web server or some functionality we need to enable in 3.1.  Manually adding funding in our existing production Profiles (version 2.9) causes a  Research Activities and Funding  section to be created, but  the same test in the staging Profiles (upgraded to 3.1) does not produce a section.  So the job may not be the direct cause of the missing  Research Activities and Funding section. Once we have downgraded our staging Profiles environment, we can try the test in our staging Profiles environment again. 

 

Meanwhile, if someone can confirm what the values for the URLs specified in PRNSWebService should be, it would be helpful.  Also, if anyone has insight into why the Research Activities and Funding section is not displaying, could you let us know? 

 

There is no property group for Research Activities and Funding in [Ontology.].[PropertyGroup],although there is a Research property group label.  We think some of the advice in the groups might be related to older versions of Profiles.

 

The DBA did not import the packages into the database but deployed them to an SSIS catalog. Our previous jobs executed using the SSIS catalog and the functionality should not be impacted by using the catalog instead of the database to store the package.  However, it should be noted in case there is any configuration internal to the msdb database that would be helpful.

[Brown, Nicholas William]

In version 3.0.0 we added a general SSIS package for multiple jobs, but still kept the existing packages for the existing jobs. In 3.1.0 we updated the new package so that we could use it for all jobs. You need to use the 3.1.0 version of the package with 3.1.0, the 3.0.0 version will fail.

 

The main reason we transitioned to a single SSIS package for all jobs was that maintaining multiple SSIS packages for lots of versions of SQL Server was taking too much time to be feasible for our team. As we were adding additional APIs we needed to limit the number of packages so we could continue to maintain them.

 

I confirmed the URLs you have listed for each of the APIs and they are all correct.

 

 

For the funding issue, try running:

 

Exec [Profile.Data].[Funding.LoadDisambiguationResults]

And then the nightly jobs.

 

This should add the funding data to your site. If it works, you should add the [Profile.Data].[Funding.LoadDisambiguationResults] stored procedure as a step in the funding job after funding disambiguation is complete. I think this is missing for some reason.

 

Nick

--
You received this message because you are subscribed to the Google Groups "ProfilesRNS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to profilesrns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/profilesrns/75971338-4ed0-4525-b75f-3b702fc87660n%40googlegroups.com.