EXTRACT services still maintained?

25 views
Skip to first unread message

Renato Cortinovis

unread,
Feb 12, 2016, 12:11:53 PM2/12/16
to Learning Registry Developers List
Hello everybody,

I am developing a prototype injecting custom rich snippets with embedded discovery functionalities in Google results pages, based on Learning Registry data (I will be pleased to share it as soon as it is ready).

I am currently exploiting your EXTRACT API, but apparently some queries that used to work perfectly seem to behave erratically.  For example,


used to correctly retrieve the ASN standards that resource is aligned to (http://purl.org/ASN/resources/S1143424, http://purl.org/ASN/resources/S114344B…), but I now get an empty list (following a recent restart of the service I think). Yet, the API works properly in other cases.

I have a question: is this API still maintained (the documentation available is a bit old, and I am aware about your plans to migrate from CouchDB to Postgres)?

Are you using these data services in your search engine widget, or are you parsing separately the full documents?

I look forward to your reply!

Thank you,
Renato

Steve Midgley

unread,
Feb 12, 2016, 12:40:21 PM2/12/16
to learnin...@googlegroups.com
Hi Renato,

I'm not familiar with that data service, but I know that Joe uses a data service to operate the "search widget" system, so I believe they are still being maintained generally. But I'm sorry I don't have info on that specific one (who created it for example).

Things that used to work should still work - the effort to migrate to Pg has not had any impact on the current system yet.

The new system, which will operate side-by-side to the current system for a time, will eliminate a lot of these types of interfaces. But I hope we will replace them with some more generic and easy to use ones (and more reliable too).

Hopefully Joe will have some input on what's wrong with this specific data service - it is probably something specific and technical, rather than a deprecation..

If you can share your requirements for the service: what data you want out of the system, etc, that would be helpful so we can make sure the new search API will accommodate it..

Steve


--
You received this message because you are subscribed to the Google Groups "Learning Registry Developers List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to learningreg-d...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jim Klo

unread,
Feb 12, 2016, 1:27:25 PM2/12/16
to Learning Registry Developers, rmcort...@gmail.com
Hi Renato,


They were never part of the official API, but more of a simplified way for node owners to create and host dedicated custom slice-like apis. I think these examples were installed on all the node01/node02 servers. (Steve, recall this is one of those custom data service samples I built a long time ago after the Boulder conference)

I’m not sure how much these have been maintained in the current versions. Joe would have to be the once to comment, since I no longer maintain the project.  Likely there is just some kind of encoding issue that is causing the indexer to fail or if the node had needed reindexing, those API’s had not been reindexed yet - which wouldn’t happen until someone makes are get request.


Jim Klo
Senior Software Engineer
Center for Software Engineering
SRI International
t. @nsomnac
p. 650.859.6328 (new)

Pat L (pgogy)

unread,
Feb 12, 2016, 4:04:44 PM2/12/16
to learnin...@googlegroups.com
You may find my ye olde chrome extension handy  (as it sounds similar)

joe hobson

unread,
Feb 12, 2016, 5:29:35 PM2/12/16
to learnin...@googlegroups.com
hey Renato,

It’s funny because I just had a thought about that data services (extract) tool yesterday. Jim Klo did a great job setting up the code and we built a tool on top of it. The original code was a NodeJS app that gave the user a browsable set of standards and resources from the LR that are aligned to those standards. You can get the code on github and it probably still works - https://github.com/navnorth/LRnodejs

The only issue you’re really having is that you’re remembering the data service wrong — It doesn’t give you the standards that are aligned to a resource, it gives you the resources aligned to a standard. So your discriminator should be an ASN url. For example: http://node01.public.learningregistry.net/extract/standards-alignment-related/resource-by-discriminator?discriminator=http://purl.org/ASN/resources/S1013C40

And then of course I went looking for the code and found that you were actually on the right track. I forgot there was a flip-side that allowed you to feed in a resource and get standards alignments back. Here’s how you get the standards for a resource: http://node01.public.learningregistry.net/extract/standards-alignment-related/discriminator-by-resource?resource=http://illuminations.nctm.org/WebResourceReview.aspx?ID=2079

Looking at the LR records for your sample resource, I think i see the problem… the metadata records associated with that resource are in LRMI and the dataservice was built before LRMI was in use. It only supports nsdl_dc and LRParadata records (you can see the code in lib/alignment.js). But, not a huge problem (in theory). I haven’t worked with these dataservices in awhile, but it seems like it would be worthwhile to update. Would be even better if it handled CCSS urls and dot-notation IDs as discriminators as well.  

I’m pretty swamped getting other LR stuff updated in prep for the GoOpen Exchange at the end of the month. If you (or anyone else out there) wants to take on the code update, I’d be happy to push it to the live node when it’s ready. Or you can cross your fingers that I’ll find an extra couple of hours to get to it.

Sorry I can’t offer an immediate fix, but at least we know what the issue is. Let me know if there’s anything else I can do to help in the short term. … .joe


-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:-:
joe hobson
   Navigation North Learning




Jim Klo

unread,
Feb 13, 2016, 12:20:22 AM2/13/16
to learnin...@googlegroups.com
Piggybacking off of Pat... I had built a similar tool to what you're trying.


It's also in the Chrome store. 


Jim Klo
Senior Software Engineer
SRI International
t: @nsomnac

Renato Cortinovis

unread,
Feb 14, 2016, 3:03:06 PM2/14/16
to Learning Registry Developers List

Steve, Jim, Pat, and Joe,


Thank you very much for your extended and much appreciated feedback! Some comments here (in a logical sequence).

 

Pat and Jim:


I was well aware of your brilliant Chrome extensions. My objective is to extend the idea, by providing discovery “functionalities” (in addition to meta or para “data”) from the rich snippets, that is Queries By Examples starting from the specific learning resource. This requires me to go back and forth from the LR, hence my need for efficient filtering data services, like those provided by EXTRACT.

 

Jim:


I built this set of data services some time ago […]  They were never part of the official API, but more of a simplified way for node owners to create and host dedicated custom slice-like apis.


You developed them as an example of a general design pattern for a layer of services on top of core LR services. But your “example” is actually offering a very useful set of services for developers close to final users (without the need to be a node owner) – and I am really happy to learn that there is every intention to maintain (and possibly extend) them.

 

Joe:


And then of course I went looking for the code and found that you were actually on the right track. I forgot there was a flip-side that allowed you to feed in a resource and get standards alignments back. Here’s how you get the standards for a resource: http://node01.public.learningregistry.net/extract/standards-alignment-related/discriminator-by-resource?resource=http://illuminations.nctm.org/WebResourceReview.aspx?ID=2079


Indeed: in my example I was using “discriminator-by-resource” in this way.  But I was alarmed because the *same* call that used to produce correct results a few days ago (I am 100% sure), stopped producing results after the LR node01 was down for a little while. The problem might not be due to pre-LRMI development, but perhaps to reindexing as suggested by Jim (plus caching, I suspect). It’s fine for the moment - I’ll let you know if I spot additional erratic behaviour.


 I haven’t worked with these dataservices in a while, but it seems like it would be worthwhile to update. Would be even better if it handled CCSS urls and dot-notation IDs as discriminators as well. 


Yes, definitely - that would be very useful! Some comments about this below in Steve’s section.


Or you can cross your fingers that I’ll find an extra couple of hours to get to it.


Fingers crossed then!

 

Steve:


 If you can share your requirements for the service: what data you want out of the system, etc, that would be helpful so we can make sure the new search API will accommodate it.

 

My perspective is as an edge node consumer agent (while I do understand your core focus is on the distribution infrastructure).


My personal wish list would certainly start with Joe’s suggestion, that is to extend the existing EXTRACT service to cover additional standards beyond ASN, while covering common payload schemas:

resources / ID_only / COUNT_only - associated to a given standard  [and vice-versa, to make it more efficient than getting the whole resource data and parse them]

While not essential, COUNT_only would be useful to quickly provide “volume indicators” to final users.


This can be further extended to other “filtering” capabilities on basic metadata as provided by other learning resources portals. While tag and author can be currently handled via the SLICE API, additional ones could include topic / audience / age / resource_type… (but I am well aware of the complexity arising from your schema agnostic strategy).

 

Thank you all again for your precious feedback,

Renato





Reply all
Reply to author
Forward
0 new messages