Reading and mapping JSON files

[Daniel Lavoie]

Hi,

Does TopQuadrant has any tool allowing us to read JSON files and to map them to produce RDF turtle files?  Can SPARQLMotion do that?

Thanks

Daniel

[Holger Knublauch]

Hi Daniel,

there is a variety of options here.

Obviously, if the file is JSON-LD then it can be opened like any other RDF graph, assuming it ends with .jsonld. JSON-LD has been designed with a notion of Contexts that provide some flexibility as an on-the-fly transform from many JSON tree structures into JSON-based graph structure.

If the file is not JSON-LD and you cannot define a suitable JSON-LD context then you could use our generic JSON importer that can transform any JSON into either a "default" RDF graph or into a graph that is described using SHACL shapes. The SPARQLMotion module to invoke that feature is sml:ConvertJSONToRDF. Here is the documentation comment of that:

---

Takes a JSON object or array (represented as text) and converts it to RDF triples with the same structure. The result graph will only contain the generated triples - the input graph will be ignored and may need to be passed on with a separate sm:next relationship. The graph uses the namespace prefix.

This module operates in two modes. By default it will use the "json" namespace (http://topbraid.org/json#) for properties and create blank nodes of type json:Object. However, if sml:service is set it will look for GraphQL shapes and walk them in parallel to the JSON object tree.

The conversion will start at the JSON root and does a recursive walk through of the JSON objects and arrays. Each JSON object becomes a resource. Each attribute of the JSON object is mapped into a property. In the simple case, it will pick a property from the json namespace, e.g. attribute "firstName" becomes a property json:firstName. In simple mode, the values of those properties depend on the JSON attribute value and arrays are converted to rdf:Lists, JSON objects recursively become new blank nodes. Numbers, booleans and strings become corresponding RDF literals. In GraphQL mode, the shapes define how the mapping is performed and the root node must be a JSON object.

Optionally, the module can bind a new variable pointing at the root object of the new JSON data structure in RDF. This does not work if the provided JSON string is an Array with multiple entries.

---

With the SHACL-based approach, you start by declaring the shape of the target data, e.g. if the RDF data should use xsd:double for a property ex:value and your JSON only contains { value: 4.2 } then the shape would define a sh:property constraint of sh:datatype xsd:double on sh:path ex:value. A side benefit of this design is that you can validate the conformance of your JSON with the shapes. This topic is briefly mentioned in this presentation (around minute 47), but you may want to watch most of the presentation to understand the general design philosophy of the mapping from JSON/GraphQL to RDF/SHACL:

https://www.youtube.com/watch?v=Muj0m8Qrdig

The SHACL-less alternative would be to use the default "json" namespace and then use modules such as sml:ApplyConstruct to produce the target triples in their respective namespaces.

Note that sml:ConvertJSONToRDF can also be called from within SWP scripts, which is sometimes less busy than SPARQLMotion.

Speaking about SWP, there is a number of built-in features to process arbitrary JSON on a more "native" level, and then produce whatever you want with it, see https://uispin.org/ui.html#json

A completely different alternative is to use a 3rd party component or program to produce the RDF/Turtle.

HTH
Holger

--
You received this message because you are subscribed to the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/d98fbd11-a8b3-49a4-bb8c-4350676bc0f3%40googlegroups.com.

[Daniel Lavoie]

Thank you very much Holger for your answer. I have listened to the webminar you have suggested, but most of it is about producing JSON from RDF. However, I want to do the opposite.  I am very interested of learning more about the JSON to RDF conversion using SHACL.  In fact, I would want to:

• Generate the IRIs as we want (its seems that is possible using GRAPH QL templates)
• Map the JSON field names to Property IRIs.
• Validate the JSON fields and values.

Where can I find more information about the JSON to RDF conversion using GraphQL and SHACL?

Regards

Daniel

To unsubscribe from this group and stop receiving emails from it, send an email to topbrai...@googlegroups.com.

[Daniel Lavoie]

And:

• Can we use TBC Maestro for the development?
• Can we implement the result in a triple store? Does it have to be EDG triple store or can we use another one like Stardog?

Thanks again!

Daniel

[Irene Polikoff]

Hi Daniel,

The key piece is to have SHACL Shapes that correspond to the structure of your JSON. They are used to guide the conversion.

If you already have a GraphQL schema for your JSON, TopBraid EDG can create SHACL Shapes from it. 

For example, in TBC, you could use Import:

Technical details behind how this works are here: https://www.topquadrant.com/graphql/graphql-shacl.html

sml:ConvertJSONToRDF described below by Holger, has the following arguments:

• sml:keepRootObject (xsd:boolean): [Optional] In GraphQL mode (sml:service is provided) then the root object is typically just a container derived from the Query. By default, the triples of this root object will not be kept. Set to true to keep these triples in the result graph.
• sm:outputVariable (xsd:string): [Optional] The name of the result variable (defaults to "root") that will contain the root object of the converted JSON code.
• sml:service (graphql:Schema): [Optional] A GraphQL service object providing information on how to map the JSON to RDF using SHACL shapes.
• sml:text (xsd:string): The JSON input text, either a JSON object or array.

If you do not have a GraphQL Schema for your JSON, then you could start by creating a SHACL model that correspond to your JSON. Internally, TopBraid will automatically generate a GraphQL Schema from SHACL that will be used to guide the import and transformation of JSON.

Technical details of how GraphQL Schemas are generated from SHACL are here https://www.topquadrant.com/graphql/shacl-graphql.html. Yes, this is used to query RDF with GraphQL and get JSON out. But it is also used in the other direction, to convert JSON to RDF.

Also see answers below

On Aug 20, 2019, at 1:46 PM, Daniel Lavoie <daniel.j...@gmail.com> wrote:

And:

• Can we use TBC Maestro for the development?

Yes. You can develop the import process using the suggested module/API. You can test the script in TBC. However, TBC is a single user, not networked tool. For doing the scheduled processing (assuming this is not a one-of and your JSON comes in on a regular basis), you will need a server.

• Can we implement the result in a triple store?

Yes, sure

• Does it have to be EDG triple store or can we use another one like Stardog?

The transformation will create RDF from JSON, you can decide what to do with it e.g., write it into EDG’s triple store, export it as file into one of the serialized formats, etc. 

To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/e9dd8639-8b2a-420c-a106-4f54e7d39808%40googlegroups.com.

[Jack Hodges]

I am doing the same thing and want to convert a SPIN Map mapping that I am using with XML and have been asked to use JSON instead.

TBCME has a ConvertXMLToRDF SPARQL Motion module that takes plain (but valid) XML and produces a default RDF graph. I then use a SPIN Map to map from that generic RDF to my target schema. This works great.

I expected the ConvertJSONToRDF module to work the same way; that TBCME would read the JSON, convert it to a generic RDF that would look the same as the previous one (since I converted the XML to JSON to test everything). All I expected to have to do was to swap the ConvertXMLToRDF module with the ConvertJSONToRDF module. If I stop the execution after this module there are a number of instances of the http://topbraid.org/json namespace (which doesn't appear to exist) but the next conversion doesn't work.

I do not want to use GraphQL or SHACL or SWP to do this, just SPARQL Motion, SPIN Map, and the published modules. Please advise.

Jack

[Irene Polikoff]

Hi Jack,

We continuously evolve and improve our technologies based on the feedback from the marketplace and our users. 

SPARQLMotion and SPINMap are cool, but, for one reason or another, they did not get broadly used by our customers. In response to this, we have developed our next generation of technology - it is heavily based on SHACL and includes GraphQL and Active Data Shapes (ADS). To some extent, SWP as well, but we recommend ADS over SWP - it has less of a learning curve and better tooling. SPARQLMotion modules themselves continue to get used, but the scripts do not.

The original e-mail below from 2 years ago describes how the ConvertJSONToRDF module works. This module does not produce the same output as the ConvertXMLToRDF module, so you would need to change your next steps.

I understand that you would prefer to continue to use what you have been using starting with 6 or 7 years ago (or may be even longer). However, our advise for any new work is to chose what is being currently broadly used, developed and supported.

Our current recommendation for converting JSON into RDF is to use ADS scripting: https://www.topquadrant.com/configuring-topbraid-edg-with-javascript/. The other options described in the e-mail below are still applicable, but we believe that ADS will be the most optimal in dealing with transformations that you may need. It was not yet released in the summer of 2019.

ADS scripts can be developed directly in EDG which you, presumably, could launch on the localhost, single user model from TBC - if you kept up to date with the current releases. 7.0 release introduces Files option so that you could even do this in a file. If you are dealing with large scale data, though, we recommend creation of proper EDG ontologies that are separate from the asset collections that contain instance data. As I recall, you are in process of purchasing EDG server license to upgrade your outdate server version of TopBraid Live. Given this, you should be able to readily run these scripts on the server.

You will find development information, examples and videos here http://www.datashapes.org/active/. If you have some experience with JavaScript, it should be easy for you to get started with ADS.

Irene

To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/11500407-0172-4a4b-a8bd-6717ec33fc11n%40googlegroups.com.

[Jack Hodges]

I of course read everything TQ had on the use of SPARQL Motion (and many times over the years) before posting anything, as well as to search the forum archives. To do less would have been disrespectful of your time.

We are up to date on our support agreement but I wanted to move to the end of this project without new learning curves being involved. The project is just too tight on hours for that.

You said "This module does not produce the same output as the ConvertXMLToRDF module, so you would need to change your next steps". Do you have any documentation or are you willing to share what you mean by "change your next steps"? I will reread Holger's original response and try to go there, but I didn't see the answer in his answer.

Jack

[Irene Polikoff]

To answer how ConvertXMLToRDF and ConvertJSONToRDF are the same and different:

Same:

1. Both modules can convert to RDF from the corresponding format

2. Both, can do so with and without a domain model (ontology) provided to them

Different:

1. They convert from the different formats - obviously

2. Correspondingly, what they produce without a domain ontology provided is different:

ConvertXMLToRDF will walk the XML tree and do the following (from the documentation page):

• Each XML element name will be mapped into an OWL class. For example, for the tag <Process> it will create an OWL class Process.
• Each attribute will be mapped into an OWL datatype property. For example, for the tag <Process name="Pizza Order"> it will create a datatype property name.
• Each class generated by the Semantic XML importer will be annotated with a value for sxml:element that points back to the XML element name or the full URI of the element.
• Each datatype property generated by the Semantic XML importer will be annotated with a value for sxml:attribute that points back to the XML attribute name.
• Each element instance will become an instance in the target ontology, e.g. <Process name="Pizza Order"> will become an anonymous instance of Process.
• The parent-child relationships between the XML elements are expressed as values of the properties composite:child from the composite ontology. The ordering of children is kept using composite:index property values.
• Locally defined namespaces of the XML file will be stored using sxml:prefix.

ConvertJSONToRDF will walk the JSON object structure and do the following:

• Each JSON object becomes a resource of type json:Object

• Each attribute of the JSON object is mapped into a property. In the simple case, it will pick a property from the json namespace, e.g. attribute "firstName" becomes a property json:firstName. 
• In simple mode, the values of those properties depend on the JSON attribute value and arrays are converted to rdf:Lists, JSON objects recursively become new blank nodes. 
• Numbers, booleans and strings become corresponding RDF literals.

You can run the module and examine the results e.g., put some JSON into the module, run it and infer results.

For example, if you put the following into the sml:text

{

    "glossary": {

        "title": "example glossary",

"GlossDiv": {

            "title": "S",

"GlossList": {

                "GlossEntry": {

                    "ID": "SGML",

"SortAs": "SGML",

"GlossTerm": "Standard Generalized Markup Language",

"Acronym": "SGML",

"Abbrev": "ISO 8879:1986",

"GlossDef": {

                        "para": "A meta-markup language, used to create markup languages such as DocBook.",

"GlossSeeAlso": ["GML", "XML"]

                    },

"GlossSee": "markup"

                }

            }

        }

    }

}

You will get 6 json:Objects, each a blank node. Something like this:

[

  rdf:type json:Object ;

  json:glossary [

      rdf:type json:Object ;

      json:GlossDiv [

          rdf:type json:Object ;

          json:GlossList [

              rdf:type json:Object ;

              json:GlossEntry [

                  rdf:type json:Object ;

                  json:Abbrev "ISO 8879:1986" ;

                  json:Acronym "SGML" ;

                  json:GlossDef [

                      rdf:type json:Object ;

                      json:GlossSeeAlso (

                          "GML"

                          "XML"

                        ) ;

                      json:para "A meta-markup language, used to create markup languages such as DocBook." ;

                    ] ;

                  json:GlossSee "markup" ;

                  json:GlossTerm "Standard Generalized Markup Language" ;

                  json:ID "SGML" ;

                  json:SortAs "SGML" ;

                ] ;

            ] ;

          json:title "S" ;

        ] ;

      json:title "example glossary" ;

    ] ;

].

3. How these modules work with ontologies and what they require is different.

You already know about ConvertXMLToRDF. You provide it an ontology in RDFS/OWL. The ontology is used to control different aspects of conversion, with semantic annotations, etc.

ConvertJSONToRDF requires that you supply an instance of graphql:Schema class. It will provide information on how to map the JSON to RDF using SHACL shapes. The value of graphql:queryShape is used as root shape for the traversal.

To use this option, you will need to have a model in SHACL that corresponds to the structure of your JSON objects. It can be a separate model just for what is in the JSON objects or a a part of a larger SHACL ontology.

This module is closely connected to our GraphQL capabilities. GraphQL technology pages are available here: https://www.topquadrant.com/technology/graphql/. 

Technical details of how GraphQL Schemas are generated from SHACL are here https://www.topquadrant.com/graphql/shacl-graphql.html. Yes, this is used to query RDF with GraphQL and get JSON out. But it is also used in the other direction, to convert JSON to RDF. 

You would also need to create an instance of the graphql:Schema class and declare which node shapes/classes are part of the schema. If you identify a class/node shape as public class of the scheme, all its subclasses/node shapes are also a part of the scheme.

In short, these are two very different modules. 

To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/4e4a91af-4d2d-4265-933f-3662048a80b1n%40googlegroups.com.