Mapping Azure Resources to EDG Ontologies

[Tim Smith]

Hi,

I am going to model some of my Azure environments using EDG.  The first task is to determine what classes are required to represent the Azure resources.  Below is a small list of available Azure resources.  There are hundreds more.

The hardest part of this effort will be to sort through the EDG ontologies to find either the class that best semantically represents the Azure resource or which class to extend to capture Azure specific details.

Given how common cloud resources are, has their been any work done to map these environments to EDG to facilitate using EDG to govern cloud environments? I'm not looking for an import from the cloud capability (although that would be fantastic!), just a resource to EDG ontology mapping.  I'm sure I'm not the first person wanting to do this.

Thanks,

Tim

Azure Resource
AKS (Kubernetes)
AKS (Kubernetes) Pod
Analysis Services
App Insights
App Service Plan
Automation Account
Azure Active Directory
Cognitive Services
Container Instance
Container Registry
Cosmos DB
Databricks
Disk
Event Grid Topic
Event Hub
Function App
Key Vault
Load Balancer
Log Analytics Workspace
Logic App
Machine Learning Service
Maps
MySQL Server
Network Security Group
Notification Hub
PostgresSQL Database
Power BI
Power BI Gateway
Private DNS
Private Endpoint
Private Link Service
QnA Maker
Redis Cache
Resouce Group
Secret
Service Principal
SignalR
SQL Data Warehouse
SQL Database
SQL Database Managed Instance
Storage Account
Stream Analytics
Subscription
Synapse Analytics
Tags
Tenant
Virtual Network
VMs
Web App
Web App PaaS Blueprint

[Fan Li]

Hi Tim, I have similar needs to model AWS resources. Additionally I would like to model Office365 services (e.g. SharePoint lists / document libraries, MS Teams). 

[Ralph TQ [Gmail]]

Tim,

Currently EDG has some coverage of ETL and Infrastructure/Cloud resources. The intent has been to consider ETL blocks as “black boxes”  - that is, not to model the details pf what metadata repositories are already modeling but to depict how these building blocks play in the larger ecosystem.

Here is an Azure example, a pipeline:

edg:AzureDatabricksPipeline

a edg:AssetClass ;

edg:acronym "ADBPL" ;

rdfs:comment "An 'Azure Databricks Pipeline' is a pipeline that is based on Databricks running on Microsoft Azure." ;

rdfs:isDefinedBy <http://edg.topbraid.solutions/1.0/schema/technical-assets/ETL> ;

rdfs:label "Azure Databricks Pipeline" ;

rdfs:subClassOf edg:ETLpipeline ;

.

This is defined in the ETL schema (as stated by the refs:isDefinedBy statement)

Diving deeper into other ETL and Infrastructure components can be done. I would recommend extending EDG’s classes for functional components. There are 3 schemas to look into:

1. SCHEMA_EDG-technical-assets-v1.0.ttl
2. SCHEMA_EDG-technical-assets-ETL-v1.0.ttl
3. SCHEMA_EDG-technical-assets-Infrastructure-v1.0.ttl

Ralph

-- 
You received this message because you are subscribed to the Google Groups "TopBraid Suite Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to topbraid-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/topbraid-users/ae908d28-4925-4127-9711-fdfef4b6b5dbn%40googlegroups.com.

[Tim Smith]

Hi Ralph,

Thank you for pointing me to those schemas.

Of course, this is the exact issue I was hoping to avoid.  For example, in the three graphs you mentioned, there are 261 definitions, 107 (41%) of which do not have any comments explaining their intended meaning or use (see query below).  Of the 261 definitions, roughly 120 are classes.

Now my task must be understanding the meaning and intended use of all of the classes and determining what Azure (or GCP or AWS) resource most closely aligns so that I can extend or use the correct class, otherwise, I won't be able to use EDG functionality like lineage, etc...  If I didn't care about integrating with the EDG ontologies, it would be easier to just make my own ontologies to support Azure, GCP & AWS.  I can export a json file from Azure that effectively contains instance data from which I can derive an ontology - which also means I can auto-populate EDG with the Azure structure.

This type of mapping needs to be done only once.  Maybe this is an opportunity for TQ to create another asset collection type and sell it as an add-on?

As a side note, the ETL and Infrastructure schemas you shared are not accessible from the Includes tab.  I had to open them using the Files asset collection to get them to show up in the EDG UI.  (This also entailed unlocking the workspace).  Maybe there is another way that I missed.

Also, I noticed a number of "empty property shapes" on edg:AzureDataBricksPipeline, coming from a superclass. (see attached file).  Walking up the tree shows one undefined property shape at edg:System.  The label on this class does not match the URI - rdfs:label = "Software System".  Five more can be found at edg:EnterpriseEnabler.  Additionally, edg:AssetClass contains four undefined property shapes.  This is only what I found in the cursory look.  A query would be better (or maybe a SHACL rule?) to find all of the "hanging" shapes.  Of course, it may be that the missing shapes are defined graphs that are not imported into these graphs.

Tim

SELECT ?commentflag (COUNT(?commentflag) AS ?numcomments)
WHERE {
  ?s rdfs:isDefinedBy ?defininggraph .
  ?s rdf:type ?stype .
  OPTIONAL { ?s rdfs:comment ?scomment . } .
  FILTER (?defininggraph IN (<http://edg.topbraid.solutions/1.0/schema/technical-assets/infrastructure>, <http://edg.topbraid.solutions/1.0/schema/technical-assets/ETL>, <http://edg.topbraid.solutions/1.0/schema/technical-assets>))
  FILTER (?stype NOT IN (sh:PropertyShape,  sh:PropertyGroup))
  BIND (BOUND(?scomment) AS ?commentflag)
}
GROUP BY ?commentflag