ErgoAI for Databricks and Delta Table

[Danyil Butkovskyi]

Hello team. I've been working on POC for the project that requires rules processing for big data. Data is sitting in Delta Lake. 
What would be the best way to load facts from delta table, considering the fact those records are huge(original JSON files have thousands keys/each object)?
What are the best practices on creating rules from big regulation files? IS there any tutorials on creating rules in general? 
Thank you!

[Michael Kifer]

On 6/16/23 2:24 PM, Danyil Butkovskyi wrote:

Hello team. I've been working on POC for the project that requires rules processing for big data. Data is sitting in Delta Lake. 
What would be the best way to load facts from delta table, considering the fact those records are huge(original JSON files have thousands keys/each object)?

I would start with Ergo's Json interface to read in the Json term and convert it into a bunch of queriable objects.

What are the best practices on creating rules from big regulation files? IS there any tutorials on creating rules in general?

Are these rules already encoded in Json? If so, after the above conversion, use the created objects to reassemble the rules in the valid Ergo syntax.

If the rules need to be developed from scratch, then it is art. Analyze the problem, decide on the schema, what is given, what needs to be derived, etc.

We have a tutorial explaining the process using a real life example:

https://sites.google.com/a/coherentknowledge.com/tutorial-capturing-real-world-knowledge

--

       --- michael

[Danyil Butkovskyi]

            I would start with Ergo's Json interface to read in the Json term and convert it into a bunch of queriable objects.

Is there any guide/overview on how to do that? Or what part of ergo manual v3 covers it? I read it, but must be blind not seeing part about reading data in. 
Thank you,
Dan.

[Michael Kifer]

It's in the Guide to ErgoAI Packages, Ch 11.

Also, the Examples Bank includes a fairly extensive example of using Json in ErgoAI.

https://github.com/ErgoAI

--

       --- michael


 

[Danyil Butkovskyi]

Hey there, thanks a lot for the help, I am getting better at it, slowly, but surely! I think I understood what I am trying to do, but please let me know if I am wrong here.
We have json objects sitting in Database, in order to queri them I need to convert them into facts. I do not have rules created, so I will need to create them directly in ergo file. Once I have those I'll be able to query those facts(objects from db) and return results in json format. So workflow looks like: Get json from db -> json to facts -> run rules(queries) -> return json output into db. 

Also, have anyone tried to run ergoAI in databricks? It seems seems like a ood idea for big data processing, but was wondering if there is any known disadvantages/challenges I should be aware of.

Thank a lot, I really appreciate your time! 

On Friday, June 16, 2023 at 4:04:20 PM UTC-4 Michael Kifer wrote:

[Michael Kifer]

I haven't heard that somebody tried to use ErgoAI with databricks, but it seems like a useful thing,