Re: Next Steps

[Malith Jayasinghe]

adding @LDF Dev 

On Mon, Nov 18, 2024 at 2:25 PM Vibhatha Abeykoon <vibh...@datafoundation.lk> wrote:

Hi Visal, 

Sounds like a plan. Thank you. 

Best, 

Vibhatha

On Mon, Nov 18, 2024 at 2:22 PM Visal Vitharana (Intern) <vi...@wso2.com> wrote:

Hi Vibhatha,

I'll share the details of the MongoDB and GIG service with you. Unfortunately, I can't share the GIG client details since it hasn't been deployed yet.

I'll post these secrets in the Lanka Data Foundation chat group with chat history off, so they will automatically delete after 24 hours. Please make sure to save them before that.

regards,

Visal.

On Mon, Nov 18, 2024 at 1:16 PM Vibhatha Abeykoon <vibh...@datafoundation.lk> wrote:

Hi Visal and Malith, 

Malith thanks for setting this up and clearing up the high level requirements. 

Thanks a lot for the wonderful informative session, we learned a lot about the GIG deployment, Choreo and the basic workflow used to keep the services running smoothly. Visal explained a great deal of details and shared a lots of details with us. 

I think as the next steps, I will try out the local docker containers and see how it goes. Things to figure out

1. Lagging data set in the org chart

2. Figure out an easy workflow to add new data ( I think we discussed a few ways, such as dump the required document shape directly to the database as the last resort, but try using services as you already have. 

Could we get the URLs to access the existing services? 

1. GIG Services

2 GIG client

3. Mongodb database

Best, 

Vibhatha

[Malith Jayasinghe]

On Mon, Nov 18, 2024 at 1:16 PM Vibhatha Abeykoon <vibh...@datafoundation.lk> wrote:

Hi Visal and Malith, 

Malith thanks for setting this up and clearing up the high level requirements. 

Thanks a lot for the wonderful informative session, we learned a lot about the GIG deployment, Choreo and the basic workflow used to keep the services running smoothly. Visal explained a great deal of details and shared a lots of details with us. 

I think as the next steps, I will try out the local docker containers and see how it goes. Things to figure out

1. Lagging data set in the org chart

2. Figure out an easy workflow to add new data ( I think we discussed a few ways, such as dump the required document shape directly to the database as the last resort, but try using services as you already have. 

@Visal Vitharana (Intern) said that there is a script that reads the CSV (with orgchart data) and inserts the data into MongoDB. Did  you try out that script? 

[Vibhatha Abeykoon]

We tried that script. It works well. We should be able to use that to directly inject data. 

And also, we are trying to see if we can do it directly using the GIG Script API. 

[Malith Jayasinghe]

OK.

What is the difference between the two? We need to get these terms corrected... "GIG Script API*? 

[Vibhatha Abeykoon]

Right, we need to clarify it in terms of usage. 

AFAIU, the GIG Script includes various scripts to update the database, and there is an OrgChart module in this GIG Script. It contains the extracted ministry and department format data in CSV files. And, it uses GIG-SDK another library that makes a POST request to save the data in the database we provide. It is a matter of abstraction I would say, what it does is write the data in the CSV by converting it to the entity format defined in the data structure (attributes + core metadata). IMHO that's a bit of a syntactic sugar on encapsulating the calls to the DB via the REST API, and the user only provides the auth details to the services. 

In theory, if we have the data (JSON object in the expected format), we should be able to save it directly to the database. I think that's what the other script is doing. 

@Visal Vitharana (Intern) @Zaeema Nashath please correct me if I'm wrong. 

[Umayanga Gunawardhana]

Hello everyone,

Let me provide some clarity on this topic. The GIG-SDK serves as a foundational framework and set of building blocks designed to help developers create scripts efficiently by leveraging reusable, generalized libraries commonly required for interacting with the GIG-API. An example of this in action is GIG-Scripts, which demonstrates how the GIG-SDK can be utilized. The primary goal is to enable open-source contributors to develop additional crawlers or integrate new data into the GIG-API with minimal effort, reducing code duplication and avoiding the need to start from scratch.

Thanks.

Umayanga 

--
You received this message because you are subscribed to the Google Groups "LDF Dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@datafoundation.lk.
To view this discussion visit https://groups.google.com/a/datafoundation.lk/d/msgid/dev/CAGdkeKKoDzkLKj5orz%3DNMCvBruUwknmjxzs7N6%3DVsx9ZUmHYCg%40mail.gmail.com.