Quick Info on Deploying My_local Service

[Visal Vitharana (Intern)]

Hi Ramith,

Since you're in the process of deploying the my_local service, I wanted to provide you with some important details on how it operates.

Currently, the service goes through a warm-up process where it checks if the necessary files are present, either in the local machine or within the Docker image. If these files are not found, the program automatically downloads and saves the required data files from the GitHub repository that hosts them. This warm-up process can take several hours if the files are missing.

However, if the data files are already available, the server will start in just a few minutes. I’ve run this process locally, and I already have the necessary files. If you’d like, I can share them with you to expedite the deployment on your end.

Please let me know how you’d like to proceed.

Best regards,
Visal.

[Sanjiva Weerawarana]

We have to get that down to a few seconds :).

To unsubscribe from this group and stop receiving emails from it, send an email to dev+uns...@datafoundation.lk.

--

Sanjiva Weerawarana

[Ramith Jayasinghe]

Yes, of course. I got the data files now. let me send an update at the end of the day.

--

Ramith Jayasinghe | Director Engineering | WSO2 Inc.

(w) +94777542851 | Email: ram...@wso2.com

[Ramith Jayasinghe]

Hi All,

Visal and I took a look at the issue, and it seems like the Python service isn’t starting because of some missing dependencies at runtime. I suspect it’s related to what the Python build packs in Choreo are including. Visal promised to raise an issue on this.

Since it’s a blocker, we switched to a Docker build. However, we ran into a Trivy scan failure after the build, and even trying to ignore it didn’t work (might be a bug). Updating the dependency to fix the Trivy failure isn’t an option either, as it would need some code changes.

We’re thinking that loading a large number of data files (currently from GitHub repos) during startup is likely to take time no matter what. So, I plan to bring the data files closer to where the mylocal service runs. We could do this by setting up Nginx in front of the data files and running them in Choreo ([1], [2]). Then, we can expose project-level endpoint URLs for the data sets to my-local-service. I think this should work as a temporary fix until we move the data to a database.

[1] https://github.com/ramith/gig-data/blob/master/Dockerfile

[2] https://github.com/ramith/mylocal-data/blob/main/Dockerfile

will send another update once we figure out how to fix the 

[Ramith Jayasinghe]

will send another update once we figure out how to fix the build issue.

[Malith Jayasinghe]

On Thu, Oct 10, 2024 at 4:41 PM 'Ramith Jayasinghe' via LDF Dev <d...@datafoundation.lk> wrote:

Hi All,

Visal and I took a look at the issue, and it seems like the Python service isn’t starting because of some missing dependencies at runtime. I suspect it’s related to what the Python build packs in Choreo are including. Visal promised to raise an issue on this.

Since it’s a blocker, we switched to a Docker build. However, we ran into a Trivy scan failure after the build, and even trying to ignore it didn’t work (might be a bug).

Seems to be a bug. Let's ask this in the discord and please share the discord issue link here as well.  

[Visal Vitharana (Intern)]

Hi All,

I wanted to share an update regarding the recent issue with ignoring vulnerabilities in Choreo using the .trivyignore file. As you know, we typically use the CVE code to bypass specific vulnerabilities, but this time Choreo didn’t allow it. After consulting with the Choreo team, I tested another vulnerability code GHSA-q5fm-55c2-v6j9 and was able to successfully build the service.

The service is currently in the warm-up phase, which may take some time. Once this process completes, I’ll proceed with testing to ensure everything is functioning properly by making some API calls. I’ll keep you updated as soon as I have more information.

Best regards,
Visal.