Doubts about "OneTime" Ingestion Task

[Giovanni Orio]

Dear Druid devs,

 

I am a user of Apache Druid. I already did the “Droid Basics” course provided by Imply. I am using Druid within a research project for boosting and powering our AI algorithms. In the context of this project we are also developing a dynamic pipeline engine. We have an external client that needs to access ingestion specifications and change them if something happens in the pipeline. However, I had a doubt we are using Kafka for streaming ingestion and we are also ingesting data from minIO. In the latter case, as soon as I ingest data a new ingestion task is created. After the task has been successfully executed after a time it disappears from the UI and I really do not know how to retrieve the specification task by using the API. Could you please provide any suggestion/help about this?

 

Cheers,

 

Giovanni Di Orio

 

 

 

[Mark Herrera]

Hi Giovanni,

Here's the task APIs doc.

If you'd like another resource to explore, my colleague @Sergio_Ferragut wrote a great blogpost on clustered deployment with k8s and min.io.

Best,

Mark

[Sergio Ferragut]

Hi Giovanni,

Have you configured logging storage? You can store the task logs in S3, for example.

https://druid.apache.org/docs/latest/configuration/index.html#log-long-term-storage

Let us know how it goes,

Sergio

[Giovanni Orio]

Thank you Mark,

I already passed through the task API methods and my problem is still there... I will try to store the task logs in S3 like @Sergi_Ferragut suggested.

Cheers,

Giovanni