Doubts about "OneTime" Ingestion Task

19 views
Skip to first unread message

Giovanni Orio

unread,
Mar 14, 2022, 7:49:34 AMMar 14
to druid...@googlegroups.com

Dear Druid devs,

 

I am a user of Apache Druid. I already did the “Droid Basics” course provided by Imply. I am using Druid within a research project for boosting and powering our AI algorithms. In the context of this project we are also developing a dynamic pipeline engine. We have an external client that needs to access ingestion specifications and change them if something happens in the pipeline. However, I had a doubt we are using Kafka for streaming ingestion and we are also ingesting data from minIO. In the latter case, as soon as I ingest data a new ingestion task is created. After the task has been successfully executed after a time it disappears from the UI and I really do not know how to retrieve the specification task by using the API. Could you please provide any suggestion/help about this?

 

Cheers,

 

Giovanni Di Orio

 

 

 

Mark Herrera

unread,
Mar 14, 2022, 11:27:11 AMMar 14
to Druid User
Hi Giovanni,

Here's the task APIs doc.

If you'd like another resource to explore, my colleague @Sergio_Ferragut wrote a great blogpost on clustered deployment with k8s and min.io.

Best,

Mark

Sergio Ferragut

unread,
Mar 14, 2022, 8:36:10 PMMar 14
to Druid User
Hi Giovanni,
Have you configured logging storage? You can store the task logs in S3, for example.
Let us know how it goes,

Sergio

Giovanni Orio

unread,
Mar 15, 2022, 5:11:42 AMMar 15
to Druid User
Thank you Mark,

I already passed through the task API methods and my problem is still there... I will try to store the task logs in S3 like @Sergi_Ferragut suggested.

Cheers,

Giovanni

Reply all
Reply to author
Forward
0 new messages