REPLACE INTO and task slot usage

[richarde]

Hi All,

I am trying to drop part of my old streaming telemetry data no longer required after a certain time. I ran this test query:

REPLACE INTO test OVERWRITE WITH SELECT * FROM "ProdTelegraf" WHERE metric_group = 'ifcounters' AND __time < CURRENT_TIMESTAMP - INTERVAL '1' YEAR

And I got this error:

The cluster does not currently have enough available task slots (current usage: 4 of 8) to run this query which is set to use up to 8 tasks. This query will have to wait for task slots to become available before running.

Are you sure you want to run it?

Why does this query require so many task slots? Note I am replacing into a new table as a test. If I replace into the same table will I still need 8x task slots? In my setup the query will never run as I will never have 8  task slots become available.

My setup is:

2x historicals/middle managers (same hw)

Kafka ingestion to 2x partitions

Are my options:

More hardware with bigger storage so that I don't have to expire parts of my old data

More historicals (maybe just temporary) so that I can run the query

Something else?

[Ben Krug]

I don't think the query *needs* 8 slots per se, but it's configured to use 8.  You can adjust maxNumTasks, either in query context, or, in the druid console, in the drop-downs.

MSQ does always need at least 2 (controller and worker).  You could configure to use 2, but it will likely be slower.  (For more, see the docs here.)

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/druid-user/276ad92f-7d91-4ab4-9ffc-363d09f48a50n%40googlegroups.com.

[richarde]

Ah thanks Ben, its right there on the screen on the console. My bad.

[Ben Krug]

no problem - please don't hesitate to ask more questions going forward!  It's nice (to me) when I know how to answer them!  :)

To view this discussion visit https://groups.google.com/d/msgid/druid-user/db7d457e-069d-426a-b2ba-2d65f01b19den%40googlegroups.com.