New cluster on k8 - can only run 1 kafka ingestion job at a time?

[Jay G]

I followed these install steps to create the cluster: https://github.com/datainfrahq/druid-operator/blob/master/docs/getting_started.md

Here is the yaml for the cluster: https://github.com/datainfrahq/druid-operator/blob/master/examples/tiny-cluster.yaml

For some reason, I can only run one kafka supervisor task at a time.  The second task is always stuck as "pending".  

Any idea why?

Here are some logs that may be relevant:

{
  "id": "index_kafka_crypto_bulk_15m_6fa92e3a918604c_iielogil",
  "groupId": "index_kafka_crypto_bulk_15m",
  "type": "index_kafka",
  "createdTime": "2023-09-15T03:43:55.380Z",
  "queueInsertionTime": "1970-01-01T00:00:00.000Z",
  "statusCode": "RUNNING",
  "status": "RUNNING",
  "runnerStatusCode": "PENDING",
  "duration": -1,
  "location": {
    "host": null,
    "port": -1,
    "tlsPort": -1
  },
  "dataSource": "crypto_bulk_15m",
  "errorMsg": null
}

Thanks

[Jay G]

I can't even run an index_parallel job to load a file if a kafka supervisor is running.  I have to stop the kafka supervisor and then the index_parallel job will run.  It seems like I can only run 1 task at a time which is pretty strange.

[Sergio Ferragut]

Hi Jay. The task execution system in Druid depends on your deployment. I believe the tiny cluster configuration does not configure Middle Managers which are the providers of worker slots for parallel job execution.

Instead the tiny cluster is using local execution druid.indexer.runner.type=local.

This means that it can only run one task at a time. For more task parallelism you will need Middle Managers in your deployment.

If you are just testing functionality (not scale) you can also try this project that uses docker compose to deploy locally and contains many examples in jupyter notebooks: https://github.com/implydata/learn-druid

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/816f1371-0052-4ae3-a941-07985220156fn%40googlegroups.com.