How to specify hadoop job queue name for HadoopDruidIndexer?

[hmx...@gmail.com]

I am using HadoopDruidIndexer to do batch ingestion but couldn't figure out a way to specify the hadoop job queue. I tried to add -Dmapred.job.queue.name=...

Still my job failed with error because it goes to default qeueue

Caused by: java.io.IOException: Failed to run job : Application appattempt_1400781790269_6844295_000001 submitted by user druid to unknown queue: default

What is the correct way to specify the queue name?

Thanks!

java -Xmx256m -Dmapred.job.queue.name=hourly_q1 -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:<hadoop_config_path> io.druid.cli.Main index hadoop <config_file>

[Gian Merlino]

Try -Dhadoop.mapred.job.queue.name=xxx.

[hmx...@gmail.com]

Thanks so much. It worked.

BTW, where can I get the information like this? I tried to search the documents on druid.io but couldn't get any.

[Gian Merlino]

I think this is an undocumented feature. We'll add some documentation about how to do this, although we might end up talking about the other method (also currently undocumented): providing a "jobProperties" map in your hadoop indexing config json.

[Fangjin Yang]

We just merged some docs about this into master.

[hmx...@gmail.com]

Thanks. Just found the "jobProperties" in the latest doc. I guess this will be the formal documented way to pass parameters for hadoop jobs.

From which version it is formally supported?

[Gian Merlino]

I think it was added in 0.6.106.

[Fangjin Yang]

The latest stable will have the changes.

--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/bcda4add-9dc9-4ccf-8acf-7f4337a47600%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.