Benchmarking Druid 0.7.0

Samy Chambi

unread,

Dec 1, 2014, 6:00:53 PM12/1/14

to druid-de...@googlegroups.com

Hi guys,

I've been able to reproduce these benchmarks: http://druid.io/blog/2014/03/17/benchmarking-druid.html, on Druid 0.6.146 version. However, when I tryed to use the new Druid's 0.7.0 version, I've encountered some troubles when loading data on Druid. Please, can you guys confirme me that this indexing service task's code: https://github.com/druid-io/druid-benchmark/blob/master/lineitem_small.task.json, is compatible with the 0.7.0 version of Druid.

I'm using a 0.20.2 hadoop cluster started in local mode, and this is the error I'm encountering when running the load commad :

<html>

<head>

<title>Error 500 </title>

</head>

<body>

<h2>HTTP ERROR: 500</h2>

<p>Problem accessing /druid/indexer/v1/task. Reason:

<pre> javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.indexing.common.task.HadoopIndexTask] value failed: null</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

Thanks,

Greetings!

Samy.

Charles Allen

unread,

Dec 1, 2014, 7:00:42 PM12/1/14

to druid-de...@googlegroups.com

Hadoop batch ingestion is broken in master right now. For some reason the config does not get properly persisted to the internal hadoop task. There's something funky going on in the object mapper when its reading the values. Hopefully the internal hadoop indexer task is simply missing a magic binding to allow it to get the configs read properly, but I haven't been able to dig into it much yet.

Samy Chambi

unread,

Dec 1, 2014, 11:29:46 PM12/1/14

to druid-de...@googlegroups.com

Ok. Thanks.

Are there any other solution to get data into Druid 0.7.0 ? I've tried your ingestion spec published here: https://groups.google.com/forum/#!searchin/druid-development/roaring$20bitmap/druid-development/VBNYgoafQwk/rQANLBwdmKYJ, after changing data location's path and data source's name, but it is still returning an error:

<html>

<head>

<title>Error 500 </title>

</head>

<body>

<h2>HTTP ERROR: 500</h2>

<p>Problem accessing /druid/indexer/v1/task. Reason:

<pre> javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Unexpected token (END_OBJECT), expected FIELD_NAME: missing property 'type' that is to contain type id (for class io.druid.indexing.common.task.Task)

at [Source: HttpInputOverHTTP@2b744a22; line: 1, column: 2817]</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

Thanks,

Samy.

Charles Allen

unread,

Dec 2, 2014, 2:27:44 PM12/2/14

to druid-de...@googlegroups.com

Sambi, I just pushed a fix for the CLI indexer, hopefully it alleviates the errors you are seeing. Can you please try again with a fresh pull of master?

Charles Allen

unread,

Dec 2, 2014, 2:31:14 PM12/2/14

to druid-de...@googlegroups.com

Worth noting: the error in your original post was the one I was seeing in the CLI hadoop indexer. The patch should fix that particular error. I haven't tried to reproduce the JSON parse one, but I suspect it has something to do with errors not being propagated correctly.

Any cases where this JSON one shows up would be great to document.

Charles Allen

unread,

Dec 2, 2014, 2:31:51 PM12/2/14

to druid-de...@googlegroups.com

and one of these days I'll stop calling you Sambi >.<

On Tuesday, December 2, 2014 11:27:44 AM UTC-8, Charles Allen wrote:

Samy Chambi

unread,

Dec 2, 2014, 4:29:00 PM12/2/14

to druid-de...@googlegroups.com

Hi Charles,

Thanks for the fixing. I'm still getting the same errors after running a new Druid's cluster using the current code on github.

I'm always running an indexing service as described in the documentation: java -Xmx2g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -classpath lib/*:..hadoop-0.20.2/conf:config/overlord io.druid.cli.Main server overlord. After that, when launching the load command, a lineitem_small.task.json will be sent to the indexing service. The following error is encountered at that point:

<html>

<head>

<title>Error 500 </title>

</head>

<body>

<h2>HTTP ERROR: 500</h2>

<p>Problem accessing /druid/indexer/v1/task. Reason:

<pre> javax.servlet.ServletException: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class io.druid.indexing.common.task.HadoopIndexTask] value failed: null</pre></p>

<hr /><i><small>Powered by Jetty://</small></i>

</body>

</html>

>> and one of these days I'll stop calling you Sambi >.<

Cool !

Thanks,

Samy.

Charles Allen

unread,

Dec 3, 2014, 1:41:30 PM12/3/14

to druid-de...@googlegroups.com

Hi Samy, the good news is that I can reproduce the error. Investigating now.

Charles Allen

unread,

Dec 3, 2014, 2:11:22 PM12/3/14

to druid-de...@googlegroups.com

Hi Samy, can you please try the following ingestion spec:

You will need to adjust

ioConfig -> inputSpec -> paths

and

dataSchema -> dataSource

{
    "hadoopCoordinates": "org.apache.hadoop:hadoop-core:0.20.205",
    "spec": {
        "dataSchema": {
            "dataSource": "test_small",
            "granularitySpec": {
                "intervals": [
                    "1980-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z"
                ],
                "queryGranularity": "day",
                "type": "arbitrary"
            },
            "metricsSpec": [
                {
                    "name": "count",
                    "type": "count"
                },
                {
                    "fieldName": "L_QUANTITY",
                    "name": "L_QUANTITY",
                    "type": "longSum"
                },
                {
                    "fieldName": "L_EXTENDEDPRICE",
                    "name": "L_EXTENDEDPRICE",
                    "type": "doubleSum"
                },
                {
                    "fieldName": "L_DISCOUNT",
                    "name": "L_DISCOUNT",
                    "type": "doubleSum"
                },
                {
                    "fieldName": "L_TAX",
                    "name": "L_TAX",
                    "type": "doubleSum"
                }
            ],
            "parser": {
                "parseSpec": {
                    "columns": [
                        "l_orderkey",
                        "l_partkey",
                        "l_suppkey",
                        "l_linenumber",
                        "l_quantity",
                        "l_extendedprice",
                        "l_discount",
                        "l_tax",
                        "l_returnflag",
                        "l_linestatus",
                        "l_shipdate",
                        "l_commitdate",
                        "l_receiptdate",
                        "l_shipinstruct",
                        "l_shipmode",
                        "l_comment"
                    ],
                    "delimiter": "|",
                    "dimensionsSpec": {
                        "dimensionExclusions": [
                            "l_shipdate",
                            "L_TAX",
                            "count",
                            "L_QUANTITY",
                            "L_DISCOUNT",
                            "L_EXTENDEDPRICE"
                        ],
                        "dimensions": [
                            "l_orderkey",
                            "l_suppkey",
                            "l_linenumber",
                            "l_returnflag",
                            "l_linestatus",
                            "l_commitdate",
                            "l_receiptdate",
                            "l_shipinstruct",
                            "l_shipmode",
                            "l_comment"
                        ]
                    },
                    "format": "tsv",
                    "timestampSpec": {
                        "column": "l_shipdate",
                        "format": "yyyy-MM-dd"
                    }
                },
                "type": "string"
            }
        },
        "ioConfig": {
            "inputSpec": {
                "paths": "lineitem.tbl.small.gz",
                "type": "static"
            },
            "type": "hadoop"
        }
    },
    "type": "index_hadoop"
}

Charles Allen

unread,

Dec 3, 2014, 6:27:18 PM12/3/14

to druid-de...@googlegroups.com

Hi Sammy, with a recent patch the hadoop stuff "should" work. I've been able to get ingestion to go with the following hadoop task:

{
    "hadoopCoordinates": "org.apache.hadoop:hadoop-client:2.3.0",
    "spec": {
        "dataSchema": {
            "dataSource": "test_small2",


                "paths": "lineitem.small.tbl",


                "type": "static"
            },
            "type": "hadoop"
        }
    },
    "type": "index_hadoop"
}

And I can get the Firehose ingestion started with the following, although performance is... bad for the local firehose right now:

{
    "spec": {
        "dataSchema": {
            "dataSource": "test_small2",


            "granularitySpec": {
                "intervals": [
                    "1980-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z"
                ],
                "queryGranularity": "day",


                "type": "uniform"


            "type":"index",
            "firehose":{"baseDir":"/Users/charlesallen/bin/wrk","filter":"lineitem.small.tbl","type":"local"}
        }
    },
    "type": "index"
}

Samy Chambi

unread,

Dec 3, 2014, 6:28:18 PM12/3/14

to druid-de...@googlegroups.com

Hi Charles,

The ingestion spec has worked. I got the task id. Good job!

Actually, I have some trouble to ingest new tasks that are already prensent in TaskLock. After processing the received task, the indexing service is returning a task failed log: 2014-12-03 23:04:19,129 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Task FAILED: HadoopIndexTask{id=index_hadoop_tpch_lineitem_small_2014-12-03T23:04:09.899Z, type=index_hadoop, dataSource=tpch_lineitem_small} (3369 run duration).

Is there any way to drop the tasks present in TaskLock?

Thanks,

Samy.

Samy Chambi

unread,

Dec 3, 2014, 6:33:41 PM12/3/14

to druid-de...@googlegroups.com

Please, note also that the new tasks are not added to the druid_segments table in mysql db.

Charles Allen

unread,

Dec 3, 2014, 6:36:39 PM12/3/14

to druid-de...@googlegroups.com

If you're using the hadoop one and haven't applied this patch: https://github.com/metamx/druid/pull/934/files (not yet in master as of this message), then it is probably the error that merge requests fixes. You can probably cherry-pick 2e6c25493738e45daf4b71fc852a6c3d62cd73f3 if you just want to get your local master branch functional for hadoop ingestion.

xrvl is cleaning up the PR a bit more before it goes into master, but 2e6c25493738e45daf4b71fc852a6c3d62cd73f3 seems to at least be stable enough to do some local testing.

Samy Chambi

unread,

Dec 4, 2014, 2:30:04 PM12/4/14

to druid-de...@googlegroups.com

Hi Charles,

I've pulled the current code on github which has merged the 2e6c25493738e45daf4b71fc852a6c3d62cd73f3 stuff. I'm still getting a failed status when ingesting the data segment. The metadata weren't ingested too. Please see bellow:

2014-12-04 18:33:20,775 INFO [pool-6-thread-2] io.druid.indexing.overlord.ForkingTaskRunner - Logging task index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z output to: /tmp/persistent/task/index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z/a67d0315-9806-47bf-bfe5-9ed5a09cecb1/log

2014-12-04 18:33:26,053 INFO [qtp264498757-24] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z]: LockTryAcquireAction{interval=1980-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z}

2014-12-04 18:33:26,054 INFO [qtp264498757-24] io.druid.indexing.overlord.TaskLockbox - Task[index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z] already present in TaskLock[index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z]

2014-12-04 18:33:29,369 INFO [qtp264498757-23] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z]: LockListAction{}

2014-12-04 18:33:29,818 INFO [pool-6-thread-2] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[0] for task: index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z

2014-12-04 18:33:29,819 INFO [pool-6-thread-2] io.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: log/index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z.log

2014-12-04 18:33:29,819 INFO [pool-6-thread-2] io.druid.indexing.overlord.ForkingTaskRunner - Removing temporary directory: /tmp/persistent/task/index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z/a67d0315-9806-47bf-bfe5-9ed5a09cecb1

2014-12-04 18:33:29,820 INFO [pool-6-thread-2] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z

2014-12-04 18:33:29,820 INFO [pool-6-thread-2] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z

2014-12-04 18:33:29,821 INFO [pool-6-thread-2] io.druid.indexing.overlord.HeapMemoryTaskStorage - Updating task index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z to status: TaskStatus{id=index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z, status=FAILED, duration=3317}

2014-12-04 18:33:29,821 INFO [pool-6-thread-2] io.druid.indexing.overlord.TaskLockbox - Removing task[index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z] from TaskLock[index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z]

2014-12-04 18:33:29,821 INFO [pool-6-thread-2] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z, dataSource=tpch_lineitem_small, interval=1980-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z, version=2014-12-04T18:33:20.768Z}

2014-12-04 18:33:29,821 INFO [pool-6-thread-2] io.druid.indexing.overlord.TaskQueue - Task done: HadoopIndexTask{id=index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z, type=index_hadoop, dataSource=tpch_lineitem_small}

2014-12-04 18:33:29,822 INFO [pool-6-thread-2] io.druid.indexing.overlord.TaskQueue - Task FAILED: HadoopIndexTask{id=index_hadoop_tpch_lineitem_small_2014-12-04T18:33:20.767Z, type=index_hadoop, dataSource=tpch_lineitem_small} (3317 run duration)

2014-12-04 18:33:37,525 INFO [TaskQueue-StorageSync] io.druid.indexing.overlord.TaskQueue - Synced 0 tasks from storage (0 tasks added, 0 tasks removed).

Thanks,

Samy.

Samy Chambi

unread,

Dec 11, 2014, 5:35:38 PM12/11/14

to druid-de...@googlegroups.com

Hi,

Even with a log stating "Task failed", I've found that new segments wa succesfully created and persisted in local deep storage, and I've been able to query them.

Greetings,

Samy.

Charles Allen

unread,

Dec 12, 2014, 8:19:40 PM12/12/14

to druid-de...@googlegroups.com

If you happen to have the log files from the run can you send them my way?

I've made a few fixes to various ingestion stuff over the last few weeks/days. I have not run into the issue you describe where a segment reports failure but still shows up in deep storage.

Samy Chambi

unread,

Dec 12, 2014, 10:19:31 PM12/12/14

to druid-de...@googlegroups.com

I'm running the current code on master branch. The indexing service log after sending a new hadoop indexing task:

2014-12-13 02:56:44,087 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Logging task index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z output to: /tmp/persistent/task/index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z/14ac891f-2007-4cec-b64b-df560e727035/log

2014-12-13 02:56:49,386 INFO [qtp1681108815-22] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z]: LockTryAcquireAction{interval=1980-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z}

2014-12-13 02:56:49,389 INFO [qtp1681108815-22] io.druid.indexing.overlord.TaskLockbox - Task[index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z] already present in TaskLock[index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z]

2014-12-13 02:56:52,772 INFO [qtp1681108815-25] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z]: LockListAction{}

2014-12-13 02:56:53,224 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Process exited with status[0] for task: index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z

2014-12-13 02:56:53,227 INFO [pool-6-thread-1] io.druid.indexing.common.tasklogs.FileTaskLogs - Wrote task log to: log/index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z.log

2014-12-13 02:56:53,231 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Removing temporary directory: /tmp/persistent/task/index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z/14ac891f-2007-4cec-b64b-df560e727035

2014-12-13 02:56:53,238 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z

2014-12-13 02:56:53,238 INFO [pool-6-thread-1] io.druid.indexing.overlord.ForkingTaskRunner - Ignoring request to cancel unknown task: index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z

2014-12-13 02:56:53,239 INFO [pool-6-thread-1] io.druid.indexing.overlord.HeapMemoryTaskStorage - Updating task index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z to status: TaskStatus{id=index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z, status=FAILED, duration=3382}

2014-12-13 02:56:53,239 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskLockbox - Removing task[index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z] from TaskLock[index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z]

2014-12-13 02:56:53,239 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskLockbox - TaskLock is now empty: TaskLock{groupId=index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z, dataSource=tpch_lineitem_small, interval=1980-01-01T00:00:00.000Z/2020-01-01T00:00:00.000Z, version=2014-12-13T02:56:43.998Z}

2014-12-13 02:56:53,240 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Task done: HadoopIndexTask{id=index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z, type=index_hadoop, dataSource=tpch_lineitem_small}

2014-12-13 02:56:53,241 INFO [pool-6-thread-1] io.druid.indexing.overlord.TaskQueue - Task FAILED: HadoopIndexTask{id=index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z, type=index_hadoop, dataSource=tpch_lineitem_small} (3382 run duration)

When I check in "/tmp/persistent/task/", I find that "index_hadoop_tpch_lineitem_small_2014-12-13T02:56:43.993Z" file is added.

Samy.

Samy Chambi

unread,

Dec 18, 2014, 5:08:50 PM12/18/14

to druid-de...@googlegroups.com

Hi,

>> I have not run into the issue you describe where a segment reports failure but still shows up in deep storage.

I think that it was due to a conflict caused by using two different versions of Druid in local mode on the same server. After removing the raw segment from the mysql druid_segments table and restarting a druid cluster using the 0.7.0 version, I was no able to ingest the data segment by sending a new indexing task.

When I checked the log file of the task, I find that it was due to a not sufficient DirectMemorySize on my server. Please see bellow the error which was reported on the task's log file:

1) Not enough direct memory. Please adjust -XX:MaxDirectMemorySize, druid.processing.buffer.sizeBytes, or druid.processing.numThreads: maxDirectMemory[3 728 211 968], memoryNeeded[8 589 934 592] = druid.processing.buffer.sizeBytes[1 073 741 824] * ( druid.processing.numThreads[7] + 1 )

at io.druid.guice.DruidProcessingModule.getIntermediateResultsPool(DruidProcessingModule.java:82)

while locating io.druid.collections.StupidPool<java.nio.ByteBuffer> annotated with @io.druid.guice.annotations.Global()

for parameter 1 at io.druid.query.groupby.GroupByQueryEngine.<init>(GroupByQueryEngine.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:82)

while locating io.druid.query.groupby.GroupByQueryEngine

for parameter 0 at io.druid.query.groupby.GroupByQueryRunnerFactory.<init>(GroupByQueryRunnerFactory.java:79)

at io.druid.guice.QueryRunnerFactoryModule.configure(QueryRunnerFactoryModule.java:79)

while locating io.druid.query.groupby.GroupByQueryRunnerFactory

while locating io.druid.query.QueryRunnerFactory annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=24, type=MAPBINDER)

at io.druid.guice.DruidBinders.queryRunnerFactoryBinder(DruidBinders.java:38)

while locating java.util.Map<java.lang.Class<? extends io.druid.query.Query>, io.druid.query.QueryRunnerFactory>

for parameter 0 at io.druid.query.DefaultQueryRunnerFactoryConglomerate.<init>(DefaultQueryRunnerFactoryConglomerate.java:36)

while locating io.druid.query.DefaultQueryRunnerFactoryConglomerate

at io.druid.guice.StorageNodeModule.configure(StorageNodeModule.java:55)

while locating io.druid.query.QueryRunnerFactoryConglomerate

for parameter 9 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:78)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:129)

while locating io.druid.indexing.common.TaskToolboxFactory

for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:69)

at io.druid.cli.CliPeon$1.configure(CliPeon.java:155)

while locating io.druid.indexing.overlord.ThreadPoolTaskRunner

while locating io.druid.query.QuerySegmentWalker

for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:93)

while locating io.druid.server.QueryResource

Actually, I have a 10 gig of free RAM space. Even when fixing the server parameter to -XX:MaxDirectMemorySize=10G, I'm always getting this error; perhaps Druid needs a lot more space than memoryNeeded[8 589 934 592] to ingest the lineitem.tbl.gz data ?

Another thing I wanted to point, is that I'm specifying to use only 1 thread in the overlord config, but Druid will calculate the needed memory size in function of the number of available processors in the machine (please see the link) which is 7 in this case, in lieu of the number of threads specified in the overlord config file. This point shouldn't be fixed ?

Thanks,

Samy.

Fangjin Yang

unread,

Dec 18, 2014, 8:26:35 PM12/18/14

to druid-de...@googlegroups.com

Samy, can you share your config?

Samy Chambi

unread,

Dec 18, 2014, 10:49:09 PM12/18/14

to druid-de...@googlegroups.com

Hi Fangjin,

Here is the overlord config file :

-server

-Xmx2g

-Xms2g

-XX:MaxDirectMemorySize=10G

-Ddruid.extensions.coordinates=[\"io.druid.extensions:mysql-metadata-storage:0.7.0-SNAPSHOT\"]

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Ddruid.host=localhost

-Ddruid.port=8080

-Ddruid.service=overlord

-Ddruid.zk.service.host=localhost

-Ddruid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid

-Ddruid.db.connector.user=druid

-Ddruid.db.connector.password=diurd

-Ddruid.selectors.indexing.serviceName=overlord

-Ddruid.indexer.queue.startDelay=PT0M

-Ddruid.indexer.runner.javaOpts="-server -Xmx2g"

-Ddruid.indexer.runner.startPort=8088

-Ddruid.indexer.fork.property.druid.processing.numThreads=1

-Ddruid.indexer.fork.property.druid.computation.buffer.size=1000000000

Samy.

Charles Allen

unread,

Dec 18, 2014, 11:55:53 PM12/18/14

to druid-de...@googlegroups.com

I usually encounter this error when druid.processing.numThreads isn't set correctly. You can see in the log that it *thinks* its supposed to be 7, but the forking properties you have should set it to 1.

I was encountering some other peon settings issues today where some tasks weren't picking up settings I expected. No hard evidence either way yet though.

Charles Allen

unread,

Dec 18, 2014, 11:58:35 PM12/18/14

to druid-de...@googlegroups.com

Can you try setting the druid.processing.numThreads to 1 in the overlord as well?

Also, can you look at the command line that gets output from the overlord/middleManager log right when the task is run and look to make sure the peon's properties are populated correctly (specifically druid.processing.numThreads)?

Samy Chambi

unread,

Dec 19, 2014, 11:25:16 AM12/19/14

to druid-de...@googlegroups.com

I've added the command druid.processing.numThreads=1 to the overlord config and sent a new indexing task and It was successfully processed:

2014-12-19 01:15:48,616 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {

"id" : "index_hadoop_tpch_lineitem_small_2014-12-19T00:28:56.252-05:00",

"status" : "SUCCESS",

"duration" : 2806565

}

However, I observed that the raw segment wasn't added to the druid_segments table in Mysql database. When checking the task log file, I found this log output:

2014-12-19 00:29:05,559 INFO [task-runner-0] io.druid.indexer.HadoopDruidIndexerJob - No metadataStorageUpdaterJob set in the config. This is cool if you are running a hadoop index task, otherwise nothing will be uploaded to database.

How can I fix that to have Druid sending a raw segment to the druid_segments table?

One more thing, should I add this command line : druid.processing.bitmap.type='roaring' to the overlord config to switch to a Roaring bitmap compression?

Thanks,

Samy.

Samy Chambi

unread,

Dec 19, 2014, 11:30:32 AM12/19/14

to druid-de...@googlegroups.com

Also,

>> Also, can you look at the command line that gets output from the overlord/middleManager log right when the task is run and look to make sure the peon's properties are populated correctly (specifically druid.processing.numThreads)?

I checked that and the druid.processing.numThreads=1 command was correctly picked up.

Samy.

Charles Allen

unread,

Dec 30, 2014, 3:52:46 PM12/30/14

to druid-de...@googlegroups.com

Hi Samy, Back at it after a sporadic availability winter break.

Have you added this option to the overlord?

-Ddruid.indexer.runner.type=local

Samy Chambi

unread,

Dec 30, 2014, 4:02:04 PM12/30/14

to druid-de...@googlegroups.com

Hi Charles,

>> ...after a sporadic availability winter break.

Happy for you!

>> Have you added this option to the overlord?

>> -Ddruid.indexer.runner.type=local

No, not yet. I'll check with that soon...

Thanks,

Samy.

Charles Allen

unread,

Dec 30, 2014, 6:36:55 PM12/30/14

to druid-de...@googlegroups.com

FYI, the overlord / middle manager relationship is kind of annoying to configure currently. If you set the runner to local (instead of remote) then the overlord should spawn the tasks directly, and the tasks should be easier to configure since they inherit druid.* properties (see io.druid.indexing.overlord.ForkingTaskRunner )

Samy Chambi

unread,

Dec 30, 2014, 8:47:27 PM12/30/14

to druid-de...@googlegroups.com

I've added the option: -Ddruid.indexer.runner.type=local to the Overlord, however no line is inserted in the druid_segments table after the success of the ingestion.

Samy Chambi

unread,

Dec 31, 2014, 10:09:26 AM12/31/14

to druid-de...@googlegroups.com

Hi guys,

I've launched an indexing task using the overlord of an old version of druid (1.6.146) and the line item was correctly inserted into the druid_sgments table after the indexing task has been successfully completed. However, with the curent Druid's version, no line related to the processed segment is added to the druid_segments table after the successful completeness of the indexing task.

Can you give me some directions to fix that problem up?

Thanks.

Nishant Bangarwa

unread,

Dec 31, 2014, 11:30:26 AM12/31/14

to druid-de...@googlegroups.com

Hi Samy,

can you share the runtime.props for the overlord that you are using with 0.7 ?

when running with druid 0.7, have you added the mysql-metadata-storage module and set updated the metadata properties in runtime.props ?

fwiw, you will need to rename druid.db.* properties to druid.metadata.storage.*

here are sample metadata storage configs -

# Metadata Storage
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc\:mysql\://localhost\:3306/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=diurd

you can also look at sample configs for 0.7 from https://github.com/druid-io/druid/tree/master/examples/config

--

Nishant

Software Engineer

|

METAMARKETS

m	+91-9729200044

nishant....@metamarkets.com

Samy Chambi

unread,

Dec 31, 2014, 1:12:03 PM12/31/14

to druid-de...@googlegroups.com

Hi Nishant,

I'm using this config for the overlord :

-server

-Xmx2g

-Xms2g

-XX:MaxDirectMemorySize=2g

-Duser.timezone=UTC

-Dfile.encoding=UTF-8

-Ddruid.host=localhost

-Ddruid.port=8087

-Ddruid.service=overlord

-Ddruid.zk.service.host=localhost

-Ddruid.extensions.coordinates=[\"io.druid.extensions:mysql-metadata-storage:0.7.0-SNAPSHOT\"]

-Ddruid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid

-Ddruid.db.connector.user=druid

-Ddruid.db.connector.password=diurd

-Ddruid.selectors.indexing.serviceName=overlord

-Ddruid.indexer.queue.startDelay=PT0M

-Ddruid.indexer.runner.javaOpts="-server -Xmx2g"

-Ddruid.indexer.runner.startPort=8088

druid.processing.numThreads=1

-Ddruid.indexer.runner.type=local

-Ddruid.indexer.fork.property.druid.processing.numThreads=1

-Ddruid.indexer.fork.property.druid.computation.buffer.size=100000000

When I change the prefixes of the meatadata storage stuff to druid.metadata.storage all seems to be correct, however the addition of this option: druid.metadata.storage.type=mysql, causes errors when starting an overlord node:

~/druid-services-0.7.0-SNAPSHOT$ java -classpath lib/*:/home/samytto/druid-services-0.6.146/hadoop-1.0.3/conf:config/overlord io.druid.cli.Main server overlord

2014-12-31 12:57:46,296 INFO [main] io.druid.guice.PropertiesModule - Loading properties from runtime.properties

2014-12-31 12:57:46,325 INFO [main] org.hibernate.validator.internal.util.Version - HV000001: Hibernate Validator 5.0.1.Final

2014-12-31 12:57:46,742 INFO [main] io.druid.guice.JsonConfigurator - Loaded class[class io.druid.guice.ExtensionsConfig] from props[druid.extensions.] as [ExtensionsConfig{searchCurrentClassloader=true, coordinates=[], defaultVersion='0.7.0-SNAPSHOT', localRepository='/home/samytto/.m2/repository', remoteRepositories=[http://repo1.maven.org/maven2/, https://metamx.artifactoryonline.com/metamx/pub-libs-releases-local]}]

Exception in thread "main" com.google.inject.CreationException: Guice creation errors:

1) Unknown provider[mysql] of Key[type=io.druid.metadata.MetadataStorageConnector, annotation=[none]], known options[[derby]]

at io.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:67)

while locating io.druid.metadata.MetadataStorageConnector

at io.druid.guice.JacksonConfigManagerModule.getConfigManager(JacksonConfigManagerModule.java:52)

while locating io.druid.common.config.ConfigManager

for parameter 0 at io.druid.common.config.JacksonConfigManager.<init>(JacksonConfigManager.java:43)

at io.druid.guice.JacksonConfigManagerModule.configure(JacksonConfigManagerModule.java:41)

while locating io.druid.common.config.JacksonConfigManager

for parameter 0 at io.druid.guice.JacksonConfigProvider.configure(JacksonConfigProvider.java:80)

at io.druid.guice.JacksonConfigProvider.bind(JacksonConfigProvider.java:38)

1 error

at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448)

at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)

at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)

at com.google.inject.Guice.createInjector(Guice.java:96)

at com.google.inject.Guice.createInjector(Guice.java:73)

at com.google.inject.Guice.createInjector(Guice.java:62)

at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:371)

at io.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:56)

at io.druid.cli.ServerRunnable.run(ServerRunnable.java:39)

at io.druid.cli.Main.main(Main.java:90)

I'm sending this indexing task :

{
"spec": {
"dataSchema": {
"dataSource": "tpch_lineitem_small",
"granularitySpec": {
"intervals": [
"1980/2020"

"dimensions": [

"l_orderkey",
"l_partkey",
"l_suppkey",
"l_linenumber",

"l_returnflag",
"l_linestatus",
"l_shipdate",
"l_commitdate",
"l_receiptdate",
"l_shipinstruct",
"l_shipmode",
"l_comment"
]
},

"format": "tsv",
"timestampSpec": {
"column": "l_shipdate",
"format": "yyyy-MM-dd"
}
},
"type": "string"
}

},
"ioConfig": {
"inputSpec": {
"paths": "/home/druid-benchmark/lineitem.tbl.gz",

"type": "static"
},
"type": "hadoop"
}
},
"type": "index_hadoop"
}

Samy.

Fangjin Yang

unread,

Dec 31, 2014, 1:14:45 PM12/31/14

to druid-de...@googlegroups.com

-Ddruid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid

-Ddruid.db.connector.user=druid

-Ddruid.db.connector.password=diurd

Replace druid.db.* with druid.metadata.storage.*

Samy Chambi

unread,

Dec 31, 2014, 1:16:21 PM12/31/14

to druid-de...@googlegroups.com

After some little time, I've found these errors in the overlord console :

2014-12-31 13:12:43,062 WARN [config-manager-0] io.druid.common.config.ConfigManager - Exception when checking property[worker.config]

org.skife.jdbi.v2.exceptions.UnableToObtainConnectionException: java.sql.SQLException: Cannot create JDBC driver of class 'org.apache.derby.jdbc.ClientDriver' for connect URL 'jdbc:mysql://localhost:3306/druid'

at org.skife.jdbi.v2.DBI.open(DBI.java:210)

at org.skife.jdbi.v2.DBI.withHandle(DBI.java:257)

at io.druid.metadata.SQLMetadataConnector.lookup(SQLMetadataConnector.java:335)

at io.druid.common.config.ConfigManager.poll(ConfigManager.java:108)

at io.druid.common.config.ConfigManager.access$600(ConfigManager.java:44)

at io.druid.common.config.ConfigManager$PollingCallable.call(ConfigManager.java:250)

at io.druid.common.config.ConfigManager$PollingCallable.call(ConfigManager.java:234)

at com.metamx.common.concurrent.ScheduledExecutors$2.run(ScheduledExecutors.java:99)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: java.sql.SQLException: Cannot create JDBC driver of class 'org.apache.derby.jdbc.ClientDriver' for connect URL 'jdbc:mysql://localhost:3306/druid'

at org.apache.commons.dbcp2.BasicDataSource.createConnectionFactory(BasicDataSource.java:2023)

at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:1897)

at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1413)

at org.skife.jdbi.v2.DataSourceConnectionFactory.openConnection(DataSourceConnectionFactory.java:36)

at org.skife.jdbi.v2.DBI.open(DBI.java:192)

... 14 more

Samy Chambi

unread,

Dec 31, 2014, 1:24:50 PM12/31/14

to druid-de...@googlegroups.com

I did, and I'm guetting this error after starting an overlord node:

2014-12-31 13:23:25,885 WARN [config-manager-0] io.druid.common.config.ConfigManager - Exception when checking property[worker.config]