Error while indexing csv file

564 views
Skip to first unread message

Kasi Sankaralingam

unread,
Dec 5, 2014, 1:37:17 PM12/5/14
to druid-de...@googlegroups.com
I have the following spec file

{
  "type" : "index",
  "dataSource" : "orderepa",
  "granularitySpec" : {
    "type" : "uniform",
    "gran" : "DAY",
    "intervals" : [ "2013-08-01/2013-09-30" ]
  },
  "aggregators" : [{
     "type" : "count",
     "name" : "NumOrders"
    }, {
     "type" : "doubleSum",
     "name" : "OrderDifficultyAvg",
     "fieldName" : "OrderDifficultyAvg"
    }, {
     "type" : "doubleSum",
     "name" : "Fill-Ratio",
     "fieldName" : "Fill-Ratio"
    }, {
     "type" : "doubleSum",
     "name" : "NoFill-Ratio",
     "fieldName" : "NoFill-Ratio"
    }, {
     "type" : "doubleSum",
     "name" : "VolFill-Ratio",
     "fieldName" : "VolFill-Ratio"
    }, {
     "type" : "doubleSum",
     "name" : "AVGFillRatio",
     "fieldName" : "AVGFillRatio"
    }, {
     "type" : "doubleSum",
     "name" : "OrderDurnAvg",
     "fieldName" : "OrderDurnAvg"
    }, {
     "type" : "doubleSum",
     "name" : "Num_SweepsAvg",
     "fieldName" : "Num_SweepsAvg"
    }, {
     "type" : "doubleSum",
     "name" : "OrderAmtUSDAvg",
     "fieldName" : "OrderAmtUSDAvg"
    }, {
     "type" : "doubleSum",
     "name" : "OrderAmtUSDSum",
     "fieldName" : "OrderAmtUSDSum"
    }, {
     "type" : "doubleSum",
     "name" : "PIPipAvg",
     "fieldName" : "PIPipAvg"
    }, {
     "type" : "doubleSum",
     "name" : "PIPipMed",
     "fieldName" : "PIPipMed" 
    }, {
     "type" : "doubleSum",
     "name" : "TotalOrderDurationMax",
     "fieldName" : "TotalOrderDurationMax" 
    }, {
     "type" : "doubleSum",
     "name" : "TotalTradeDurationMax",
     "fieldName" : "TotalTradeDurationMax" 
    }, {
     "type" : "doubleSum",
     "name" : "NumSweepsMax",
     "fieldName" : "NumSweepsMax" 
    }, {
     "type" : "doubleSum",
     "name" : "NumTrades",
     "fieldName" : "NumSweepsMax" 
    }, {
     "type" : "doubleSum",
     "name" : "NumConfirmed",
     "fieldName" : "NumConfirmed" 
    }, {
     "type" : "doubleSum",
     "name" : "NumRejected",
     "fieldName" : "NumRejected" 
    }, {
     "type" : "doubleSum",
     "name" : "NumFailed",
     "fieldName" : "NumFailed" 
    }, {
     "type" : "doubleSum",
     "name" : "FilledAmountUSDSum",
     "fieldName" : "FilledAmountUSDSum" 
    }, {
     "type" : "doubleSum",
     "name" : "Confirmed-Ratio",
     "fieldName" : "Confirmed-Ratio" 
    }, {
     "type" : "doubleSum",
     "name" : "Rejected-ratio",
     "fieldName" : "Rejected-ratio" 
    }, {
     "type" : "doubleSum",
     "name" : "Failed-Ratio",
     "fieldName" : "Failed-Ratio" 
   }
],
"firehose" : {
      "type" : "local",
      "baseDir" : "examples/indexing/",
      "filter" : "testepa.csv",
      "parser" : {
        "timestampSpec" : {
          "column" : "createddate",
   "format": "auto"
        },
        "data" : {
          "type" : "csv",
          "columns" : ["takerorg","ccypair","createddate","channel","ordertype","MarketShiftCode","TIF","NumOrders","OrderDifficultyAvg","Fill-Ratio","NoFill-Ratio","ParFill-Ratio","VolFill-Ratio","AVGFillRatio","OrderDurnAvg","Num_SweepsAvg","OrderAmtUSDAvg","OrderAmtUSDSum","PIPipAvg","PIPipMed","TotalOrderDurationMax","TotalTradeDurationMax","TotalTradeDurationavg","isOrginatingOrg","NumSweepsMax","NumTrades","NumConfirmed","NumRejected","NumFailed","FilledAmountUSDSum","Confirmed-Ratio","Rejected-ratio","Failed-Ratio"],
          "dimensions" : ["takerorg","ccypair","channel","ordertype","MarketShiftCode","TIF"]

        }
      }
    }
}


This is my sample data file:

"xxxx" "GBP/USD" 2014-09-02T00:00:00Z "BA/ESP" "LIMIT" "Very Hard" "GTD" 1.00 -0.67 0.00 0.00 1.00 0.38 0.38 631.00 5.00 829950.00 829950.00 -11.31 -11.31 631.00 423.00 423.00 0 5 14 7 4 3 312061 0.5 0.29 0.21
"xxxx" "GBP/USD" 2014-09-16T00:00:00Z "BA/ESP" "LIMIT" "Very Hard" "GTD" 1.00 -2.16 0.00 0.00 1.00 0.91 0.91 1757.00 5.00 536828.00 536828.00 -6.50 -6.50 1757.00 944.00 944.00 0.00 5 7 1 6 0 488025 0.14 0.86 0
"xxxx" "GBP/USD" 2014-09-15T00:00:00Z "BA/ESP" "LIMIT" "Hard" "GTD" 1.00 -2.77 0.00 0.00 1.00 0.97 0.97 1255.00 38.00 73203750.00 73203750.00 -1.28 -1.28 1255.00 845.00 845.00 0.00 38 96 42 43 11 71088975 0.44 0.45 0.11
"xxxx" "GBP/USD" 2014-09-12T00:00:00Z "BA/ESP" "LIMIT" "Very Hard" "GTD" 1.00 -3.11 0.00 0.00 1.00 0.50 0.50 1528.00 9.00 3227000.00 3227000.00 -1.00 -1.00 1528.00 1432.00 1432.00 0.00 9 9 1 7 1 1613500 0.11 0.78 0.11
"xxxxx" "GBP/USD" 2014-09-10T00:00:00Z "BA/ESP" "LIMIT" "Hard" "GTD" 1.00 -0.85 0.00 0.00 1.00 0.05 0.05 3506.00 3.00 32332000.00 32332000.00 -0.70 -0.70 3506.00 77.00 77.00 0.00 3 3 1 2 0 1616600 0.33 0.67 0

I am seeing this error:

com.metamx.common.parsers.ParseException: Unparseable timestamp found!
	at io.druid.data.input.impl.MapInputRowParser.parse(MapInputRowParser.java:76)
	at io.druid.data.input.impl.StringInputRowParser.parseMap(StringInputRowParser.java:124)
	at io.druid.data.input.impl.StringInputRowParser.parse(StringInputRowParser.java:119)
	at io.druid.data.input.impl.FileIteratingFirehose.nextRow(FileIteratingFirehose.java:54)
	at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:220)
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:177)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:219)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:198)

What ever I do I am always seeing this error and also even if I specify csv it is reporting as tsv, I am about to give up on Druid.

"format" : "tsv",
          "timestampSpec" : {
            "column" : "createddate",
            "format" : "auto"

Thanks for any help

Kasi


Fangjin Yang

unread,
Dec 5, 2014, 5:17:12 PM12/5/14
to druid-de...@googlegroups.com
Hi Kasi, on first glance looking at your raw data is that the data does not appear to be in CSV format.

Kasi Sankaralingam

unread,
Dec 5, 2014, 6:25:37 PM12/5/14
to druid-de...@googlegroups.com
Sorry! here is the right data

"TakerOrg","CcyPair","CreatedDate","Channel","OrderType","MarketShiftCode","TIF","NumOrders","OrderDifficultyAvg","Fill-Ratio","NoFill-Ratio","ParFill-Ratio","VolFill-Ratio","AVGFillRatio","OrderDurnAvg","Num_SweepsAvg","OrderAmtUSDAvg","OrderAmtUSDSum","PIPipAvg","PIPipMed","TotalOrderDurationMax","TotalTradeDurationMax","TotalTradeDurationavg","isOrginatingOrg","NumSweepsMax","NumTrades","NumConfirmed","NumRejected","NumFailed","FilledAmountUSDSum","Confirmed-Ratio","Rejected-ratio","Failed-Ratio"
"xxxxx","GBP/USD",2014-09-02,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-0.67,0.00,0.00,1.00,0.38,0.38,631.00,5.00,829950.00,829950.00,-11.31,-11.31,631.00,423.00,423.00,0,5,14,7,4,3,312061,0.5,0.29,0.21
"xxxx","GBP/USD",2014-09-16,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-2.16,0.00,0.00,1.00,0.91,0.91,1757.00,5.00,536828.00,536828.00,-6.50,-6.50,1757.00,944.00,944.00,"",5,7,1,6,0,488025,0.14,0.86,0
"xxxxx","GBP/USD",2014-09-15,"BA/ESP","LIMIT","Hard","GTD",1.00,-2.77,0.00,0.00,1.00,0.97,0.97,1255.00,38.00,73203750.00,73203750.00,-1.28,-1.28,1255.00,845.00,845.00,"",38,96,42,43,11,71088975,0.44,0.45,0.11
"xxxx","GBP/USD",2014-09-12,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-3.11,0.00,0.00,1.00,0.50,0.50,1528.00,9.00,3227000.00,3227000.00,-1.00,-1.00,1528.00,1432.00,1432.00,"",9,9,1,7,1,1613500,0.11,0.78,0.11
"xxxx","GBP/USD",2014-09-10,"BA/ESP","LIMIT","Hard","GTD",1.00,-0.85,0.00,0.00,1.00,0.05,0.05,3506.00,3.00,32332000.00,32332000.00,-0.70,-0.70,3506.00,77.00,77.00,"",3,3,1,2,0,1616600,0.33,0.67,0
"xxxxx","GBP/USD",2014-09-11,"BA/ESP","LIMIT","Hard","GTD",1.00,-0.57,0.00,0.00,1.00,0.06,0.06,2996.00,3.00,16097000.00,16097000.00,-0.60,-0.60,2996.00,77.00,77.00,"",3,10,2,8,0,889359,0.2,0.8,0
"xxxxx","GBP/USD",2014-09-10,"BA/ESP","LIMIT","Hard","GTD",1.00,-0.69,0.00,0.00,1.00,0.31,0.31,3000.00,13.00,16166000.00,16166000.00,-0.42,-0.42,3000.00,540.00,540.00,"",13,25,4,21,0,5011460,0.16,0.84,0
"xxxxx","GBP/USD",2014-09-08,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-1.56,0.00,0.00,1.00,0.21,0.21,1004.00,2.00,12602425.00,12602425.00,-0.40,-0.40,1004.00,169.00,169.00,"",2,5,1,4,0,2643260,0.2,0.8,0
"xxxxx","GBP/USD",2014-08-21,"BA/ESP","LIMIT","Not Easy","GTD",1.00,-0.48,0.00,0.00,1.00,0.20,0.20,1259.00,1.00,833575.00,833575.00,-0.40,-0.40,1259.00,0.00,0.00,"",1,1,1,0,0,166715,1,0,0

Fangjin Yang

unread,
Dec 5, 2014, 6:29:38 PM12/5/14
to druid-de...@googlegroups.com
Hi Kasi,

The "auto" format tries to parse the timestamp as either millis or ISO 8601, of which timestamps such as '2014-09-02' are not.

The format can also understand datetime formats (http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html). You can try specifying the format as 'yyyy-MM-DD'


--
You received this message because you are subscribed to the Google Groups "Druid Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/b1d1f38c-c073-460e-b4f5-4e180e3a6d99%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Fangjin Yang

unread,
Dec 5, 2014, 6:31:09 PM12/5/14
to druid-de...@googlegroups.com
Okay, clearly I responded too fast. That should be a valid timestamp and something else is going on. Looking again.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-development+unsubscribe@googlegroups.com.
To post to this group, send email to druid-development@googlegroups.com.

Fangjin Yang

unread,
Dec 5, 2014, 6:35:16 PM12/5/14
to druid-de...@googlegroups.com
Hi Kasi,

2 questions:
1) The interval of the data appears to be for 2013, but the timestamps appear to be in 2014.
2) Does the csv file include a header row? If so, can you remove it and try again?

Fangjin Yang

unread,
Dec 5, 2014, 6:55:13 PM12/5/14
to druid-de...@googlegroups.com
Hi Kasi, I found the problem

change
"data": {
      "type": "csv",

to 
"data": {
      "format": "csv",

Kasi Sankaralingam

unread,
Dec 9, 2014, 5:05:09 PM12/9/14
to druid-de...@googlegroups.com
Hi Fangjin,

Thanks a lot, I will try
To unsubscribe from this group and stop receiving emails from it, send an email to druid-developm...@googlegroups.com.
To post to this group, send email to druid-de...@googlegroups.com.

Kasi Sankaralingam

unread,
Dec 9, 2014, 6:26:58 PM12/9/14
to druid-de...@googlegroups.com
It does not work, I have pretty much tried everything, have spent over a week on this, why is it so hard to use this?

Fangjin Yang

unread,
Dec 9, 2014, 6:32:58 PM12/9/14
to druid-de...@googlegroups.com
Kasi, do you have a stack trace? Also, I'd be happy to help out over IRC.

Kasi Sankaralingam

unread,
Dec 9, 2014, 6:55:02 PM12/9/14
to druid-de...@googlegroups.com
Hi Fangjin,

Here is the stack trace and the latest config file:

},
  "dataSource" : "orderepa",
  "groupId" : "index_orderepa_2014-12-09T23:42:23.359Z",
  "interval" : "2014-08-01T00:00:00.000Z/2014-09-30T00:00:00.000Z",
  "resource" : {
    "availabilityGroup" : "index_orderepa_2014-12-09T23:42:23.359Z",
    "requiredCapacity" : 1
  }
}
2014-12-09 23:42:28,728 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_orderepa_2014-12-09T23:42:23.359Z]: LockTryAcquireAction{interval=2014-08-01T00:00:00.000Z/2014-09-30T00:00:00.000Z}
2014-12-09 23:42:28,736 INFO [main] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_orderepa_2014-12-09T23:42:23.359Z] to overlord[http://localhost:8087/druid/indexer/v1/action]: LockTryAcquireAction{interval=2014-08-01T00:00:00.000Z/2014-09-30T00:00:00.000Z}
2014-12-09 23:42:28,749 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://localhost:8087
2014-12-09 23:42:28,782 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://localhost:8087
2014-12-09 23:42:28,783 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://localhost:8087
2014-12-09 23:42:28,783 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://localhost:8087
2014-12-09 23:42:28,783 INFO [main] com.metamx.http.client.pool.ChannelResourceFactory - Generating: http://localhost:8087
2014-12-09 23:42:28,834 INFO [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Running task: index_orderepa_2014-12-09T23:42:23.359Z
2014-12-09 23:42:28,835 INFO [task-runner-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Performing action for task[index_orderepa_2014-12-09T23:42:23.359Z]: LockListAction{}
2014-12-09 23:42:28,841 INFO [task-runner-0] io.druid.indexing.common.actions.RemoteTaskActionClient - Submitting action for task[index_orderepa_2014-12-09T23:42:23.359Z] to overlord[http://localhost:8087/druid/indexer/v1/action]: LockListAction{}
2014-12-09 23:42:28,862 ERROR [task-runner-0] io.druid.indexing.overlord.ThreadPoolTaskRunner - Exception while running task[IndexTask{id=index_orderepa_2014-12-09T23:42:23.359Z, type=index, dataSource=orderepa}]
com.metamx.common.parsers.ParseException: Unparseable timestamp found!
	at io.druid.data.input.impl.MapInputRowParser.parse(MapInputRowParser.java:76)
	at io.druid.data.input.impl.StringInputRowParser.parseMap(StringInputRowParser.java:124)
	at io.druid.data.input.impl.StringInputRowParser.parse(StringInputRowParser.java:119)
	at io.druid.data.input.impl.FileIteratingFirehose.nextRow(FileIteratingFirehose.java:54)
	at io.druid.indexing.common.task.IndexTask.getDataIntervals(IndexTask.java:220)
	at io.druid.indexing.common.task.IndexTask.run(IndexTask.java:177)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:219)
	at io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:198)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: Null timestamp in input: {takerorg=null}
	at io.druid.data.input.impl.MapInputRowParser.parse(MapInputRowParser.java:67)
	... 11 more
2014-12-09 23:42:28,868 INFO [task-runner-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
  "id" : "index_orderepa_2014-12-09T23:42:23.359Z",
  "status" : "FAILED",
  "duration" : 30

Config file:

{
  "type" : "index",
  "dataSource" : "orderepa",
  "granularitySpec" : {
    "type" : "uniform",
    "gran" : "DAY",
    "intervals" : [ "2013-08-01/2013-09-30" ]
  },
  "aggregators" : [{
     "type" : "count",
     "name" : "NumOrders"
    }, {
     "type" : "doubleSum",
     "name" : "OrderDifficultyAvg",
     "fieldName" : "OrderDifficultyAvg"
    }, {
     "type" : "doubleSum",
     "name" : "Fill-Ratio",
     "fieldName" : "Fill-Ratio"
    }, {
     "type" : "doubleSum",
     "name" : "NoFill-Ratio",
     "fieldName" : "NoFill-Ratio"
    }, {
     "type" : "doubleSum",
     "name" : "VolFill-Ratio",
     "fieldName" : "VolFill-Ratio"
    }, {
     "type" : "doubleSum",
     "name" : "AVGFillRatio",
     "fieldName" : "AVGFillRatio"
    }, {
     "type" : "doubleSum",
     "name" : "OrderDurnAvg",
     "fieldName" : "OrderDurnAvg"
    }, {
     "type" : "doubleSum",
     "name" : "Num_SweepsAvg",
     "fieldName" : "Num_SweepsAvg"
    }, {
     "type" : "doubleSum",
     "name" : "OrderAmtUSDAvg",
     "fieldName" : "OrderAmtUSDAvg"
    }, {
     "type" : "doubleSum",
     "name" : "OrderAmtUSDSum",
     "fieldName" : "OrderAmtUSDSum"
    }, {
     "type" : "doubleSum",
     "name" : "PIPipAvg",
     "fieldName" : "PIPipAvg"
    }, {
     "type" : "doubleSum",
     "name" : "PIPipMed",
     "fieldName" : "PIPipMed" 
    }, {
     "type" : "doubleSum",
     "name" : "TotalOrderDurationMax",
     "fieldName" : "TotalOrderDurationMax" 
    }, {
     "type" : "doubleSum",
     "name" : "TotalTradeDurationMax",
     "fieldName" : "TotalTradeDurationMax" 
    }, {
     "type" : "doubleSum",
     "name" : "NumSweepsMax",
     "fieldName" : "NumSweepsMax" 
    }, {
     "type" : "doubleSum",
     "name" : "NumTrades",
     "fieldName" : "NumSweepsMax" 
    }, {
     "type" : "doubleSum",
     "name" : "NumConfirmed",
     "fieldName" : "NumConfirmed" 
    }, {
     "type" : "doubleSum",
     "name" : "NumRejected",
     "fieldName" : "NumRejected" 
    }, {
     "type" : "doubleSum",
     "name" : "NumFailed",
     "fieldName" : "NumFailed" 
    }, {
     "type" : "doubleSum",
     "name" : "FilledAmountUSDSum",
     "fieldName" : "FilledAmountUSDSum" 
    }, {
     "type" : "doubleSum",
     "name" : "Confirmed-Ratio",
     "fieldName" : "Confirmed-Ratio" 
    }, {
     "type" : "doubleSum",
     "name" : "Rejected-ratio",
     "fieldName" : "Rejected-ratio" 
    }, {
     "type" : "doubleSum",
     "name" : "Failed-Ratio",
     "fieldName" : "Failed-Ratio" 
   }
],
"firehose" : {
      "type" : "local",
      "baseDir" : "examples/indexing/",
      "filter" : "testepa.csv",
      "parser" : {
        "timestampSpec" : {
          "column" : "createddate",
 	  "format": "yyyy-MM-dd"
        },
        "data" : {
          "format" : "csv",
          "columns" : ["takerorg","ccypair","createddate","channel","ordertype","MarketShiftCode","TIF","NumOrders","OrderDifficultyAvg","Fill-Ratio","NoFill-Ratio","ParFill-Ratio","VolFill-Ratio","AVGFillRatio","OrderDurnAvg","Num_SweepsAvg","OrderAmtUSDAvg","OrderAmtUSDSum","PIPipAvg","PIPipMed","TotalOrderDurationMax","TotalTradeDurationMax","TotalTradeDurationavg","isOrginatingOrg","NumSweepsMax","NumTrades","NumConfirmed","NumRejected","NumFailed","FilledAmountUSDSum","Confirmed-Ratio","Rejected-ratio","Failed-Ratio"],
          "dimensions" : ["takerorg","ccypair","channel","ordertype","MarketShiftCode","TIF"]

        }
      }
    }
}


sample data:

"xxxx","GBP/USD",2014-09-02,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-0.67,0.00,0.00,1.00,0.38,0.38,631.00,5.00,829950.00,829950.00,-11.31,-11.31,631.00,423.00,423.00,0,5,14,7,4,3,312061,0.5,0.29,0.21

Kasi Sankaralingam

unread,
Dec 9, 2014, 6:55:44 PM12/9/14
to druid-de...@googlegroups.com
Thanks a lot for the help, how do i contact you on IRC, I am available on skype


On Tuesday, December 9, 2014 2:05:09 PM UTC-8, Kasi Sankaralingam wrote:

Fangjin Yang

unread,
Dec 9, 2014, 7:02:17 PM12/9/14
to druid-de...@googlegroups.com

Ping fj on irc

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-development/4dddadd8-a452-4a3b-86e7-0cca5e428a7c%40googlegroups.com.

Fangjin Yang

unread,
Dec 9, 2014, 7:03:20 PM12/9/14
to druid-de...@googlegroups.com
Hi Kasi, you can ping 'fj' on IRC

Fangjin Yang

unread,
Dec 9, 2014, 7:03:39 PM12/9/14
to druid-de...@googlegroups.com

Fangjin Yang

unread,
Dec 9, 2014, 7:12:27 PM12/9/14
to druid-de...@googlegroups.com
I also just tested ingesting the data and was successful

examples/indexing/testepa.csv:
x druid-services-0.6.164/examples/twitter/
"xxxxx","GBP/USD",2014-09-02,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-0.67,0.00,0.00,1.00,0.38,0.38,631.00,5.00,829950.00,829950.00,-11.31,-11.31,631.00,423.00,423.00,0,5,14,7,4,3,312061,0.5,0.29,0.21
"xxxx","GBP/USD",2014-09-16,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-2.16,0.00,0.00,1.00,0.91,0.91,1757.00,5.00,536828.00,536828.00,-6.50,-6.50,1757.00,944.00,944.00,"",5,7,1,6,0,488025,0.14,0.86,0
"xxxxx","GBP/USD",2014-09-15,"BA/ESP","LIMIT","Hard","GTD",1.00,-2.77,0.00,0.00,1.00,0.97,0.97,1255.00,38.00,73203750.00,73203750.00,-1.28,-1.28,1255.00,845.00,845.00,"",38,96,42,43,11,71088975,0.44,0.45,0.11
"xxxx","GBP/USD",2014-09-12,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-3.11,0.00,0.00,1.00,0.50,0.50,1528.00,9.00,3227000.00,3227000.00,-1.00,-1.00,1528.00,1432.00,1432.00,"",9,9,1,7,1,1613500,0.11,0.78,0.11
"xxxx","GBP/USD",2014-09-10,"BA/ESP","LIMIT","Hard","GTD",1.00,-0.85,0.00,0.00,1.00,0.05,0.05,3506.00,3.00,32332000.00,32332000.00,-0.70,-0.70,3506.00,77.00,77.00,"",3,3,1,2,0,1616600,0.33,0.67,0
"xxxxx","GBP/USD",2014-09-11,"BA/ESP","LIMIT","Hard","GTD",1.00,-0.57,0.00,0.00,1.00,0.06,0.06,2996.00,3.00,16097000.00,16097000.00,-0.60,-0.60,2996.00,77.00,77.00,"",3,10,2,8,0,889359,0.2,0.8,0
"xxxxx","GBP/USD",2014-09-10,"BA/ESP","LIMIT","Hard","GTD",1.00,-0.69,0.00,0.00,1.00,0.31,0.31,3000.00,13.00,16166000.00,16166000.00,-0.42,-0.42,3000.00,540.00,540.00,"",13,25,4,21,0,5011460,0.16,0.84,0
"xxxxx","GBP/USD",2014-09-08,"BA/ESP","LIMIT","Very Hard","GTD",1.00,-1.56,0.00,0.00,1.00,0.21,0.21,1004.00,2.00,12602425.00,12602425.00,-0.40,-0.40,1004.00,169.00,169.00,"",2,5,1,4,0,2643260,0.2,0.8,0
"xxxxx","GBP/USD",2014-08-21,"BA/ESP","LIMIT","Not Easy","GTD",1.00,-0.48,0.00,0.00,1.00,0.20,0.20,1259.00,1.00,833575.00,833575.00,-0.40,-0.40,1259.00,0.00,0.00,"",1,1,1,0,0,166715,1,0,0

My task file:
{
  "type" : "index",
  "dataSource" : "orderepa",
  "granularitySpec" : {
    "type" : "uniform",
    "gran" : "DAY",
    "intervals" : [ "2014-08-01T00:00:00.000Z/2014-09-30T00:00:00.000Z" ]
          "format" : "csv",

Fangjin Yang

unread,
Dec 9, 2014, 7:12:52 PM12/9/14
to druid-de...@googlegroups.com

Kasi Sankaralingam

unread,
Dec 10, 2014, 1:06:55 PM12/10/14
to druid-de...@googlegroups.com
Hi,

I exactly copied the following config file and I am still seeing the stack trace. 
I tried to reach you on IRC it did not work, are you available on skype?
Also, when I set the date format as 'yyyy-MM-dd' why does it still expect
the time stamp

Thanks,

kasi

Kasi Sankaralingam

unread,
Dec 10, 2014, 1:08:48 PM12/10/14
to druid-de...@googlegroups.com
I am seeing this exception with 'yyyy-MM-dd' format.

Caused by: java.lang.IllegalArgumentException: Invalid format: "2014-09-02T00:00:00Z" is malformed at "T00:00:00Z"
	at org.joda.time.format.DateTimeFormatter.parseDateTime(DateTimeFormatter.java:866)
	at com.metamx.common.parsers.TimestampParser$6.apply(TimestampParser.java:88)
	at com.metamx.common.parsers.TimestampParser$6.apply(TimestampParser.java:83)
	at io.druid.data.input.impl.TimestampSpec.extractTimestamp(TimestampSpec.java:51)
	at io.druid.data.input.impl.MapInputRowParser.parse(MapInputRowParser.java:64)

Fangjin Yang

unread,
Dec 10, 2014, 1:20:55 PM12/10/14
to druid-de...@googlegroups.com
Kasi: I am on IRC right now. The spec should work for the sample data you provided.

This exception you are pasting is a completely different exception. Please change your timestamp format to 'iso'.

Kasi Sankaralingam

unread,
Dec 10, 2014, 1:33:47 PM12/10/14
to druid-de...@googlegroups.com
what client do you use to irc chat, web chat does not seem to work

Kasi Sankaralingam

unread,
Dec 10, 2014, 1:34:51 PM12/10/14
to druid-de...@googlegroups.com
This is the command I am using for indexing

curl -X 'POST' -H 'Content-Type:application/json' -d @examples/indexing/fyang_index_task.json localhost:8087/druid/indexer/v1/task &

Kasi Sankaralingam

unread,
Dec 10, 2014, 1:37:01 PM12/10/14
to druid-de...@googlegroups.com
I am on irc chat, nick name kasi can you see me?

Thanks
Reply all
Reply to author
Forward
0 new messages