Druid can't parse timestamp after update to 0.7

[Lukáš Havrlant]

Hi!

We updated Druid from 0.6 to 0.7 and Tranquility from 0.3.0 to 0.4.0. We didn't change our code. But now Druid rejects all messages because of unparseable timestamp. Can you think of anything that could cause it? Maybe some JSON issue? Everything was fine before the update. Is it possible to see the messages with unparseable timestamp? 

LH

[Xavier Léauté]

Hi Lukáš, The latest tranquility we released is 0.3.5, 0.4.0 doesn't
exist yet :) Can you share a bit more information about your setup?
Do you have an example json with the timestamps that are being
rejected? Can you also share what your realtime task config looks
like?

Thanks!

> --
> You received this message because you are subscribed to the Google Groups
> "Druid User" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to druid-user+...@googlegroups.com.
> To post to this group, send email to druid...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/druid-user/5181a2d0-a1f3-4683-9103-5e4058188299%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gian Merlino]

Hi Lukáš,

Are you getting any exceptions on either the tranquility side or the druid side (in the task logs, specifically)? Are you overriding the TimestampSpec or are you using the default?

[Prajwal Tuladhar]

https://github.com/metamx/tranquility/tree/v0.4.0

There seems to be v.0.4.0 tag which is also available in maven central http://search.maven.org/#artifactdetails|com.metamx|tranquility_2.10|0.4.0|jar

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/f0ab2327-26eb-4dab-a596-5004319d2f63%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

--

Cheers,

Praj

[Xavier Léauté]

Ah yes, my bad, I didn't see it since it wasn't merged to master yet.
0.3.5 should work with 0.7 as well, but Gian can provide more
information.

> https://groups.google.com/d/msgid/druid-user/CAM5z5B81p4%2BucqQEis2w0NTnBCxzb8znhcgsWKWTt2%3DopczDmg%40mail.gmail.com.

[Lukáš Havrlant]

I cannot show you the JSON that failed because I do not know how to retrieve it. I can send you only the JSON that we are reading in our tranquility app but not the message that is send to Druid. And the input messages are really OK. Is there a way how to print the message that was failed to ingest? The exact error in the task log is:

com.metamx.common.parsers.ParseException: Unparseable timestamp found!
	at io.druid.data.input.impl.MapInputRowParser.parse(MapInputRowParser.java:55)

...

Caused by: java.lang.NullPointerException: Null timestamp in input: {keys=[floorPrice, responseTime, providerPrice, ibbSiteId, serverId, dspProvider, dimensionsWidth, h...
	at io.druid.data.input.impl.MapInputRowParser.parse(MapInputRowParser.java:46)
	... 52 more

I used tcpdump and it seems that our tranq app doesn't send the messages at all, just the keys, which is weird. I didn't see any JSON in the tcp dump. Part of the tcpdump (tcpdump -i eth0 -A):

..'
..H.x.............s}W.....
I'..I$..asChosenBid","userInfoStat","naturalKey","currency","impId","homeCountry","ibbInstallId","timestamp"]},{"keys":["floorPrice","responseTime","providerPrice","ibbSiteId","serverId","dspProvider","dimensionsWidth","hasResponse","accountId","hostname","userInfoRegion","dimensionsHeight","publisherPrice","adomain","publisherCurrency","pageImpressionKey","hasChosenBid","userInfoStat","naturalKey","currency","impId","homeCountry","ibbInstallId","timestamp"]},{"keys":["floorPrice","responseTime","providerPrice","ibbSiteId","serverId","dspProvider","dimensionsWidth","hasResponse","accountId","hostname","userInfoRegion","dimensionsHeight","publisherPrice","adomain","publisherCurrency","pageImpressionKey","hasChosenBid","userInfoStat","naturalKey","currency","impId","homeCountry","ibbInstallId","timestamp"]},{"keys":["floorPrice","responseTime","providerPrice","ibbSiteId","serverId","dspProvider","dimensionsWidth","hasResponse","accountId","hostname","userInfoRegion","dimensionsHeight","publisherPrice","adomain","publisherCurrency","pageImpressionKey","hasChosenBid","userInfoStat","naturalKey","currency","impId","homeCountry","ibbInstallId","timestamp"]},{"keys":["floorPrice","responseTime","providerPrice","ibbSiteId","serverId","dspProvider","dimensionsWidth","hasResponse","accountId","hostname","userInfoRegion","dimensionsHeight","publisherPrice","adomain","publisherCurrency","pageImpressionKey","hasChosenBid","userInfoStat","natural
07:32:47.388444 IP charger01.49016 > druidmiddlemanager02.8093: Flags [.], seq 15929:17377, ack 1, win 115, options [nop,nop,TS val 1227297702 ecr 1227157896], length 1448
E...h^@.@...
..'


The code responsible for sending messages in tranquility is quite simple:

packetizer.send(json);

and the json is SimpleJSON. We use the default timestampSpec.

[Lukáš Havrlant]

We also get a lot of these warnings:

 WARN  com.metamx.tranquility.finagle.FutureRetry$ - Transient error, will try again in 1522 ms

and these errors:

Caused by: java.io.IOException: Failed to send 43 events to task[index_realtime_ssp-auction_2015-03-20T08:00:00.000Z_1_1_mcipbmkh]: 500 Server Error

at com.metamx.tranquility.druid.DruidBeam$$anonfun$4$$anonfun$apply$4$$anonfun$apply$6$$anonfun$apply$7$$anonfun$apply$8.apply(DruidBeam.scala:134) ~[tranquility_2.10-0.3.1.jar:0.3.1]

at com.metamx.tranquility.druid.DruidBeam$$anonfun$4$$anonfun$apply$4$$anonfun$apply$6$$anonfun$apply$7$$anonfun$apply$8.apply(DruidBeam.scala:121) ~[tranquility_2.10-0.3.1.jar:0.3.1]

(We tried go back to tranq 0.3.1) No errors/warns in middle manager or overlord. In task log just "Unparseable timestamp found!" errors.

On Thursday, 19 March 2015 22:34:38 UTC+1, Lukáš Havrlant wrote:

[Nishant Bangarwa]

Hi Lukas, 

one thing to note when migrating from druid 0.6 to 0.7 is that druid 0.7 is now case sensitive, 

do you happen to have mixed casing in your timestamp column name, in ingestion spec or in the data being ingested ?

If that is the case then can you try lowercasing your dim names and see if that works or using the same mixed casing convention in both ingestion spec as well as data being ingested ? 

Also If you can post the task spec file generated by tranquility (will be printed in task logs ) and a sample JSON containing any dummy data that you are sending it will be helpful.  

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/b2862f18-6afa-42b9-8bb9-a5c6a2f95ec4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Nishant Software Engineer | METAMARKETS m  +91-9729200044 nishant....@metamarkets.com

[Lukáš Havrlant]

OK, We fixed it. I don't really know why but now I had to inject my own objectWriter to the beam. Now it's working with all versions of tranq. 

But still -- is there a way how to print raw messages that Druid is ingesting? 

On Thursday, 19 March 2015 22:34:38 UTC+1, Lukáš Havrlant wrote:

[Gian Merlino]

If you set com.metamx.tranquility's log level to "trace" you'll see the requests and responses to and from Druid.

It's interesting that the default objectWriter worked with one version of tranquility and not with another. The default one just uses Jackson with the Joda and Scala modules registered, so it should work on most POJOs, Scala case classes, Java and Scala Maps, and anything with Jackson annotations. Fancier objects usually need a custom objectWriter.