No data in S3

[Nicholas McKoy]

Hey all,

For some reason no data is getting pushed to S3. I have this working in another environment but when i moved my druid cluster to a more permanent environment my s3 bucket is still empty. Realtime nodes arent showing logs about segment at all either. This is my common.runtime.properties:

user.timezone=UTC

file.encoding=UTF-8

java.io.tmpdir=/data

# Extensions

druid.extensions.coordinates=["io.druid.extensions:mysql-metadata-storage", "io.druid.extensions:druid-s3-extensions:0.8.0", "io.druid.extensions:druid-kafka-eight:0.8.0"]

# Zookeeper

druid.zk.service.host={zk}

druid.zk.paths.base={zk-paths}

druid.discovery.curator.path=/prod/discovery

# Request logging, monitoring, and metrics

druid.request.logging.type=emitter

druid.request.logging.feed=druid_requests

druid.monitoring.monitors=["com.metamx.metrics.JvmMonitor"]

druid.emitter=noop

# Metadata storage

druid.metadata.storage.type=mysql

druid.metadata.storage.connector.connectURI={db-connectURI}

druid.metadata.storage.connector.user={db-user}

druid.metadata.storage.connector.password={db-password}

# Deep storage

druid.storage.type=s3

druid.storage.bucket={bucket}

druid.s3.accessKey={access}

druid.s3.secretKey={secret}

The s3 extensions are getting pulled so I'm really not sure why theres still no data. Any help is appreciated.

Thanks,

[Nicholas McKoy]

any ideas?

[Nicholas McKoy]

this is also my specfile attached

On Thursday, October 8, 2015 at 10:41:05 AM UTC-4, Nicholas McKoy wrote:

[Gian Merlino]

Hey Nicholas,

Do you know if that realtime node is actually ingesting data? If you don't see any logs about it building and merging segments, it may not actually be ingesting anything. You could also add the RealtimeMetricsMonitor and set the emitter to "logging" in order to see specific metrics about how many events are being processed.

Another thing to try is that you currently have your rejectionPolicy set to "messageTime". This can cause things to not be handed off properly if you don't have a continuous stream of data. You may have better luck with "serverTime".

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/c9b800f5-e41e-404f-951c-a85b422729ad%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Nicholas McKoy]

ill change it to serverTime and and turn the realtimemetrics on and post back here.

Thanks,

On Thursday, October 8, 2015 at 10:41:05 AM UTC-4, Nicholas McKoy wrote:

[Nicholas McKoy]

Hey Gian,

So messageTime works perfectly at the moment. I do have a continuous stream of data coming in so that helps. I can also see a bunch of data in s3. Thanks

But when i use serverTime, everything gets thrown out. i now have my windowPeriod at 10m (default). can you elaborate more on this? The data coming in is at real time and the timestamps for that data is current time.

And what does "hand-off" really mean? is that just segment persistence?

Thanks for your help dude.

On Thursday, October 8, 2015 at 10:41:05 AM UTC-4, Nicholas McKoy wrote:

[Gian Merlino]

Hey Nicholas,

"handoff" means uploading a segment to deep storage, inserting a row for that segment in the metadata store, and waiting for a historical node to load that segment.

You shouldn't be getting any thrownAway counts if all your timestamps are within your windowPeriod. If your data is being entirely thrown out then perhaps either the server clock is wrong, or something is off in how the times are being parsed (mismatched timezone maybe).

--

You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/e7a03359-4a65-4337-81bc-65363013f48e%40googlegroups.com.