Problem pushing s3object from realtime node

[Andres Gomez]

Hi all,

I am currently using druid 0.7.0, but when my realtime node try push object to my local S3 (riak-cs), it can't push the object and report this:

2015-03-16_17:44:30.49674 2015-03-16T17:44:30,490 INFO [rb_monitor-2015-03-16T15:00:00.000Z-persist-n-merge] io.druid.storage.s3.S3DataSegmentPusher - Pushing S3Object [key=rbdata/rb_monitor/2015-03-16T15:00:00.000Z_2015-03-16T16:00:00.000Z/2015-03-16T15:00:00.000Z/0/index.zip, bucket=redborder, lastModified=null, dataInputStream=null, Metadata={x-amz-acl=bucket-owner-full-control, Content-Length=3146, Content-MD5=d/IMvCNLnHyzdzTNk+Txdw==, md5-hash=77f20cbc234b9c7cb37734cd93e4f177, Content-Type=application/zip}].

2015-03-16_17:44:31.85804 2015-03-16T17:44:31,856 ERROR [rb_monitor-2015-03-16T15:00:00.000Z-persist-n-merge] io.druid.segment.realtime.plumber.RealtimePlumber - Failed to persist merged index[rb_monitor]: {class=io.druid.segment.realtime.plumber.RealtimePlumber, exceptionType=class java.io.IOException, exceptionMessage=org.jets3t.service.S3ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>BEMDOVMREUPFF7OOWN9H</AWSAccessKeyId><RequestId>3D9E9D8664CD67B1</RequestId><HostId>EA8TGdR71EoEJg/rQEQIzPr8kbv8HzUTjHYop4RNXnP6mnYBd5kf6r8tqDnVbwGrlIKXl1nJ00Y=</HostId></Error>, interval=2015-03-16T15:00:00.000Z/2015-03-16T16:00:00.000Z}

2015-03-16_17:44:31.85809 java.io.IOException: org.jets3t.service.S3ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>BEMDOVMREUPFF7OOWN9H</AWSAccessKeyId><RequestId>3D9E9D8664CD67B1</RequestId><HostId>EA8TGdR71EoEJg/rQEQIzPr8kbv8HzUTjHYop4RNXnP6mnYBd5kf6r8tqDnVbwGrlIKXl1nJ00Y=</HostId></Error>

2015-03-16_17:44:31.85810 at io.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:132) ~[?:?]

2015-03-16_17:44:31.85811 at io.druid.segment.realtime.plumber.RealtimePlumber$4.doRun(RealtimePlumber.java:447) [druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85811 at io.druid.common.guava.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:40) [druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85813 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85813 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85813 at java.lang.Thread.run(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85814 Caused by: org.jets3t.service.S3ServiceException: Service Error Message.

2015-03-16_17:44:31.85814 at org.jets3t.service.S3Service.putObject(S3Service.java:2069) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85814 at io.druid.storage.s3.S3DataSegmentPusher$1.call(S3DataSegmentPusher.java:93) ~[?:?]

2015-03-16_17:44:31.85815 at io.druid.storage.s3.S3DataSegmentPusher$1.call(S3DataSegmentPusher.java:77) ~[?:?]

2015-03-16_17:44:31.85815 at com.metamx.common.RetryUtils.retry(RetryUtils.java:38) ~[druid-services-0.7.0-selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85816 at io.druid.storage.s3.S3Utils.retryS3Operation(S3Utils.java:77) ~[?:?]

2015-03-16_17:44:31.85816 at io.druid.storage.s3.S3DataSegmentPusher.push(S3DataSegmentPusher.java:75) ~[?:?]

2015-03-16_17:44:31.85816 ... 5 more

I have a jets3t.properties on the realtime classpath. This is my jets3t.properties:

s3service.s3-endpoint=s3.redborder.cluster

s3service.https-only=false

s3service.s3-endpoint-http-port=8088

s3service.disable-dns-buckets=true

But my realtime try to push de s3Object to AWS S3, instead of upload to my local s3 ...

Any idea about this?

Regards,

Andres Gomez

[Fangjin Yang]

Hi Andres, is this something new that is broken because of the update to 0.7.0? If you turn on debug logging for jets3t, do you see the endpoint being overwritten?

[Andres Gomez Ferrer]

Hi FY,

The problem isn’t new at 0.7.0 release. I had the problem on the old releases but I fixed it doing something like this:

Jets3tProperties jets3=new Jets3tProperties();
jets3.loadAndReplaceProperties(new FileInputStream("/root/Desktop/jets3t.properties"), "jets3t.properties”);

but … I think this isn’t a definitely solution.

I have run the realtime again using the debug mode on jet3st and I have could see this: 

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint=s3.amazonaws.com

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Creating object bucketName=redborder, objectKey=rbdata/rb_flow/2015-03-16T21:00:00.000Z_2015-03-16T22:00:00.000Z/2015-03-16T21:00:00.000Z/0/index.zip, storageClass=null. Content-Type=application/zip Including data? true Metadata: {x-amz-acl=bucket-owner-full-control, x-amz-storage-class=STANDARD, Content-Length=128121024, Content-MD5=ujqr11BThp/O7iEZOahI4g==, md5-hash=ba3aabd75053869fceee211939a848e2, Content-Type=application/zip} ACL: GSAccessControlList [owner=null, grants=[]]

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.disable-dns-buckets=false

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint=s3.amazonaws.com

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint-virtual-path=

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint-https-port=443

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - S3 URL: https://redborder.s3.amazonaws.com:443/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip

2015-03-17 08:24:59,078 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Performing PUT request for 'https://redborder.s3.amazonaws.com:443/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip', expecting response codes: [200,204]

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Headers: [Date: Tue, 17 Mar 2015 08:24:59 GMT, x-amz-acl: bucket-owner-full-control, x-amz-storage-class: STANDARD, Content-MD5: ujqr11BThp/O7iEZOahI4g==, Content-Type: application/zip, x-amz-meta-md5-hash: ba3aabd75053869fceee211939a848e2]

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - httpclient.retry-max=5

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Adding authorization for Access Key 'BEMDOVMREUPFF7OOWN9H'.

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint=s3.amazonaws.com

2015-03-17 08:24:59,079 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - For creating canonical string, using uri: /redborder/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip

2015-03-17 08:24:59,080 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Canonical string ('|' is a newline): PUT|ujqr11BThp/O7iEZOahI4g==|application/zip|Tue, 17 Mar 2015 08:24:59 GMT|x-amz-acl:bucket-owner-full-control|x-amz-meta-md5-hash:ba3aabd75053869fceee211939a848e2|x-amz-storage-class:STANDARD|/redborder/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip

2015-03-17 08:24:59,080 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.utils.RestUtils$ThreadSafeConnManager - Get connection: {s}->https://redborder.s3.amazonaws.com:443, timeout = 0

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Response for 'PUT'. Content-Type: application/xml, Headers: [x-amz-request-id: DCF046C10174E725, x-amz-id-2: jARjSK979X7sJigbog2LkiwfVszNP5Scq/K65FzXtsCxnkt7qdmnEKQFc1cpLUIX, Content-Type: application/xml, Transfer-Encoding: chunked, Date: Tue, 17 Mar 2015 08:25:00 GMT, Connection: close, Server: AmazonS3]

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Response entity: org.apache.http.conn.BasicManagedEntity@587b8159

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Entity length: -1

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Received response code 403; matches one of expected set ([200, 204])? false

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Error response xml: true

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Error response entity: org.apache.http.conn.BasicManagedEntity@587b8159

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Error response entity length: -1

2015-03-17 08:25:00,473 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Response '/rbdata%2Frb_flow%2F2015-03-16T21%3A00%3A00.000Z_2015-03-16T22%3A00%3A00.000Z%2F2015-03-16T21%3A00%3A00.000Z%2F0%2Findex.zip' - Received error response with XML message

2015-03-17 08:25:00,474 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.utils.RestUtils$ThreadSafeConnManager - Released connection is not reusable.

2015-03-17 08:25:00,474 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.HttpMethodReleaseInputStream - Released HttpMethod as its response data stream is fully consumed

2015-03-17 08:25:00,475 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Rethrowing as a ServiceException error in performRequest: org.jets3t.service.ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: null, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error><Code>InvalidAccessKeyId</Code><Message>The AWS Access Key Id you provided does not exist in our records.</Message><AWSAccessKeyId>BEMDOVMREUPFF7OOWN9H</AWSAccessKeyId><RequestId>DCF046C10174E725</RequestId><HostId>jARjSK979X7sJigbog2LkiwfVszNP5Scq/K65FzXtsCxnkt7qdmnEKQFc1cpLUIX</HostId></Error>

2015-03-17 08:25:00,475 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.impl.rest.httpclient.RestStorageService - Releasing HttpClient connection after error: Service Error Message.

I have seen that the endpoint and port are:

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint=s3.amazonaws.com

2015-03-17 08:24:59,077 DEBUG [rb_flow-2015-03-16T21:00:00.000Z-persist-n-merge] org.jets3t.service.Jets3tProperties - s3service.s3-endpoint-https-port=443

but in my jets3t.properties are:

s3service.s3-endpoint=s3. redborder.cluster

s3service.https-only=false

s3service.s3-endpoint-http- port=8088

s3service.disable-dns-buckets= true

any idea??

Regards and thanks FY.

Andrés Gómez

Developer  

redborder.net /  ago...@redborder.net

Phone: +34 955 60 11 60

   

Piénsalo antes de imprimir este mensaje

 

Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.

 

This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. I f you have received this email in error, please notify the sender and delete it from your system. 

En 17 de marzo de 2015 en 2:01:24, Fangjin Yang (fangj...@gmail.com ) escrito:

Hi Andres, is this something new that is broken because of the update to 0.7.0? If you turn on debug logging for jets3t, do you see the endpoint being overwritten?

On Monday, March 16, 2015 at 11:09:22 AM UTC-7, Andres Gomez wrote:

Hi all,

I am currently using druid 0.7.0, but when my realtime node try push object to my local S3 (riak-cs), it can't push the object and report this:

2015-03-16_17:44:30.49674 2015-03-16T17:44:30,490 INFO [rb_monitor-2015-03-16T15:00: 00.000Z-persist-n-merge] io.druid.storage.s3. S3DataSegmentPusher - Pushing S3Object [key=rbdata/rb_monitor/2015- 03-16T15:00:00.000Z_2015-03- 16T16:00:00.000Z/2015-03- 16T15:00:00.000Z/0/index.zip, bucket=redborder, lastModified=null, dataInputStream=null, Metadata={x-amz-acl=bucket- owner-full-control, Content-Length=3146, Content-MD5=d/IMvCNLnHyzdzTNk+ Txdw==, md5-hash= 77f20cbc234b9c7cb37734cd93e4f1 77, Content-Type=application/zip}] .

2015-03-16_17:44:31.85804 2015-03-16T17:44:31,856 ERROR [rb_monitor-2015-03-16T15:00: 00.000Z-persist-n-merge] io.druid.segment.realtime. plumber.RealtimePlumber - Failed to persist merged index[rb_monitor]: {class=io.druid.segment. realtime.plumber. RealtimePlumber, exceptionType=class java.io.IOException, exceptionMessage=org.jets3t. service.S3ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error>< Code>InvalidAccessKeyId</Code> <Message>The AWS Access Key Id you provided does not exist in our records.</Message>< AWSAccessKeyId> BEMDOVMREUPFF7OOWN9H</ AWSAccessKeyId><RequestId> 3D9E9D8664CD67B1</RequestId>< HostId>EA8TGdR71EoEJg/ rQEQIzPr8kbv8HzUTjHYop4RNXnP6m nYBd5kf6r8tqDnVbwGrlIKXl1nJ00Y =</HostId></Error>, interval=2015-03-16T15:00:00. 000Z/2015-03-16T16:00:00.000Z}

2015-03-16_17:44:31.85809 java.io.IOException: org.jets3t.service. S3ServiceException: Service Error Message. -- ResponseCode: 403, ResponseStatus: Forbidden, XML Error Message: <?xml version="1.0" encoding="UTF-8"?><Error>< Code>InvalidAccessKeyId</Code> <Message>The AWS Access Key Id you provided does not exist in our records.</Message>< AWSAccessKeyId> BEMDOVMREUPFF7OOWN9H</ AWSAccessKeyId><RequestId> 3D9E9D8664CD67B1</RequestId>< HostId>EA8TGdR71EoEJg/ rQEQIzPr8kbv8HzUTjHYop4RNXnP6m nYBd5kf6r8tqDnVbwGrlIKXl1nJ00Y =</HostId></Error>

2015-03-16_17:44:31.85810 at io.druid.storage.s3. S3DataSegmentPusher.push( S3DataSegmentPusher.java:132) ~[?:?]

2015-03-16_17:44:31.85811 at io.druid.segment.realtime. plumber.RealtimePlumber$4. doRun(RealtimePlumber.java: 447) [druid-services-0.7.0- selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85811 at io.druid.common.guava. ThreadRenamingRunnable.run( ThreadRenamingRunnable.java: 40) [druid-services-0.7.0- selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85813 at java.util.concurrent. ThreadPoolExecutor.runWorker( Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85813 at java.util.concurrent. ThreadPoolExecutor$Worker.run( Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85813 at java.lang.Thread.run(Unknown Source) [?:1.7.0_03]

2015-03-16_17:44:31.85814 Caused by: org.jets3t.service. S3ServiceException: Service Error Message.

2015-03-16_17:44:31.85814 at org.jets3t.service.S3Service. putObject(S3Service.java:2069) ~[druid-services-0.7.0- selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85814 at io.druid.storage.s3. S3DataSegmentPusher$1.call( S3DataSegmentPusher.java:93) ~[?:?]

2015-03-16_17:44:31.85815 at io.druid.storage.s3. S3DataSegmentPusher$1.call( S3DataSegmentPusher.java:77) ~[?:?]

2015-03-16_17:44:31.85815 at com.metamx.common.RetryUtils. retry(RetryUtils.java:38) ~[druid-services-0.7.0- selfcontained.jar:0.7.0]

2015-03-16_17:44:31.85816 at io.druid.storage.s3.S3Utils. retryS3Operation(S3Utils.java: 77) ~[?:?]

2015-03-16_17:44:31.85816 at io.druid.storage.s3. S3DataSegmentPusher.push( S3DataSegmentPusher.java:75) ~[?:?]

2015-03-16_17:44:31.85816 ... 5 more

I have a jets3t.properties on the realtime classpath. This is my jets3t.properties:

s3service.s3-endpoint=s3. redborder.cluster

s3service.https-only=false

s3service.s3-endpoint-http- port=8088

s3service.disable-dns-buckets= true

But my realtime try to push de s3Object to AWS S3, instead of upload to my local s3 ...

Any idea about this?

Regards,

Andres Gomez

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/npI24dkxTUw/unsubscribe .
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com .
To post to this group, send email to druid...@googlegroups.com .
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/bce8de37-8a2e-4f15-bf76-1b94b950e58d%40googlegroups.com .
For more options, visit https://groups.google.com/d/optout .

[charles.allen]

I've looked into this a bit, ant it seems that you can specify a classpath in the overlord and include a jets3t.properties in the classpath to set how jets3t handles endpoints.

The major downside is that the classpath is not simply append able currently. Meaning that if you simply want to specify a jets3t.properties or log4j2.xml for the peons, you cannot do so without specifying the entire classpath.

Here's an example of the parameter I set in the overlord. Only the first entry is my custom entry. The rest was grabbed from the log of a prior task run where the classpath option was NOT overridden:

-Ddruid.indexer.runner.classpath="/Users/charlesallen/bin/wrk/classpath:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/ant-javafx.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/dt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/javafx-doclet.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/javafx-mx.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/jconsole.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/sa-jdi.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/lib/tools.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/deploy.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/htmlconverter.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/javaws.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jfxrt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/management-agent.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/plugin.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/dnsns.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/localedata.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/sunec.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/sunjce_provider.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar:/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home/jre/lib/ext/zipfs.jar:/Users/charlesallen/src/druid/services/target/classes:/Users/charlesallen/src/druid/common/target/classes:/Users/charlesallen/.m2/repository/com/metamx/java-util/0.26.14/java-util-0.26.14.jar:/Users/charlesallen/.m2/repository/org/slf4j/slf4j-api/1.6.4/slf4j-api-1.6.4.jar:/Users/charlesallen/.m2/repository/net/sf/opencsv/opencsv/2.3/opencsv-2.3.jar:/Users/charlesallen/.m2/repository/io/druid/druid-api/0.3.4/druid-api-0.3.4.jar:/Users/charlesallen/.m2/repository/commons-codec/commons-codec/1.7/commons-codec-1.7.jar:/Users/charlesallen/.m2/repository/org/apache/commons/commons-dbcp2/2.0.1/commons-dbcp2-2.0.1.jar:/Users/charlesallen/.m2/repository/org/apache/commons/commons-pool2/2.2/commons-pool2-2.2.jar:/Users/charlesallen/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/Users/charlesallen/.m2/repository/commons-pool/commons-pool/1.6/commons-pool-1.6.jar:/Users/charlesallen/.m2/repository/org/skife/config/config-magic/0.9/config-magic-0.9.jar:/Users/charlesallen/.m2/repository/org/hibernate/hibernate-validator/5.1.3.Final/hibernate-validator-5.1.3.Final.jar:/Users/charlesallen/.m2/repository/org/jboss/logging/jboss-logging/3.1.3.GA/jboss-logging-3.1.3.GA.jar:/Users/charlesallen/.m2/repository/com/fasterxml/classmate/1.0.0/classmate-1.0.0.jar:/Users/charlesallen/.m2/repository/javax/el/javax.el-api/3.0.0/javax.el-api-3.0.0.jar:/Users/charlesallen/.m2/repository/javax/validation/validation-api/1.1.0.Final/validation-api-1.1.0.Final.jar:/Users/charlesallen/.m2/repository/com/google/guava/guava/16.0.1/guava-16.0.1.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.4.4/jackson-core-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/core/jackson-annotations/2.4.4/jackson-annotations-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.4.4/jackson-databind-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-guava/2.4.4/jackson-datatype-guava-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/datatype/jackson-datatype-joda/2.4.4/jackson-datatype-joda-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/dataformat/jackson-dataformat-smile/2.4.4/jackson-dataformat-smile-2.4.4.jar:/Users/charlesallen/.m2/repository/com/google/inject/guice/4.0-beta/guice-4.0-beta.jar:/Users/charlesallen/.m2/repository/aopalliance/aopalliance/1.0/aopalliance-1.0.jar:/Users/charlesallen/.m2/repository/com/google/inject/extensions/guice-multibindings/4.0-beta/guice-multibindings-4.0-beta.jar:/Users/charlesallen/.m2/repository/org/jdbi/jdbi/2.32/jdbi-2.32.jar:/Users/charlesallen/.m2/repository/joda-time/joda-time/2.6/joda-time-2.6.jar:/Users/charlesallen/.m2/repository/com/google/code/findbugs/jsr305/2.0.1/jsr305-2.0.1.jar:/Users/charlesallen/.m2/repository/org/apache/logging/log4j/log4j-api/2.1/log4j-api-2.1.jar:/Users/charlesallen/.m2/repository/org/apache/logging/log4j/log4j-core/2.1/log4j-core-2.1.jar:/Users/charlesallen/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.1/log4j-slf4j-impl-2.1.jar:/Users/charlesallen/.m2/repository/com/lmax/disruptor/3.3.0/disruptor-3.3.0.jar:/Users/charlesallen/src/druid/server/target/classes:/Users/charlesallen/src/druid/processing/target/classes:/Users/charlesallen/.m2/repository/com/metamx/bytebuffer-collections/0.1.5/bytebuffer-collections-0.1.5.jar:/Users/charlesallen/.m2/repository/com/metamx/extendedset/1.3.8/extendedset-1.3.8.jar:/Users/charlesallen/.m2/repository/org/roaringbitmap/RoaringBitmap/0.4.5/RoaringBitmap-0.4.5.jar:/Users/charlesallen/.m2/repository/com/metamx/emitter/0.3.0/emitter-0.3.0.jar:/Users/charlesallen/.m2/repository/com/ning/compress-lzf/1.0.3/compress-lzf-1.0.3.jar:/Users/charlesallen/.m2/repository/com/google/protobuf/protobuf-java/2.5.0/protobuf-java-2.5.0.jar:/Users/charlesallen/.m2/repository/com/ibm/icu/icu4j/4.8.1/icu4j-4.8.1.jar:/Users/charlesallen/.m2/repository/org/mozilla/rhino/1.7R4/rhino-1.7R4.jar:/Users/charlesallen/.m2/repository/org/mapdb/mapdb/1.0.6/mapdb-1.0.6.jar:/Users/charlesallen/src/druid/aws-common/target/classes:/Users/charlesallen/.m2/repository/com/amazonaws/aws-java-sdk/1.8.11/aws-java-sdk-1.8.11.jar:/Users/charlesallen/.m2/repository/com/amazonaws/aws-java-sdk-core/1.8.11/aws-java-sdk-core-1.8.11.jar:/Users/charlesallen/.m2/repository/com/metamx/http-client/1.0.0/http-client-1.0.0.jar:/Users/charlesallen/.m2/repository/io/netty/netty/3.9.5.Final/netty-3.9.5.Final.jar:/Users/charlesallen/.m2/repository/com/metamx/server-metrics/0.1.0/server-metrics-0.1.0.jar:/Users/charlesallen/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/Users/charlesallen/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar:/Users/charlesallen/.m2/repository/javax/inject/javax.inject/1/javax.inject-1.jar:/Users/charlesallen/.m2/repository/org/glassfish/javax.el/3.0.0/javax.el-3.0.0.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-framework/2.7.0/curator-framework-2.7.0.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-client/2.7.0/curator-client-2.7.0.jar:/Users/charlesallen/.m2/repository/org/apache/zookeeper/zookeeper/3.4.6/zookeeper-3.4.6.jar:/Users/charlesallen/.m2/repository/log4j/log4j/1.2.16/log4j-1.2.16.jar:/Users/charlesallen/.m2/repository/jline/jline/0.9.94/jline-0.9.94.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-x-discovery/2.7.0/curator-x-discovery-2.7.0.jar:/Users/charlesallen/.m2/repository/org/apache/curator/curator-recipes/2.7.0/curator-recipes-2.7.0.jar:/Users/charlesallen/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-json-provider/2.4.4/jackson-jaxrs-json-provider-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-base/2.4.4/jackson-jaxrs-base-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb-annotations/2.4.4/jackson-module-jaxb-annotations-2.4.4.jar:/Users/charlesallen/.m2/repository/com/fasterxml/jackson/jaxrs/jackson-jaxrs-smile-provider/2.4.4/jackson-jaxrs-smile-provider-2.4.4.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/jersey-server/1.17.1/jersey-server-1.17.1.jar:/Users/charlesallen/.m2/repository/asm/asm/3.1/asm-3.1.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/jersey-core/1.17.1/jersey-core-1.17.1.jar:/Users/charlesallen/.m2/repository/com/google/inject/extensions/guice-servlet/4.0-beta/guice-servlet-4.0-beta.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/contribs/jersey-guice/1.17.1/jersey-guice-1.17.1.jar:/Users/charlesallen/.m2/repository/com/sun/jersey/jersey-servlet/1.17.1/jersey-servlet-1.17.1.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-server/9.2.5.v20141112/jetty-server-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/javax/servlet/javax.servlet-api/3.1.0/javax.servlet-api-3.1.0.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-http/9.2.5.v20141112/jetty-http-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-io/9.2.5.v20141112/jetty-io-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-proxy/9.2.5.v20141112/jetty-proxy-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-client/9.2.5.v20141112/jetty-client-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-util/9.2.5.v20141112/jetty-util-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/io/tesla/aether/tesla-aether/0.0.5/tesla-aether-0.0.5.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-spi/0.9.0.M2/aether-spi-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-util/0.9.0.M2/aether-util-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-impl/0.9.0.M2/aether-impl-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-connector-file/0.9.0.M2/aether-connector-file-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/io/tesla/aether/aether-connector-okhttp/0.0.9/aether-connector-okhttp-0.0.9.jar:/Users/charlesallen/.m2/repository/com/squareup/okhttp/okhttp/1.0.2/okhttp-1.0.2.jar:/Users/charlesallen/.m2/repository/org/apache/maven/wagon/wagon-provider-api/2.4/wagon-provider-api-2.4.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-aether-provider/3.1.1/maven-aether-provider-3.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-model/3.1.1/maven-model-3.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-model-builder/3.1.1/maven-model-builder-3.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-repository-metadata/3.1.1/maven-repository-metadata-3.1.1.jar:/Users/charlesallen/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.15/plexus-utils-3.0.15.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-settings-builder/3.1.1/maven-settings-builder-3.1.1.jar:/Users/charlesallen/.m2/repository/org/codehaus/plexus/plexus-interpolation/1.19/plexus-interpolation-1.19.jar:/Users/charlesallen/.m2/repository/org/apache/maven/maven-settings/3.1.1/maven-settings-3.1.1.jar:/Users/charlesallen/.m2/repository/org/eclipse/aether/aether-api/0.9.0.M2/aether-api-0.9.0.M2.jar:/Users/charlesallen/.m2/repository/org/antlr/antlr4-runtime/4.0/antlr4-runtime-4.0.jar:/Users/charlesallen/.m2/repository/org/abego/treelayout/org.abego.treelayout.core/1.0.1/org.abego.treelayout.core-1.0.1.jar:/Users/charlesallen/.m2/repository/net/spy/spymemcached/2.11.4/spymemcached-2.11.4.jar:/Users/charlesallen/.m2/repository/net/jpountz/lz4/lz4/1.3.0/lz4-1.3.0.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-servlet/9.2.5.v20141112/jetty-servlet-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-security/9.2.5.v20141112/jetty-security-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-servlets/9.2.5.v20141112/jetty-servlets-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/org/eclipse/jetty/jetty-continuation/9.2.5.v20141112/jetty-continuation-9.2.5.v20141112.jar:/Users/charlesallen/.m2/repository/com/ircclouds/irc/irc-api/1.0-0011/irc-api-1.0-0011.jar:/Users/charlesallen/.m2/repository/com/maxmind/geoip2/geoip2/0.4.0/geoip2-0.4.0.jar:/Users/charlesallen/.m2/repository/com/maxmind/maxminddb/maxminddb/0.2.0/maxminddb-0.2.0.jar:/Users/charlesallen/.m2/repository/com/google/http-client/google-http-client/1.15.0-rc/google-http-client-1.15.0-rc.jar:/Users/charlesallen/.m2/repository/xpp3/xpp3/1.1.4c/xpp3-1.1.4c.jar:/Users/charlesallen/.m2/repository/com/google/http-client/google-http-client-jackson2/1.15.0-rc/google-http-client-jackson2-1.15.0-rc.jar:/Users/charlesallen/.m2/repository/org/apache/derby/derby/10.11.1.1/derby-10.11.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/derby/derbynet/10.11.1.1/derbynet-10.11.1.1.jar:/Users/charlesallen/.m2/repository/org/apache/derby/derbyclient/10.11.1.1/derbyclient-10.11.1.1.jar:/Users/charlesallen/src/druid/indexing-hadoop/target/classes:/Users/charlesallen/.m2/repository/commons-io/commons-io/2.0.1/commons-io-2.0.1.jar:/Users/charlesallen/.m2/repository/net/java/dev/jets3t/jets3t/0.9.1/jets3t-0.9.1.jar:/Users/charlesallen/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.12/jackson-core-asl-1.9.12.jar:/Users/charlesallen/.m2/repository/javax/activation/activation/1.1.1/activation-1.1.1.jar:/Users/charlesallen/.m2/repository/mx4j/mx4j/3.0.2/mx4j-3.0.2.jar:/Users/charlesallen/.m2/repository/javax/mail/mail/1.4.7/mail-1.4.7.jar:/Users/charlesallen/.m2/repository/org/bouncycastle/bcprov-jdk15/1.46/bcprov-jdk15-1.46.jar:/Users/charlesallen/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar:/Users/charlesallen/.m2/repository/org/apache/httpcomponents/httpclient/4.2/httpclient-4.2.jar:/Users/charlesallen/.m2/repository/org/apache/httpcomponents/httpcore/4.2/httpcore-4.2.jar:/Users/charlesallen/src/druid/indexing-service/target/classes:/Users/charlesallen/.m2/repository/io/airlift/airline/0.6/airline-0.6.jar:/Applications/IntelliJ IDEA 14 CE.app/Contents/lib/idea_rt.jar"

My jets3t.properties is the same as the one you posted, and I get the following when it is run:

org.jets3t.service.ServiceException: Request Error: s3.redborder.cluster: nodename nor servname provided, or not known

Which cis actually the error I was hoping for. It means the property properly propogates.

I'll be filing an issue about this to allow easy usage of custom properties or log4j.xml stuff.

Sorry for the confusion.

Let me know if this workaround suffices.

Regards,

Charles Allen

On Monday, March 16, 2015 at 11:09:22 AM UTC-7, Andres Gomez wrote:

[Barney Desmond]

Hi,

On Wednesday, 18 March 2015 04:13:44 UTC+11, charles.allen wrote:

Which cis actually the error I was hoping for. It means the property properly propogates.

I'll be filing an issue about this to allow easy usage of custom properties or log4j.xml stuff.

Sorry for the confusion.

Let me know if this workaround suffices.

Is there any news on this topic? We'd also like to make use of our own S3-alike installation, but won't necessarily be able to support any workarounds, so a "proper" way to do this would be much appreciated as well.


Regards,
Barney Desmond

[Andres Gomez Ferrer]

Hi Barney,

I finally solved the problem change the properties file’s position on my classpath, some like this:

This doesn’t work :

java -Xmn400420k -Xloggc:/var/log/druid/realtime/gc.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -server -Xmx800841k -Xms800841k -Djava.io.tmpdir=/opt/rb/var/druid/realtime -XX:OnOutOfMemoryError=/opt/rb/var/sv/druid_realtime/kill -Dlog4j.configuration=file:/tmp/druid/realtime/log4j.properties -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djute.maxbuffer=8388608 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDetails -Djava.rmi.server.hostname=10.0.153.153 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9082 -cp /opt/rb/var/druid/app/druid-services.jar:/opt/rb/var/druid/app/postgresql.jar:/tmp/druid/realtime io.druid.cli.Main server realtime

This work:

java -Xmn400420k -Xloggc:/var/log/druid/realtime/gc.log -verbose:gc -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintCommandLineFlags -XX:+PrintFlagsFinal -server -Xmx800841k -Xms800841k -Djava.io.tmpdir=/opt/rb/var/druid/realtime -XX:OnOutOfMemoryError=/opt/rb/var/sv/druid_realtime/kill -Dlog4j.configuration=file:/tmp/druid/realtime/log4j.properties -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Djute.maxbuffer=8388608 -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDetails -Djava.rmi.server.hostname=10.0.153.153 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9082 -cp /tmp/druid/realtime:/opt/rb/var/druid/app/druid-services.jar:/opt/rb/var/druid/app/postgresql.jar io.druid.cli.Main server realtime

I have my jets3t.properties inside this directory  /tmp/druid/realtime

Regards,

Andrés Gómez

Developer 

redborder.net / ago...@redborder.net

Phone: +34 955 60 11 60

  

Piénsalo antes de imprimir este mensaje

 

Este correo electrónico, incluidos sus anexos, se dirige exclusivamente a su destinatario. Contiene información CONFIDENCIAL cuya divulgación está prohibida por la ley o puede estar sometida a secreto profesional. Si ha recibido este mensaje por error, le rogamos nos lo comunique inmediatamente y proceda a su destrucción.

 

This email, including attachments, is intended exclusively for its addressee. It contains information that is CONFIDENTIAL whose disclosure is prohibited by law and may be covered by legal privilege. If you have received this email in error, please notify the sender and delete it from your system. 

En 20 de marzo de 2015 en 7:49:59, Barney Desmond (barney...@gmail.com) escrito:

--
You received this message because you are subscribed to a topic in the Google Groups "Druid User" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/druid-user/npI24dkxTUw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/abf0d2ff-91be-4125-a684-e221e20c64a8%40googlegroups.com.