Server connection failed: while using gpcheckcloud utility

[Yogesh Jadhav]

Hi Everyone,

We are implementing external tables on Amazon S3 server. I'm getting below error while checking the configuration using gpcheckcloud utility.

Also, I have not made any changes in config file generated by gpcheckcloud utility except my secret ID and access ID.

Command String :

gpcheckcloud -c "s3://s3.console.aws.amazon.com/s3/buckets/greenplumext config=./s3.conf"

Error:

$ gpcheckcloud -c "s3://s3.console.aws.amazon.com/s3/buckets/greenplumext config=./s3.conf"

[E]#0#(2A4D4720)../../src/s3restful_service.cpp:126  Server connection failed: Couldn't resolve host name

[E]#0#(2A4D4720)../../src/gpreader.cpp:110  reader_init caught S3ResolveError: reader_init caught a S3ResolveError exception: Server connection failed: Couldn't resolve host name, Function: performCurl, File: ../../src/s3restful_service.cpp(126). 

Failed. Please check the arguments and configuration file.

I'm able ping and traceroute to s3.console.aws.amazon.com successfully.

ping:

$ ping s3.console.aws.amazon.com

PING console.ap-south-1.amazonaws.com (52.95.80.34) 56(84) bytes of data.

64 bytes from ap-south-1.console.aws.amazon.com (52.95.80.34): icmp_seq=1 ttl=240 time=20.5 ms

64 bytes from ap-south-1.console.aws.amazon.com (52.95.80.34): icmp_seq=2 ttl=240 time=24.3 ms

^C

--- console.ap-south-1.amazonaws.com ping statistics ---

2 packets transmitted, 2 received, 0% packet loss, time 1230ms

rtt min/avg/max/mdev = 20.529/22.431/24.334/1.908 ms

traceroute:

$ traceroute s3.console.aws.amazon.com

traceroute to s3.console.aws.amazon.com (52.95.84.36), 30 hops max, 60 byte packets

 1  192.168.2.1 (192.168.2.1)  3.622 ms  4.087 ms  4.779 ms

 2  * * *

 3  10.229.0.29 (10.229.0.29)  17.653 ms  19.594 ms  21.464 ms

 4  * * *

 5  14.141.24.249.static-hyderabad.tcl.net.in (14.141.24.249)  31.111 ms 14.141.24.145.static-hyderabad.tcl.net.in (14.141.24.145)  32.925 ms 14.141.24.169.static-hyderabad.tcl.net.in (14.141.24.169)  35.738 ms

 6  * * *

 7  115.114.89.118.static-Mumbai.vsnl.net.in (115.114.89.118)  43.598 ms  41.348 ms  39.756 ms

 8  * * *

 9  52.95.66.129 (52.95.66.129)  29.197 ms 52.95.66.107 (52.95.66.107)  21.266 ms 52.95.66.129 (52.95.66.129)  17.535 ms

10  52.95.67.196 (52.95.67.196)  22.906 ms 52.95.67.192 (52.95.67.192)  21.173 ms  22.388 ms

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *

20  * * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * * *

28  * * *

29  * * *

30  * * *

1. Is there anything I'm missing here?

2. How can I fix this issue?

 

--

Regards,

Yogesh Jadhav

[Jasper Li]

What's the version parameter in your s3.conf?

You can try to add 'debug_curl = true' to find the exact url that curl try to access.

The preferred url should contain the region information, like 's3-us-west-2.amazonaws.com' if you use default version (version = 1);

Best wishes

Jasper

--

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.
To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

[Yogesh Jadhav]

Hi Jasper,

Thanks for your response.

I have created bucket in Asia Pacific (Mumbai) region and I have corrected my url to s3://s3-ap-south-1.amazonaws.com/bucket/greenplumext with version 4 as per this link.

But still, I'm facing the below issue.

AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint.

I have enabled debug in the config file as well. Here is debug information:

$ gpcheckcloud -c "s3://s3-ap-south-1.amazonaws.com/bucket/greenplumext config=/home/gpadmin/s3.conf"

*   Trying 52.219.62.41...

* TCP_NODELAY set

* Connected to s3-ap-south-1.amazonaws.com (52.219.62.41) port 443 (#0)

* ALPN, offering http/1.1

* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH

* successfully set certificate verify locations:

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-SHA

* ALPN, server did not agree to a protocol

* Server certificate:

*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3.ap-south-1.amazonaws.com

*  start date: Jul 12 00:00:00 2017 GMT

*  expire date: May  2 12:00:00 2018 GMT

*  subjectAltName: host "s3-ap-south-1.amazonaws.com" matched cert's "s3-ap-south-1.amazonaws.com"

*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2

*  SSL certificate verify ok.

> GET /bucket/?prefix=greenplumext HTTP/1.1

Host: s3-ap-south-1.amazonaws.com

Accept: */*

Authorization: AWS4-HMAC-SHA256 Credential=+Lm8k/joKlQr+qMughFGt4A9ptrGMl3izm1GGuiG/20171118//s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=e3eef4825018ecdf366772bdf5e407ca3b6c2e0f670026f8da86f820aa3346aa

x-amz-date: 20171118T145510Z

x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

< HTTP/1.1 301 Moved Permanently

< x-amz-bucket-region: us-east-1

< x-amz-request-id: DD2E7212AB582E25

< x-amz-id-2: e6HY8crXMpl+mWtCJhafQa5y2UF1Ico3DDopSC/o/uG051wWuHTrXSkAROAx4VKxdeAkFrKGI2M=

< Content-Type: application/xml

< Transfer-Encoding: chunked

< Date: Sat, 18 Nov 2017 14:59:40 GMT

< Server: AmazonS3

< 

* Closing connection 0

[E]#0#(7E21B720)../../src/s3interface.cpp:300  AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

[E]#0#(7E21B720)../../src/gpreader.cpp:110  reader_init caught S3LogicError: reader_init caught a S3LogicError exception: AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., Function: listBucket, File: ../../src/s3interface.cpp(300). 

Failed. Please check the arguments and configuration file.

Thanks,

YJ

[Jasper Li]

What's the url of the file that is displayed in AWS console? You can find it in the property tab in the screen that you take the screenshot.

It looks like the region of bucket is not the same as the endpoint.

BTW, it might be an issue of 'version = 1' because url mapping behavior is different for each AWS region. Then you could try 'version = 2'. Please refer to http://gpdb.docs.pivotal.io/510/admin_guide/external/g-s3-protocol.html#amazon-emr__s3_config_file for more detail.

Best wishes

Jasper

--

[Yogesh Jadhav]

Hi Jasper,

Thanks for your help.

With version 1 configuration is well. :-) 

$ gpcheckcloud -c "s3://s3-ap-south-1.amazonaws.com/greenplumext config=/home/gpadmin/s3_root_v1.conf"

File: Screen Shot 2017-01-23 at 1.14.21 AM.png, Size: 110279

File: sample_1.csv, Size: 43

File: sample_2.csv, Size: 43

Your configuration works well

--Yogesh

[Jasper Li]

Great.

Best wishes

Jasper

--