Error While Writing Data to Amazon S3 Using GPDB!

[Sandeep Katta]

Hi Users,

I'm trying to load data from my gpdb-sandbox to Amazon S3 (using s3:// Protocol), and during the process of loading I'm getting an error. 

FYI: I can read the data from S3 without any issues. (I'm going to share both read and write queries to Amazon S3)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -c "s3://s3.us-east-1.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(9D5F6720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(9D5F6720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

* About to connect() to s3-external-1.amazonaws.com port 443 (#0)

*   Trying 54.231.98.76... * connected

* Connected to s3-external-1.amazonaws.com (54.231.98.76) port 443 (#0)

* successfully set certificate verify locations:

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSL connection using AES128-SHA

* Server certificate:

* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3-external-1.amazonaws.com

* start date: 2016-07-18 00:00:00 GMT

* expire date: 2017-10-26 12:00:00 GMT

* subjectAltName: s3-external-1.amazonaws.com matched

* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2

* SSL certificate verify ok.

> GET /firstname HTTP/1.1

Accept: */*

Host: s3-external-1.amazonaws.com

Authorization: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

x-amz-date: 20161215T003154Z

x-amz-content-sha256: UNSIGNED-PAYLOAD

< HTTP/1.1 200 OK

< x-amz-id-2: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

< x-amz-request-id: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

< Date: Thu, 15 Dec 2016 00:32:39 GMT

< x-amz-bucket-region: us-east-1

< Content-Type: application/xml

< Transfer-Encoding: chunked

< Server: AmazonS3

<

* Closing connection #0

File: fullname.txt, Size: 28

Your configuration works well.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

tutorial=# select * from s3_read_otp_ext;

 firstname | lastname

-----------+----------

 John   | Smith

 David  | Jones

(2 rows)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox s3]$ gpcheckcloud -u "/home/gpadmin/samplefile.txt" "s3://s3.us-east-1.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(65612720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[E]#0#(65612720)../../src/s3interface.cpp:439  AWS returns error AccessDenied : Access Denied

[D]#0#(65612720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[E]#0#(65612720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: ../../src/s3interface.cpp(439).

Failed. Please check the arguments and configuration file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

tutorial=# insert into s3_write_otp_ext select * from s3_read_otp;

ERROR:  Failed to init S3 extension (segid = 0, segnum = 2), please check your configurations and network connection: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: src/s3interface.cpp(439). (gps3ext.cpp:205)  (seg0 gpdb-sandbox.localdomain:40000 pid=4382) (cdbdisp.c:1320)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

My bucket permissions has List, Upload/Delete, View Permissions Checked.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Log says -

2016-12-14 19:48:59.244971 EST,"gpadmin","tutorial",p4382,th938305312,"172.16.155.133","59787",2016-12-14 19:48:27 EST,36513,con20320,cmd35,seg0,,dx14663,x365

13,sx1,"LOG","00000","[I]#0#(37ED6720)src/s3conf.cpp:68  The logserverport is set to default value 1111

",,,,,,,0,,,,

2016-12-14 19:48:59.462988 EST,"gpadmin","tutorial",p4382,th938305312,"172.16.155.133","59787",2016-12-14 19:48:27 EST,36513,con20320,cmd35,seg0,,dx14663,x365

13,sx1,"LOG","00000","[E]#0#(37ED6720)src/s3interface.cpp:439  AWS returns error AccessDenied : Access Denied

",,,,,,,0,,,,

2016-12-14 19:48:59.467254 EST,"gpadmin","tutorial",p4382,th938305312,"172.16.155.133","59787",2016-12-14 19:48:27 EST,36513,con20320,cmd35,seg0,,dx14663,x365

13,sx1,"LOG","00000","[D]#0#(37ED6720)src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

",,,,,,,0,,,,

2016-12-14 19:48:59.467568 EST,"gpadmin","tutorial",p4382,th938305312,"172.16.155.133","59787",2016-12-14 19:48:27 EST,36513,con20320,cmd35,seg0,,dx14663,x365

13,sx1,"LOG","00000","[E]#0#(37ED6720)src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error Ac

cessDenied : Access Denied, Function: getUploadId, File: src/s3interface.cpp(439).

",,,,,,,0,,,,

2016-12-14 19:49:01.514152 EST,"gpadmin","tutorial",p4382,th938305312,"172.16.155.133","59787",2016-12-14 19:48:27 EST,36513,con20320,cmd35,seg0,,dx14663,x36513,sx1,"ERROR","XX000","Failed to init S3 extension (segid = 0, segnum = 2), please check your configurations and network connection: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: src/s3interface.cpp(439). (gps3ext.cpp:205)",,,,,,"insert into s3_write_otp_ext select * from s3_read_otp;",0,,"src/gps3ext.cpp",205,"Stack trace:

1    0xb0a80e postgres errstart (elog.c:502)

2    0x7fb621b59958 gps3ext.so s3_export (??:0)

3    0x5893a4 postgres <symbol not found> (url.c:2944)

4    0x58ef4f postgres url_fwrite (url.c:2442)

5    0x583961 postgres external_insert (fileam.c:1766)

6    0x7b1aa3 postgres ExecInsert (execDML.c:525)

7    0x74fca4 postgres <symbol not found> (execMain.c:2729)

8    0x7501b8 postgres ExecutorRun (execMain.c:910)

9    0x99fbf4 postgres <symbol not found> (pquery.c:290)

10   0x9a2610 postgres PortalRun (pquery.c:1593)

11   0x99b2c3 postgres PostgresMain (postgres.c:1367)

12   0x8f981e postgres <symbol not found> (postmaster.c:6673)

13   0x8fc4c0 postgres PostmasterMain (postmaster.c:7604)

14   0x7fdc9f postgres main (main.c:206)

15   0x382ce1ed5d libc.so.6 __libc_start_main (??:0)

16   0x4bfeb9 postgres <symbol not found> (??:0)

"

2016-12-14 19:49:01.554282 EST,"gpadmin","tutorial",p4382,th938305312,"172.16.155.133","59787",2016-12-14 19:48:27 EST,36513,con20320,cmd35,seg0,,dx14663,x36513,sx1,"ERROR","XX000","Failed to cleanup S3 extension: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: src/s3interface.cpp(439). (gps3ext.cpp:183)",,,,,,,0,,"src/gps3ext.cpp",183,"Stack trace:

1    0xb0a80e postgres errstart (elog.c:502)

2    0x7fb621b59888 gps3ext.so s3_export (??:0)

3    0x58d4aa postgres url_fclose (url.c:2944)

4    0x58632a postgres AtEOXact_ExtTables (fileam.c:1874)

5    0x54185e postgres AbortTransaction (xact.c:4107)

6    0x545235 postgres AbortCurrentTransaction (xact.c:4364)

7    0x999bcc postgres PostgresMain (postgres.c:4514)

8    0x8f981e postgres <symbol not found> (postmaster.c:6673)

9    0x8fc4c0 postgres PostmasterMain (postmaster.c:7604)

10   0x7fdc9f postgres main (main.c:206)

11   0x382ce1ed5d libc.so.6 __libc_start_main (??:0)

12   0x4bfeb9 postgres <symbol not found> (??:0)

"

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Can someone please take a look & let me know what I'm missing here / any suggestions?

Thanks,

Sandeep K

[Qi Shao]

Hi Sandeep,

1. Make sure setup is correct:

1) Create functions to access s3 protocol library
$ psql -d postgres
postgres=# CREATE OR REPLACE FUNCTION read_from_s3() RETURNS integer AS
postgres=# '$libdir/gps3ext.so', 's3_import' LANGUAGE C STABLE;
postgres=# CREATE OR REPLACE FUNCTION write_to_s3() RETURNS integer AS
postgres=# '$libdir/gps3ext.so', 's3_export' LANGUAGE C STABLE;

2) Create s3 protocol

postgres=# CREATE PROTOCOL s3(writefunc = write_to_s3, readfunc = read_from_s3);

2. Make sure you have Upload/Delete permission on that s3 bucket

http://gpdb.docs.pivotal.io/43110/admin_guide/load/topics/g-s3-protocol.html

Thanks,

Qi

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.
To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

[Sandeep Katta]

Hi Qi,

Thank you for your prompt response!

I followed the instructions as per the admin_guide and created the functions and s3 protocol, but no luck!.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

                                                                              List of functions

 Schema |     Name     | Result data type | Argument data types |  Type  | Data access | Volatility |  Owner  | Language | Source code | Description

--------+--------------+------------------+---------------------+--------+-------------+------------+---------+----------+-------------+-------------

 public | read_from_s3 | integer          |                     | normal | no sql      | stable     | gpadmin | c        | s3_import   |

Schema |    Name     | Result data type | Argument data types |  Type  | Data access | Volatility |  Owner  | Language | Source code | Description

--------+-------------+------------------+---------------------+--------+-------------+------------+---------+----------+-------------+-------------

 public | write_to_s3 | integer          |                     | normal | no sql      | stable     | gpadmin | c        | s3_export   |

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I'm attaching the screenshot of my Bucket Permissions!

Thanks,

Sandeep K

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--

You received this message because you are subscribed to the Google Groups "Greenplum Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.

[Qi Shao]

Are you able to upload files using the same credential through aws command line?

Probably you can try to install aws cli in sandbox, then upload one file and see if that works.

It seems your credential are not eligible for uploading files to that bucket.

Thanks,

Qi

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+...@greenplum.org.

[Sandeep Katta]

Hi Qi,

As per your request, I Installed aws CLI on my gpdb-sandbox and tried to upload a file named "samplefile.txt"

I used the same credentials that I have in my s3.conf file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws configure

AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXXXXX

AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXX

Default region name [None]: us-east-1

Default output format [None]: json

[root@gpdb-sandbox ~]# aws s3 ls

2016-11-20 18:39:37 firstname

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-02 04:59:09         28 fullname.txt

[root@gpdb-sandbox ~]# aws s3 cp samplefile.txt s3://firstname

upload: ./samplefile.txt to s3://firstname/samplefile.txt

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-02 04:59:09         28 fullname.txt

2016-12-15 16:03:44         31 samplefile.txt

[root@gpdb-sandbox ~]# exit

logout

[gpadmin@gpdb-sandbox ~]$ psql \tutorial

psql (8.2.15)

Type "help" for help.

tutorial=# \dx

                   List of relations

 Schema |       Name       | Type  |  Owner  | Storage

--------+------------------+-------+---------+----------

 public | s3_read_otp_ext  | table | gpadmin | external

 public | s3_write_otp_ext | table | gpadmin | external

(2 rows)

tutorial=# select * from public.s3_read_otp_ext;

NOTICE:  Found 1 data formatting errors (1 or more input rows). Rejected related input data.

 firstname | lastname

-----------+----------

 John   | Smith

 David  | Jones

(2 rows)

 

tutorial=# insert into s3_write_otp_ext select * from s3_read_otp;

ERROR:  Failed to init S3 extension (segid = 0, segnum = 2), please check your configurations and network connection: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: src/s3interface.cpp(439). (gps3ext.cpp:205)  (seg0 gpdb-sandbox.localdomain:40000 pid=21870) (cdbdisp.c:1320)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Is there anything that you want me to try further?

Thanks,

Sandeep K

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.

To post to this group, send email to gpdb-...@greenplum.org.
Visit this group at https://groups.google.com/a/greenplum.org/group/gpdb-users/.
For more options, visit https://groups.google.com/a/greenplum.org/d/optout.

--
You received this message because you are subscribed to the Google Groups "Greenplum Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to gpdb-users+unsubscribe@greenplum.org.

[Ivan Novick]

Sandeep,

Can you also show read and write test results with gpcheckcloud?

Cheers,

ivan

Ivan Novick

Product Manager Pivotal Greenplum

YouTube: http://bit.ly/gpdbvideos

[Sandeep Katta]

Hi Ivan,

Here are my results!

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -c "s3://s3.us-east-1.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(FCA1C720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(FCA1C720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

* About to connect() to s3-external-1.amazonaws.com port 443 (#0)

*   Trying 52.216.17.12... * connected

* Connected to s3-external-1.amazonaws.com (52.216.17.12) port 443 (#0)

* successfully set certificate verify locations:

*   CAfile: /etc/pki/tls/certs/ca-bundle.crt

  CApath: none

* SSL connection using AES128-SHA

* Server certificate:

* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com Inc.; CN=*.s3-external-1.amazonaws.com

* start date: 2016-07-18 00:00:00 GMT

* expire date: 2017-10-26 12:00:00 GMT

* subjectAltName: s3-external-1.amazonaws.com matched

* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2

* SSL certificate verify ok.

> GET /firstname HTTP/1.1

Accept: */*

Host: s3-external-1.amazonaws.com

Authorization: 

x-amz-date: 20161215T212121Z

x-amz-content-sha256: UNSIGNED-PAYLOAD

< HTTP/1.1 200 OK

< x-amz-id-2: 

< x-amz-request-id: 

< Date: Thu, 15 Dec 2016 21:22:23 GMT

< x-amz-bucket-region: us-east-1

< Content-Type: application/xml

< Transfer-Encoding: chunked

< Server: AmazonS3

<

* Closing connection #0

File: fullname.txt, Size: 28

File: samplefile.txt, Size: 31

Your configuration works well.

[gpadmin@gpdb-sandbox ~]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -d "s3://s3.us-east-1.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(A2343720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(A2343720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

[D]#0#(A2343720)../../src/s3bucket_reader.cpp:45  key: https://s3-external-1.amazonaws.com/firstname/fullname.txt, size: 28

[D]#0#(8C71E700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8C71E700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(8C71E700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(8DB20700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(8D11F700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8D11F700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(8D11F700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(8DB20700)../../src/s3key_reader.cpp:129  Got 28 bytes from S3

[D]#0#(8DB20700)../../src/s3key_reader.cpp:139  Reached the end of file

John,Smith

[D]#0#(8DB20700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(A2343720)../../src/s3bucket_reader.cpp:45  key: https://s3-external-1.amazonaws.com/firstname/samplefile.txt, size: 31

[D]#0#(8D11F700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8C71E700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8C71E700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(8C71E700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(8DB20700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(8DB20700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(8DB20700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(8D11F700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(8D11F700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:129  Got 31 bytes from S3

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:139  Reached the end of file

David,JonesHi AWS - This is a test file.!

[D]#0#(8BD1D700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(A2343720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

[gpadmin@gpdb-sandbox ~]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -u "/home/gpadmin/samplefile.txt" "s3://s3.us-east-1.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(3CA1A720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[E]#0#(3CA1A720)../../src/s3interface.cpp:439  AWS returns error AccessDenied : Access Denied

[D]#0#(3CA1A720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[E]#0#(3CA1A720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: ../../src/s3interface.cpp(439).

Failed. Please check the arguments and configuration file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ postgres --gp-version

postgres (Greenplum Database) 4.3.10.0 build commit: f413ff3b006655f14b6b9aa217495ec94da5c96c

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Thanks,

Sandeep

[Scott Kahler]

The result there shows you are getting Access Denied errors trying to write to 

s3://s3.us-east-1.amazonaws.com/firstname/

can you throw a --debug flag into the 

aws s3 cp samplefile.txt s3://firstname

so we can better compare that to what the s3 protocol is doing

Scott Kahler | Pivotal, R&D, Platform Engineering  | ska...@pivotal.io | 816.237.0610

[Sandeep Katta]

Hi Scott,

Thanks for jumping.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-02 04:59:09         28 fullname.txt

2016-12-15 16:03:44         31 samplefile.txt

[root@gpdb-sandbox ~]#

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 cp samplefile1.txt s3://firstname --debug

2016-12-15 17:43:01,281 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/1.11.29 Python/2.6.6 Linux/2.6.32-573.el6.x86_64 botocore/1.4.86

2016-12-15 17:43:01,281 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['s3', 'cp', 'samplefile1.txt', 's3://firstname', '--debug']

2016-12-15 17:43:01,282 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_scalar_parsers at 0x28f9500>

2016-12-15 17:43:01,282 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x2526500>

2016-12-15 17:43:01,283 - MainThread - botocore.hooks - DEBUG - Event building-command-table.s3: calling handler <function add_waiters at 0x2909410>

2016-12-15 17:43:01,285 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.s3.anonymous: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,285 - MainThread - botocore.hooks - DEBUG - Event building-command-table.cp: calling handler <function add_waiters at 0x2909410>

2016-12-15 17:43:01,288 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-key: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,289 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expected-size: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,289 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.grants: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,289 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-language: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,289 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.page-size: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,289 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source-key: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.paths: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.source-region: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-encoding: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-disposition: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,290 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.cache-control: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.dryrun: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x2905d90>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.expires: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-c-copy-source: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,291 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.ignore-glacier-warnings: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x2905d90>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.content-type: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.metadata-directive: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.only-show-errors: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x2905d90>

2016-12-15 17:43:01,292 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.force-glacier-transfer: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x2905d90>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.anonymous: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.quiet: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x2905d90>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.acl: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.storage-class: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.sse-kms-key-id: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,293 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.website-redirect: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,294 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.custom.cp.follow-symlinks: calling handler <function uri_param at 0x24d22a8>

2016-12-15 17:43:01,294 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.custom.cp: calling handler <awscli.argprocess.ParamShorthandParser object at 0x2905d90>

2016-12-15 17:43:01,294 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env

2016-12-15 17:43:01,294 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: assume-role

2016-12-15 17:43:01,294 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: shared-credentials-file

2016-12-15 17:43:01,295 - MainThread - botocore.credentials - INFO - Found credentials in shared credentials file: ~/.aws/credentials

2016-12-15 17:43:01,295 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws/lib/python2.6/site-packages/botocore/data/endpoints.json

2016-12-15 17:43:01,345 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws/lib/python2.6/site-packages/botocore/data/s3/2006-03-01/service-2.json

2016-12-15 17:43:01,512 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3

2016-12-15 17:43:01,517 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x20496e0>

2016-12-15 17:43:01,517 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x2046de8>

2016-12-15 17:43:01,518 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None

2016-12-15 17:43:01,523 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

2016-12-15 17:43:01,523 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.

2016-12-15 17:43:01,525 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3

2016-12-15 17:43:01,528 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x20496e0>

2016-12-15 17:43:01,528 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x2046de8>

2016-12-15 17:43:01,528 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None

2016-12-15 17:43:01,532 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

2016-12-15 17:43:01,533 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.

2016-12-15 17:43:01,533 - MainThread - botocore.client - DEBUG - Registering retry handlers for service: s3

2016-12-15 17:43:01,536 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_post at 0x20496e0>

2016-12-15 17:43:01,536 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.s3: calling handler <function add_generate_presigned_url at 0x2046de8>

2016-12-15 17:43:01,536 - MainThread - botocore.args - DEBUG - The s3 config key is not a dictionary type, ignoring its value of: None

2016-12-15 17:43:01,598 - MainThread - botocore.endpoint - DEBUG - Setting s3 timeout as (60, 60)

2016-12-15 17:43:01,598 - MainThread - botocore.client - DEBUG - Defaulting to S3 virtual host style addressing with path style addressing fallback.

2016-12-15 17:43:01,600 - MainThread - awscli.customizations.s3.s3handler - DEBUG - Using a multipart threshold of 8388608 and a part size of 8388608

2016-12-15 17:43:01,601 - MainThread - s3transfer.utils - DEBUG - Acquiring 0

2016-12-15 17:43:01,602 - Thread-2 - s3transfer.tasks - DEBUG - UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x3221d90>}) about to wait for the following futures []

2016-12-15 17:43:01,603 - Thread-2 - s3transfer.tasks - DEBUG - UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x3221d90>}) done waiting for dependent futures

2016-12-15 17:43:01,603 - Thread-2 - s3transfer.tasks - DEBUG - Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x3221d90>}) with kwargs {'osutil': <s3transfer.utils.OSUtils object at 0x320cd50>, 'client': <botocore.client.S3 object at 0x31ee150>, 'config': <s3transfer.manager.TransferConfig object at 0x320ccd0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x3221d90>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x320cf10>}

2016-12-15 17:43:01,605 - Thread-2 - s3transfer.futures - DEBUG - Submitting task PutObjectTask(transfer_id=0, {'extra_args': {'ContentType': 'text/plain'}, 'bucket': u'firstname', 'key': u'samplefile1.txt'}) to executor <s3transfer.futures.BoundedExecutor object at 0x320cf10> for transfer request: 0.

2016-12-15 17:43:01,605 - Thread-2 - s3transfer.utils - DEBUG - Acquiring 0

2016-12-15 17:43:01,606 - Thread-3 - s3transfer.tasks - DEBUG - PutObjectTask(transfer_id=0, {'extra_args': {'ContentType': 'text/plain'}, 'bucket': u'firstname', 'key': u'samplefile1.txt'}) about to wait for the following futures []

2016-12-15 17:43:01,606 - Thread-2 - s3transfer.utils - DEBUG - Releasing acquire 0/None

2016-12-15 17:43:01,607 - Thread-3 - s3transfer.tasks - DEBUG - PutObjectTask(transfer_id=0, {'extra_args': {'ContentType': 'text/plain'}, 'bucket': u'firstname', 'key': u'samplefile1.txt'}) done waiting for dependent futures

2016-12-15 17:43:01,607 - Thread-3 - s3transfer.tasks - DEBUG - Executing task PutObjectTask(transfer_id=0, {'extra_args': {'ContentType': 'text/plain'}, 'bucket': u'firstname', 'key': u'samplefile1.txt'}) with kwargs {'extra_args': {'ContentType': 'text/plain'}, 'client': <botocore.client.S3 object at 0x31ee150>, 'bucket': u'firstname', 'key': u'samplefile1.txt', 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x2102210>}

2016-12-15 17:43:01,610 - Thread-3 - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x20a9668>

2016-12-15 17:43:01,610 - Thread-3 - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x20a7cf8>

2016-12-15 17:43:01,610 - Thread-3 - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x20a9c80>

2016-12-15 17:43:01,610 - Thread-3 - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x20a7c80>

2016-12-15 17:43:01,610 - Thread-3 - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x31ee850>>

2016-12-15 17:43:01,610 - Thread-3 - botocore.hooks - DEBUG - Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x20a6c80>

2016-12-15 17:43:01,612 - Thread-3 - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <function conditionally_calculate_md5 at 0x20a7c08>

2016-12-15 17:43:01,613 - Thread-3 - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x20a9140>

2016-12-15 17:43:01,613 - Thread-3 - botocore.handlers - DEBUG - Adding expect 100 continue header to request.

2016-12-15 17:43:01,613 - Thread-3 - botocore.hooks - DEBUG - Event before-call.s3.PutObject: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x31ee850>>

2016-12-15 17:43:01,613 - Thread-3 - botocore.endpoint - DEBUG - Making request for OperationModel(name=PutObject) (verify_ssl=True) with params: {'body': <s3transfer.utils.ReadFileChunk object at 0x2102210>, 'url': u'https://s3.amazonaws.com/firstname/samplefile1.txt', 'headers': {'Content-MD5': u'E6H+a3e83SJ2SiowdPpLvA==', u'Content-Type': 'text/plain', 'Expect': '100-continue', 'User-Agent': 'aws-cli/1.11.29 Python/2.6.6 Linux/2.6.32-573.el6.x86_64 botocore/1.4.86'}, 'context': {'client_region': 'us-east-1', 'signing': {'bucket': u'firstname'}, 'has_streaming_input': True, 'client_config': <botocore.config.Config object at 0x31ee290>}, 'query_string': {}, 'url_path': u'/firstname/samplefile1.txt', 'method': u'PUT'}

2016-12-15 17:43:01,614 - Thread-3 - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <function disable_upload_callbacks at 0x25f0050>

2016-12-15 17:43:01,614 - Thread-3 - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x31ee110>>

2016-12-15 17:43:01,614 - Thread-3 - botocore.hooks - DEBUG - Event before-sign.s3.PutObject: calling handler <function fix_s3_host at 0x1ef9cf8>

2016-12-15 17:43:01,614 - Thread-3 - botocore.utils - DEBUG - Checking for DNS compatible bucket for: https://s3.amazonaws.com/firstname/samplefile1.txt

2016-12-15 17:43:01,614 - Thread-3 - botocore.utils - DEBUG - URI updated to: https://firstname.s3.amazonaws.com/samplefile1.txt

2016-12-15 17:43:01,615 - Thread-3 - botocore.auth - DEBUG - Calculating signature using hmacv1 auth.

2016-12-15 17:43:01,615 - Thread-3 - botocore.auth - DEBUG - HTTP request method: PUT

2016-12-15 17:43:01,615 - Thread-3 - botocore.auth - DEBUG - StringToSign:

PUT

E6H+a3e83SJ2SiowdPpLvA==

text/plain

Thu, 15 Dec 2016 22:43:01 GMT

/firstname/samplefile1.txt

2016-12-15 17:43:01,615 - Thread-3 - botocore.hooks - DEBUG - Event request-created.s3.PutObject: calling handler <function enable_upload_callbacks at 0x25f00c8>

2016-12-15 17:43:01,622 - Thread-3 - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>

2016-12-15 17:43:01,623 - Thread-3 - botocore.vendored.requests.packages.urllib3.connectionpool - INFO - Starting new HTTPS connection (1): firstname.s3.amazonaws.com

2016-12-15 17:43:01,662 - Thread-3 - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.

2016-12-15 17:43:01,682 - Thread-3 - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.

2016-12-15 17:43:01,696 - Thread-3 - botocore.vendored.requests.packages.urllib3.connectionpool - DEBUG - "PUT /samplefile1.txt HTTP/1.1" 200 0

2016-12-15 17:43:01,698 - Thread-3 - botocore.parsers - DEBUG - Response headers: {'content-length': '0', 'x-amz-id-2': 'bnemHeBtILlh0C4xQO18KuDNZCNFskI4IrBTqJzoOjsRKkLkd2BT27HUl7zqw+BbuYEe+ZTaZac=', 'server': 'AmazonS3', 'x-amz-request-id': '39CC24673362F820', 'etag': '"13a1fe6b77bcdd22764a2a3074fa4bbc"', 'date': 'Thu, 15 Dec 2016 22:43:52 GMT'}

2016-12-15 17:43:01,698 - Thread-3 - botocore.parsers - DEBUG - Response body:

2016-12-15 17:43:01,700 - Thread-3 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x30df750>

2016-12-15 17:43:01,700 - Thread-3 - botocore.retryhandler - DEBUG - No retry needed.

2016-12-15 17:43:01,700 - Thread-3 - botocore.hooks - DEBUG - Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x31ee850>>

2016-12-15 17:43:01,700 - Thread-3 - botocore.hooks - DEBUG - Event after-call.s3.PutObject: calling handler <function enhance_error_msg at 0x28f6f50>

upload: ./samplefile1.txt to s3://firstname/samplefile1.txt

2016-12-15 17:43:01,701 - Thread-3 - s3transfer.utils - DEBUG - Releasing acquire 0/None

2016-12-15 17:43:01,723 - Thread-1 - awscli.customizations.s3.results - DEBUG - Shutdown request received in result processing thread, shutting down result thread.

[root@gpdb-sandbox ~]#

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-02 04:59:09         28 fullname.txt

2016-12-15 16:03:44         31 samplefile.txt

2016-12-15 17:43:52         19 samplefile1.txt

[root@gpdb-sandbox ~]#

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Thanks,

Sandeep

[Scott Kahler]

Are you using s3.amazonaws.com or s3.us-east-1.amazonaws.com in your table definition

if you are using us-east-1 can you give s3.amazonaws.com a try and see if that makes any difference

[Sandeep Katta]

Hi Scott,

That's interesting!

I'm using "s3.us-east-1.amazonaws.com" in my table. As per your recommendation, I tried with "s3.amazonaws.com" - the file got uploaded to s3 without any errors!

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-02 04:59:09         28 fullname.txt

2016-12-15 16:03:44         31 samplefile.txt

2016-12-15 17:43:52         19 samplefile1.txt

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -u "/home/gpadmin/samplefile3.txt" "s3://s3.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(41B6A720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(41B6A720)../../src/s3key_writer.cpp:15  key: https://s3.amazonaws.com/firstname/0911fd3ba.data.gz, upload id: P.xNo.97dFIBCaykFIc1IDZWEgWnP0trKjwEjDtyERv8KjYCSrkTJWhObpnjkgtYVzIbqwk47BmZuFyLzKGlT_inVJcbQJi.8_wec3HgCuUm7AqyV_D.dgPQYD0jo3z_

[D]#0#(41B6A720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[D]#0#(29547700)../../src/s3key_writer.cpp:94  Upload thread start: 0x7f0229547700, part number: 1, data size: 39

[D]#0#(29547700)../../src/s3key_writer.cpp:109  Upload part finish: 0x7f0229547700, eTag: "077219ec316674067bc927dd9370165a", part number: 1

[D]#0#(41B6A720)../../src/s3key_writer.cpp:178  Segment 0 has finished uploading "https://s3.amazonaws.com/firstname/0911fd3ba.data.gz"

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-15 18:21:09         39 0911fd3ba.data.gz

2016-12-02 04:59:09         28 fullname.txt

2016-12-15 16:03:44         31 samplefile.txt

2016-12-15 17:43:52         19 samplefile1.txt

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -d "s3://s3.amazonaws.com/firstname/0911fd3ba.data.gz config=/home/gpadmin/s3/s3.conf"

[I]#0#(99121720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(99121720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

[D]#0#(99121720)../../src/s3bucket_reader.cpp:45  key: https://s3-external-1.amazonaws.com/firstname/0911fd3ba.data.gz, size: 39

[D]#0#(82AFB700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(82AFB700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(82AFB700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(83EFD700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(83EFD700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(83EFD700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(834FC700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(834FC700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(834FC700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(848FE700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(848FE700)../../src/s3key_reader.cpp:129  Got 39 bytes from S3

[D]#0#(848FE700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(99121720)../../src/decompress_reader.cpp:116  Decompression finished: Z_STREAM_END.

This is my 3rd test!

[D]#0#(99121720)../../src/decompress_reader.cpp:89  No more data to decompress: avail_in = 0, avail_out = 2097152, total_in = 39, total_out = 21

[D]#0#(848FE700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(99121720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Before Change (at the table level):

 External table "public.s3_write_otp_ext"

  Column   | Type | Modifiers | Storage  | Description

-----------+------+-----------+----------+-------------

 firstname | text |           | extended |

 lastname  | text |           | extended |

Type: writable

Encoding: UTF8

Format type: text

Format options: delimiter ' ' null '\N' escape '\'

External location: "s3://s3.us-east-1.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

tutorial=# insert into s3_write_otp_ext select * from s3_read_otp;

ERROR:  Failed to init S3 extension (segid = 0, segnum = 2), please check your configurations and network connection: writer_init caught a S3LogicError exception: AWS returns error AccessDenied : Access Denied, Function: getUploadId, File: src/s3interface.cpp(439). (gps3ext.cpp:205)  (seg0 gpdb-sandbox.localdomain:40000 pid=21870) (cdbdisp.c:1320)

After Change (at the table level):

External table "public.s3_write_otp_ext"

  Column   | Type | Modifiers | Storage  | Description

-----------+------+-----------+----------+-------------

 firstname | text |           | extended |

 lastname  | text |           | extended |

Type: writable

Encoding: UTF8

Format type: csv

Format options: delimiter ',' null '' escape '"' quote '"'

External location: "s3://s3.amazonaws.com/firstname/ config=/home/gpadmin/s3/s3.conf"

tutorial=# insert into s3_write_otp_ext select * from s3_read_otp;

INSERT 0 2

[root@gpdb-sandbox ~]# aws s3 ls s3://firstname

2016-12-15 18:21:09         39 0911fd3ba.data.gz

2016-12-15 18:39:14         34 09cd0e3be.csv.gz

2016-12-15 18:39:14         35 17c78d9ef.csv.gz

2016-12-02 04:59:09         28 fullname.txt

2016-12-15 16:03:44         31 samplefile.txt

2016-12-15 17:43:52         19 samplefile1.txt

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

gpadmin@gpdb-sandbox ~]$ gpcheckcloud -d "s3://s3.amazonaws.com/firstname/09cd0e3be.csv.gz config=/home/gpadmin/s3/s3.conf"

[I]#0#(3A772720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(3A772720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

[D]#0#(3A772720)../../src/s3bucket_reader.cpp:45  key: https://s3-external-1.amazonaws.com/firstname/09cd0e3be.csv.gz, size: 34

[D]#0#(2414C700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(2414C700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(25F4F700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(2554E700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(2554E700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(2554E700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(2414C700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(24B4D700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(24B4D700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(24B4D700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(25F4F700)../../src/s3key_reader.cpp:129  Got 34 bytes from S3

[D]#0#(25F4F700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(3A772720)../../src/decompress_reader.cpp:116  Decompression finished: Z_STREAM_END.

John,Smith

[D]#0#(3A772720)../../src/decompress_reader.cpp:89  No more data to decompress: avail_in = 0, avail_out = 2097152, total_in = 34, total_out = 14

[D]#0#(25F4F700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(3A772720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -d "s3://s3.amazonaws.com/firstname/17c78d9ef.csv.gz config=/home/gpadmin/s3/s3.conf"

[I]#0#(587DC720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(587DC720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

[D]#0#(587DC720)../../src/s3bucket_reader.cpp:45  key: https://s3-external-1.amazonaws.com/firstname/17c78d9ef.csv.gz, size: 35

[D]#0#(42BB7700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(42BB7700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(42BB7700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(435B8700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(435B8700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(435B8700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(43FB9700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(421B6700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(421B6700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(421B6700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(43FB9700)../../src/s3key_reader.cpp:129  Got 35 bytes from S3

[D]#0#(43FB9700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(587DC720)../../src/decompress_reader.cpp:116  Decompression finished: Z_STREAM_END.

David,Jones

[D]#0#(587DC720)../../src/decompress_reader.cpp:89  No more data to decompress: avail_in = 0, avail_out = 2097152, total_in = 35, total_out = 15

[D]#0#(43FB9700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(587DC720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

[gpadmin@gpdb-sandbox ~]$

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

So It proves that we always need to go with - "s3.amazonaws.com" (irrespective of the endpoint)!

Thanks,

Sandeep

[Ivan Novick]

Great work guys to resolve this!

Should we recommend any documentation changes?

[Scott Kahler]

I'm not sure why the original string would work for read but not write, just that the string I suggested is the recommended s3 endpoint url for east. I knew it worked because I used it. Not sure why the other doesn't work, if that's an aspect of the s3 protocol code or an AWS thing.

[Sandeep Katta]

Hi Scott,

I did a smoke test, by creating a brand new bucket in Northern California region (us-west-1).

1. If I didn't specify an endpoint, I can neither read nor write to the bucket. 
2. If I specify the endpoint I can read, but not write.
   

I'm also seeing a weird error while uploading the file to the bucket - which says The specified bucket does not exist

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 cp samplefile1.txt s3://s3northerncalifornia20161215

upload: ./samplefile1.txt to s3://s3northerncalifornia20161215/samplefile1.txt

[root@gpdb-sandbox ~]# aws s3 ls s3://s3northerncalifornia20161215

2016-12-15 20:34:46         19 samplefile1.txt

[root@gpdb-sandbox ~]#

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

gpadmin@gpdb-sandbox s3]$ gpcheckcloud -d "s3://s3.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(4258C720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(4258C720)../../src/s3interface.cpp:312  Host url is s3-external-1.amazonaws.com

[E]#0#(4258C720)../../src/s3interface.cpp:332  AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

[E]#0#(4258C720)../../src/gpreader.cpp:131  reader_init caught S3LogicError: reader_init caught a S3LogicError exception: AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., Function: listBucket, File: ../../src/s3interface.cpp(332).

Failed. Please check the arguments and configuration file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox s3]$ gpcheckcloud -d "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(296C3720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(296C3720)../../src/s3interface.cpp:312  Host url is s3-us-west-1.amazonaws.com

[D]#0#(296C3720)../../src/s3bucket_reader.cpp:45  key: https://s3-us-west-1.amazonaws.com/s3northerncalifornia20161215/samplefile1.txt, size: 19

[D]#0#(13A9E700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(13A9E700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(1309D700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(1309D700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(1309D700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(1449F700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(1449F700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(1449F700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(14EA0700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(13A9E700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(14EA0700)../../src/s3key_reader.cpp:129  Got 19 bytes from S3

[D]#0#(14EA0700)../../src/s3key_reader.cpp:139  Reached the end of file

This is my test2.!

[D]#0#(14EA0700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(296C3720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

[gpadmin@gpdb-sandbox s3]$

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox s3]$ gpcheckcloud -u "/home/gpadmin/samplefile3.txt" "s3://s3.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(11325720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[E]#0#(11325720)../../src/s3interface.cpp:439  AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

[D]#0#(11325720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[E]#0#(11325720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error PermanentRedirect : The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint., Function: getUploadId, File: ../../src/s3interface.cpp(439).

Failed. Please check the arguments and configuration file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox s3]$ gpcheckcloud -u "/home/gpadmin/samplefile3.txt" "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(2C442720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[E]#0#(2C442720)../../src/s3interface.cpp:439  AWS returns error NoSuchBucket : The specified bucket does not exist

[D]#0#(2C442720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[E]#0#(2C442720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error NoSuchBucket : The specified bucket does not exist, Function: getUploadId, File: ../../src/s3interface.cpp(439).

Failed. Please check the arguments and configuration file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws s3 ls s3://s3northerncalifornia20161215

2016-12-15 20:34:46         19 samplefile1.txt

[root@gpdb-sandbox ~]#

[root@gpdb-sandbox ~]# aws s3 cp samplefile.txt s3://s3northerncalifornia20161215

upload: ./samplefile.txt to s3://s3northerncalifornia20161215/samplefile.txt

[root@gpdb-sandbox ~]#

[root@gpdb-sandbox ~]# aws s3 ls s3://s3northerncalifornia20161215

2016-12-15 20:56:47         31 samplefile.txt

2016-12-15 20:34:46         19 samplefile1.txt

[root@gpdb-sandbox ~]#

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox s3]$ gpcheckcloud -d "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/samplefile.txt config=/home/gpadmin/s3/s3.conf"

[I]#0#(2353C720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(2353C720)../../src/s3interface.cpp:312  Host url is s3-us-west-1.amazonaws.com

[D]#0#(2353C720)../../src/s3bucket_reader.cpp:45  key: https://s3-us-west-1.amazonaws.com/s3northerncalifornia20161215/samplefile.txt, size: 31

[D]#0#(CF16700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(CF16700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(D917700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(D917700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(CF16700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(ED19700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(D917700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(E318700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(E318700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(E318700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(ED19700)../../src/s3key_reader.cpp:129  Got 31 bytes from S3

[D]#0#(ED19700)../../src/s3key_reader.cpp:139  Reached the end of file

Hi AWS - This is a test file.!

[D]#0#(ED19700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(2353C720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

[gpadmin@gpdb-sandbox s3]$

[gpadmin@gpdb-sandbox s3]$ gpcheckcloud -u "/home/gpadmin/samplefile3.txt" "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(A61B6720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[E]#0#(A61B6720)../../src/s3interface.cpp:439  AWS returns error NoSuchBucket : The specified bucket does not exist

[D]#0#(A61B6720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[E]#0#(A61B6720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error NoSuchBucket : The specified bucket does not exist, Function: getUploadId, File: ../../src/s3interface.cpp(439).

Failed. Please check the arguments and configuration file.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I think engineering team needs to relook into this again!

Thanks,

Sandeep K

[Jasper]

Hi Sandeep,

Bucket name of S3 is unique. It seems that your bucket locates in us-west-1 and default region without endpoint is us-east-1 or external-1.  It is expected behaviour that it can't read and write without endpoint.

Could you try other key name to upload on us-west-1?

Also, please check the permission of key that is used by gpcheckcloud and ensure it can write on bucket.

[Sandeep Katta]

Hi Jasper,

I have no complains when I upload a file using aws CLI. (FYI: In my previous test, I used the same credentials that I used for s3 protocol.)

As per you request, I used different credentials and repeating the same.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[root@gpdb-sandbox ~]# aws configure

AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXXXXX

AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXX

Default region name [None]: us-east-1

Default output format [None]: json

[root@gpdb-sandbox ~]# aws s3 ls

2016-11-20 18:39:37 firstname

2016-12-15 20:21:03 s3northerncalifornia20161215

[root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215

2016-12-15 20:56:47         31 samplefile.txt

2016-12-15 20:34:46         19 samplefile1.txt

[root@gpdb-sandbox ~]# aws s3 cp fullname.txt s3://s3northerncalifornia20161215

upload: ./fullname.txt to s3://s3northerncalifornia20161215/fullname.txt

[root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215

2016-12-16 17:24:04         23 fullname.txt

2016-12-15 20:56:47         31 samplefile.txt

2016-12-15 20:34:46         19 samplefile1.txt

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -d "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/fullname.txt config=/home/gpadmin/s3/s3.conf"

[I]#0#(16151720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[D]#0#(16151720)../../src/s3interface.cpp:312  Host url is s3-us-west-1.amazonaws.com

[D]#0#(16151720)../../src/s3bucket_reader.cpp:45  key: https://s3-us-west-1.amazonaws.com/s3northerncalifornia20161215/fullname.txt, size: 23

[D]#0#(FFB2B700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(192E700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(F2D700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(F2D700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(F2D700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(FFB2B700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(FFB2B700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(52C700)../../src/s3key_reader.cpp:155  Downloading thread starts

[D]#0#(52C700)../../src/s3key_reader.cpp:139  Reached the end of file

[D]#0#(52C700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(192E700)../../src/s3key_reader.cpp:129  Got 23 bytes from S3

[D]#0#(192E700)../../src/s3key_reader.cpp:139  Reached the end of file

John,Smith

David,Jones

[D]#0#(192E700)../../src/s3key_reader.cpp:182  Downloading thread ended

[D]#0#(16151720)../../src/s3bucket_reader.cpp:55  Read finished for segment: 0

[gpadmin@gpdb-sandbox ~]$

       External table "public.s3_read_otp_ext_nc"

  Column   | Type | Modifiers | Storage  | Description

-----------+------+-----------+----------+-------------

 firstname | text |           | extended |

 lastname  | text |           | extended |

Type: readable

Encoding: UTF8

Format type: csv

Format options: delimiter ',' null '' escape '"' quote '"'

External location: "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/fullname.txt config=/home/gpadmin/s3/s3.conf"

tutorial=# SELECT * FROM public.s3_read_otp_ext_nc;

 firstname | lastname

-----------+----------

 John      | Smith

 David     | Jones

(2 rows)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Upload samplefile2 using aws CLI:

[root@gpdb-sandbox ~]# aws s3 cp samplefile2.txt s3://s3northerncalifornia20161215

upload: ./samplefile2.txt to s3://s3northerncalifornia20161215/samplefile2.txt

[root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215

2016-12-16 17:24:04         23 fullname.txt

2016-12-15 20:56:47         31 samplefile.txt

2016-12-15 20:34:46         19 samplefile1.txt

2016-12-16 17:26:03         19 samplefile2.txt

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[gpadmin@gpdb-sandbox ~]$ ll

drwxrwxr-x 2 gpadmin gpadmin 4096 Nov 20 04:32 s3

-rw-rw-r-- 1 gpadmin gpadmin   21 Dec 15 18:19 samplefile3.txt

[gpadmin@gpdb-sandbox ~]$ cat samplefile3.txt

This is my 3rd test!

[gpadmin@gpdb-sandbox ~]$ gpcheckcloud -u "/home/gpadmin/samplefile3.txt" "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

[I]#0#(4C610720)../../src/s3conf.cpp:68  The logserverport is set to default value 1111

[E]#0#(4C610720)../../src/s3interface.cpp:439  AWS returns error NoSuchBucket : The specified bucket does not exist

[D]#0#(4C610720)../../src/compress_writer.cpp:105  Compression finished: Z_STREAM_END.

[E]#0#(4C610720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError: writer_init caught a S3LogicError exception: AWS returns error NoSuchBucket : The specified bucket does not exist, Function: getUploadId, File: ../../src/s3interface.cpp(439).

Failed. Please check the arguments and configuration file.

External table "public.s3_write_otp_ext_nc"

  Column   | Type | Modifiers | Storage  | Description

-----------+------+-----------+----------+-------------

 firstname | text |           | extended |

 lastname  | text |           | extended |

Type: writable

Encoding: UTF8

Format type: csv

Format options: delimiter ',' null '' escape '"' quote '"'

External location: "s3://s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/ config=/home/gpadmin/s3/s3.conf"

tutorial=# select * from s3_read_otp; --regular table

 firstname | lastname

-----------+----------

  John      | Smith

 David     | Jones

(2 rows)

tutorial=# insert into s3_write_otp_ext_nc select * from s3_read_otp;

ERROR:  Failed to init S3 extension (segid = 1, segnum = 2), please check your configurations and network connection: writer_init caught a S3LogicError exception: AWS returns error NoSuchBucket : The specified bucket does not exist, Function: getUploadId, File: src/s3interface.cpp(439). (gps3ext.cpp:205)  (seg1 gpdb-sandbox.localdomain:40001 pid=19959) (cdbdisp.c:1320)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Upload samplefile3 using aws CLI:

[root@gpdb-sandbox ~]# aws s3 cp samplefile3.txt s3://s3northerncalifornia20161215

upload: ./samplefile3.txt to s3://s3northerncalifornia20161215/samplefile3.txt

[root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215

2016-12-16 17:24:04         23 fullname.txt

2016-12-15 20:56:47         31 samplefile.txt

2016-12-15 20:34:46         19 samplefile1.txt

2016-12-16 17:26:03         19 samplefile2.txt

2016-12-16 17:45:50         21 samplefile3.txt

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Please let me know, If you want me to try anything different.

Thanks,

Sandeep K

[Adam Lee]

Thanks for this report, could you please send the corresponding GPDB log
(SEGMENT_DATA_DIRECTORY/pg_log/gpdb-xxDATAxx.csv) to me?

Remember to review first to remove confidential information.

On Fri, Dec 16, 2016 at 05:52:56PM -0500, Sandeep Katta wrote:
> Hi Jasper,
>
> I have no complains when I upload a file using aws CLI. (FYI: In my previous
> test, I used the same credentials that I used for s3 protocol.)
>
> As per you request, I used different credentials and repeating the same.
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> [root@gpdb-sandbox ~]# aws configure
> AWS Access Key ID [None]: XXXXXXXXXXXXXXXXXXXXXXX
> AWS Secret Access Key [None]: XXXXXXXXXXXXXXXXXXXXXXX
> Default region name [None]: us-east-1
> Default output format [None]: json
>
> [root@gpdb-sandbox ~]# aws s3 ls
> 2016-11-20 18:39:37 firstname
> 2016-12-15 20:21:03 s3northerncalifornia20161215
>
> [root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215
> 2016-12-15 20:56:47         31 samplefile.txt
> 2016-12-15 20:34:46         19 samplefile1.txt
>
> [root@gpdb-sandbox ~]# aws s3 cp fullname.txt s3://s3northerncalifornia20161215
> upload: ./fullname.txt to s3://s3northerncalifornia20161215/fullname.txt
>
> [root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215
> 2016-12-16 17:24:04         23 fullname.txt
> 2016-12-15 20:56:47         31 samplefile.txt
> 2016-12-15 20:34:46         19 samplefile1.txt
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>

> [gpadmin@gpdb-sandbox ~]$ gpcheckcloud -d "s3://[1]s3.us-west-1.amazonaws.com/

> s3northerncalifornia20161215/fullname.txt config=/home/gpadmin/s3/s3.conf"
> [I]#0#(16151720)../../src/s3conf.cpp:68  The logserverport is set to default
> value 1111

> [D]#0#(16151720)../../src/s3interface.cpp:312  Host url is [2]
> s3-us-west-1.amazonaws.com
> [D]#0#(16151720)../../src/s3bucket_reader.cpp:45  key: [3]https://

> External location: "s3://[4]s3.us-west-1.amazonaws.com/

> s3northerncalifornia20161215/fullname.txt config=/home/gpadmin/s3/s3.conf"
>
> tutorial=# SELECT * FROM public.s3_read_otp_ext_nc;
>  firstname | lastname
> -----------+----------
>  John      | Smith
>  David     | Jones
> (2 rows)
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> Upload samplefile2 using aws CLI:
>
> [root@gpdb-sandbox ~]# aws s3 cp samplefile2.txt s3://
> s3northerncalifornia20161215
> upload: ./samplefile2.txt to s3://s3northerncalifornia20161215/samplefile2.txt
>
> [root@gpdb-sandbox ~]# aws s3 ls s3northerncalifornia20161215
> 2016-12-16 17:24:04         23 fullname.txt
> 2016-12-15 20:56:47         31 samplefile.txt
> 2016-12-15 20:34:46         19 samplefile1.txt
> 2016-12-16 17:26:03         19 samplefile2.txt
>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>
> [gpadmin@gpdb-sandbox ~]$ ll
> drwxrwxr-x 2 gpadmin gpadmin 4096 Nov 20 04:32 s3
> -rw-rw-r-- 1 gpadmin gpadmin   21 Dec 15 18:19 samplefile3.txt
>
> [gpadmin@gpdb-sandbox ~]$ cat samplefile3.txt
> This is my 3rd test!
>
> [gpadmin@gpdb-sandbox ~]$ gpcheckcloud -u "/home/gpadmin/samplefile3.txt" "s3:/

> /[5]s3.us-west-1.amazonaws.com/s3northerncalifornia20161215/ config=/home/

> gpadmin/s3/s3.conf"
> [I]#0#(4C610720)../../src/s3conf.cpp:68  The logserverport is set to default
> value 1111
> [E]#0#(4C610720)../../src/s3interface.cpp:439  AWS returns error NoSuchBucket :
> The specified bucket does not exist
> [D]#0#(4C610720)../../src/compress_writer.cpp:105  Compression finished:
> Z_STREAM_END.
> [E]#0#(4C610720)../../src/gpwriter.cpp:118  writer_init caught S3LogicError:
> writer_init caught a S3LogicError exception: AWS returns error NoSuchBucket :
> The specified bucket does not exist, Function: getUploadId, File: ../../src/
> s3interface.cpp(439).
> Failed. Please check the arguments and configuration file.
>
> External table "public.s3_write_otp_ext_nc"
>   Column   | Type | Modifiers | Storage  | Description
> -----------+------+-----------+----------+-------------
>  firstname | text |           | extended |
>  lastname  | text |           | extended |
> Type: writable
> Encoding: UTF8
> Format type: csv
> Format options: delimiter ',' null '' escape '"' quote '"'

> External location: "s3://[6]s3.us-west-1.amazonaws.com/

--
Adam Lee