AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: null

[Frank Dai]

I created an external table on Amazon S3, and I can run all kinds of  queries on this table from Hive, but no in Presto, Presto always throws the following exceptions:

Query 20150402_232018_00007_xe2sc failed: com.facebook.presto.hive.PrestoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: null; Request ID: 29949279706A3C9D), S3 Extended Request ID: Ntdc1wcvbpSK1tOEZfyJBeMXk9ulbbrYDsz+0I1cFSRlvzYbSLrvQDy1HDD0jcjabBw2Te+3Ktg=

java.lang.RuntimeException: com.facebook.presto.hive.PrestoS3FileSystem$UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: null; Request ID: 29949279706A3C9D), S3 Extended Request ID: Ntdc1wcvbpSK1tOEZfyJBeMXk9ulbbrYDsz+0I1cFSRlvzYbSLrvQDy1HDD0jcjabBw2Te+3Ktg=

at com.google.common.base.Throwables.propagate(Throwables.java:160)

at com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:412)

at com.facebook.presto.hive.PrestoS3FileSystem.getFileStatus(PrestoS3FileSystem.java:232)

at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:253)

at com.facebook.presto.hive.ParquetHiveRecordCursor.createParquetRecordReader(ParquetHiveRecordCursor.java:308)

at com.facebook.presto.hive.ParquetHiveRecordCursor.<init>(ParquetHiveRecordCursor.java:116)

at com.facebook.presto.hive.ParquetRecordCursorProvider.createHiveRecordCursor(ParquetRecordCursorProvider.java:68)

at com.facebook.presto.hive.HivePageSourceProvider.getHiveRecordCursor(HivePageSourceProvider.java:129)

at com.facebook.presto.hive.HivePageSourceProvider.createPageSource(HivePageSourceProvider.java:107)

at com.facebook.presto.spi.classloader.ClassLoaderSafeConnectorPageSourceProvider.createPageSource(ClassLoaderSafeConnectorPageSourceProvider.java:42)

at com.facebook.presto.split.PageSourceManager.createPageSource(PageSourceManager.java:58)

at com.facebook.presto.operator.TableScanOperator.addSplit(TableScanOperator.java:144)

at com.facebook.presto.operator.Driver.processNewSource(Driver.java:259)

at com.facebook.presto.operator.Driver.processNewSources(Driver.java:224)

at com.facebook.presto.operator.Driver.access$300(Driver.java:53)

at com.facebook.presto.operator.Driver$DriverLockResult.close(Driver.java:508)

at com.facebook.presto.operator.Driver.updateSource(Driver.java:202)

at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunnerFactory.createDriver(SqlTaskExecution.java:464)

at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunnerFactory.access$1500(SqlTaskExecution.java:431)

at com.facebook.presto.execution.SqlTaskExecution$DriverSplitRunner.processFor(SqlTaskExecution.java:557)

at com.facebook.presto.execution.TaskExecutor$PrioritizedSplitRunner.process(TaskExecutor.java:452)

at com.facebook.presto.execution.TaskExecutor$Runner.run(TaskExecutor.java:586)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused by: com.facebook.presto.hive.PrestoS3FileSystem.UnrecoverableS3OperationException: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: null; Request ID: 29949279706A3C9D), S3 Extended Request ID: Ntdc1wcvbpSK1tOEZfyJBeMXk9ulbbrYDsz+0I1cFSRlvzYbSLrvQDy1HDD0jcjabBw2Te+3Ktg=

at com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$21(PrestoS3FileSystem.java:400)

at com.facebook.presto.hive.PrestoS3FileSystem$$Lambda$146/230204174.call(Unknown Source)

at com.facebook.presto.hive.RetryDriver.run(RetryDriver.java:111)

at com.facebook.presto.hive.PrestoS3FileSystem.getS3ObjectMetadata(PrestoS3FileSystem.java:391)

... 23 more

Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: null; Request ID: 29949279706A3C9D)

at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:937)

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:506)

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:273)

at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3660)

at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:997)

at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:975)

at com.facebook.presto.hive.PrestoS3FileSystem.lambda$getS3ObjectMetadata$21(PrestoS3FileSystem.java:393)

... 26 more

I've already set my AWS key and secret key to the etc/catalog/hive.properties.

It's weird, actually my Presto cluster has been working with S3  in recent days, yesterday I upgraded to 0.100 and the above exception begins to appear, then I revert to 0.99  and all my configurations, but not work, now my Presto has malfunction :)

[Nezih Yigitbasi]

We usually see this problem when the AWS secret key Presto uses doesn't have the permissions to access the S3 bucket that you want to query. 

--
You received this message because you are subscribed to the Google Groups "Presto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to presto-users...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frank Dai]

I can use my key to download and upload files to the same S3 bucket in awscli(Amazon Command Line), it's weird that Presto complains about S3 permissions.

[Nezih Yigitbasi]

You should see the s3 path in your logs, did you try using aws s3api to get that particular object? See http://docs.aws.amazon.com/cli/latest/reference/s3api/get-object.html

And since you get a permission error during get metadata call the permissions section in the following link may help: 
http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectHEAD.html

[Frank Dai]

It turns out my NTP is broken and the time of my machine is very old, thus Amazon S3 will reject any API requests.

See here:  https://github.com/facebook/presto/issues/2635

To unsubscribe from this group and stop receiving emails from it, send an email to presto-users+unsubscribe@googlegroups.com.