NativeS3FileSystem - Renaming s3n - returning false as src does not exist

[oleksandr...@ua.glomex.com]

Hello everyone!

I am trying to write test information to s3 bucket and i am getting an error:

017-04-18T15:07:48,095 DEBUG [pool-23-thread-1] org.apache.hadoop.fs.s3native.NativeS3FileSystem - getFileStatus could not find key 'druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip.0'
2017-04-18T15:07:48,095 DEBUG [pool-23-thread-1] org.apache.hadoop.fs.s3native.NativeS3FileSystem - Renaming 's3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip.0' to 's3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip' - returning false as src does not exist
2017-04-18T15:07:48,102 INFO [Thread-61] org.apache.hadoop.mapred.LocalJobRunner - reduce task executor complete.
2017-04-18T15:07:48,106 WARN [Thread-61] org.apache.hadoop.mapred.LocalJobRunner - job_local182715154_0002
java.lang.Exception: java.io.IOException: Unable to rename [s3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip.0] to [s3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip]
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) [hadoop-mapreduce-client-common-2.3.0.jar:?]
Caused by: java.io.IOException: Unable to rename [s3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip.0] to [s3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip]
    at io.druid.indexer.JobHelper.serializeOutIndex(JobHelper.java:452) ~[druid-indexing-hadoop-0.9.2.jar:0.9.2]
    at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:727) ~[druid-indexing-hadoop-0.9.2.jar:0.9.2]
    at io.druid.indexer.IndexGeneratorJob$IndexGeneratorReducer.reduce(IndexGeneratorJob.java:478) ~[druid-indexing-hadoop-0.9.2.jar:0.9.2]
    at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
    at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
    at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) ~[hadoop-mapreduce-client-core-2.3.0.jar:?]
    at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319) ~[hadoop-mapreduce-client-common-2.3.0.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473) ~[?:1.7.0_131]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[?:1.7.0_131]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[?:1.7.0_131]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[?:1.7.0_131]
    at java.lang.Thread.run(Thread.java:745) ~[?:1.7.0_131]

My task:

{
  "type" : "index_hadoop",
  "spec" : {
    "ioConfig" : {
      "type" : "hadoop",
      "inputSpec" : {
        "type" : "static",
        "paths" : "quickstart/wikiticker-2015-09-12-sampled.json.gz"
      }
    },
    "dataSchema" : {
      "dataSource" : "wikiticker-s3-new",
      "granularitySpec" : {
        "type" : "uniform",
        "segmentGranularity" : "day",
        "queryGranularity" : "none",
        "intervals" : ["2015-09-12/2015-09-13"]
      },
      "parser" : {
        "type" : "hadoopyString",
        "parseSpec" : {
          "format" : "json",
          "dimensionsSpec" : {
            "dimensions" : [
              "channel",
              "cityName",
              "comment",
              "countryIsoCode",
              "countryName",
              "isAnonymous",
              "isMinor",
              "isNew",
              "isRobot",
              "isUnpatrolled",
              "metroCode",
              "namespace",
              "page",
              "regionIsoCode",
              "regionName",
              "user"
            ]
          },
          "timestampSpec" : {
            "format" : "auto",
            "column" : "time"
          }
        }
      },
      "metricsSpec" : [
        {
          "name" : "count",
          "type" : "count"
        },
        {
          "name" : "added",
          "type" : "longSum",
          "fieldName" : "added"
        },
        {
          "name" : "deleted",
          "type" : "longSum",
          "fieldName" : "deleted"
        },
        {
          "name" : "delta",
          "type" : "longSum",
          "fieldName" : "delta"
        },
        {
          "name" : "user_unique",
          "type" : "hyperUnique",
          "fieldName" : "user"
        }
      ]
    },
    "tuningConfig" : {
      "type" : "hadoop",
      "partitionsSpec" : {
        "type" : "hashed",
        "targetPartitionSize" : 5000000
      },
      "jobProperties" : {
         "fs.s3.awsAccessKeyId" : "***",
        "fs.s3n.awsAccessKeyId" : "***",
        "fs.s3.awsSecretAccessKey" : "***",
        "fs.s3n.awsSecretAccessKey" : "***",
        "fs.s3.impl" : "org.apache.hadoop.fs.s3native.NativeS3FileSystem",
        "fs.s3n.impl" : "org.apache.hadoop.fs.s3native.NativeS3FileSystem",
         "io.compression.codecs" : "org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec"
      }
    }
  }
}

The file s3n://BUCKET_NAME/druid/segments/wikiticker-s3-new/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2017-04-18T15:07:24.820Z/0/index.zip.0 exists and i can see it in s3 bucket.

Best regards,
Oleksandr.


[oleksandr...@ua.glomex.com]

Also i found similar issue https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!searchin/druid-user/Renaming$20s3n|sort:relevance/druid-user/U9smlqQc4yc/znKjjBUzAQAJ
Currently i am using 0.9.2 version.

[oleksandr...@ua.glomex.com]

Have updated druid, to druid-0.10.0 and problem gone.

On Wednesday, April 19, 2017 at 10:37:00 AM UTC+3, oleksandr...@ua.glomex.com wrote:

[oleksandr...@ua.glomex.com]

Nope, problem was not resolved yet :(

So, please feel free to help with this stuff.
It can be the problem with some permissions, but i am not sure, please help.

Best regards,
Oleksandr.

On Wednesday, April 19, 2017 at 10:37:00 AM UTC+3, oleksandr...@ua.glomex.com wrote:

[oleksandr...@ua.glomex.com]

Now, problem is resolved, was set incorrect permissions for user keys.