Unable to load AWS credentials from any provider in the chain

1,852 views
Skip to first unread message

bwh...@uber.com

unread,
Feb 11, 2016, 11:27:45 PM2/11/16
to Druid User
Hello there!

I am running a large Hadoop cluster on version 2.6.0 so none of my Hadoop indexing jobs work. I run into the jackson dependency issue. The issue has been documented several times: 

In order to fix the jackson dependency issue, I am forced to build a fat Druid jar excluding the jackson dependency with sbt. I followed the solution documented here: 

However, with this sbt built fat jar, I run into a really weird a random issue:
com.google.inject.ProvisionException: Guice provision errors:

1) Error in custom provider, com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
  at io.druid.storage.s3.S3StorageDruidModule.getRestS3Service(S3StorageDruidModule.java:107)
  at io.druid.storage.s3.S3StorageDruidModule.getRestS3Service(S3StorageDruidModule.java:107)
  while locating org.jets3t.service.impl.rest.httpclient.RestS3Service
    for parameter 0 at io.druid.storage.s3.S3DataSegmentKiller.<init>(S3DataSegmentKiller.java:43)
  while locating io.druid.storage.s3.S3DataSegmentKiller
  at io.druid.storage.s3.S3StorageDruidModule.configure(S3StorageDruidModule.java:84)
  while locating io.druid.segment.loading.DataSegmentKiller annotated with @com.google.inject.multibindings.Element(setName=,uniqueId=124, type=MAPBINDER)
  at io.druid.guice.Binders.dataSegmentKillerBinder(Binders.java:24)
  while locating java.util.Map<java.lang.String, io.druid.segment.loading.DataSegmentKiller>
    for parameter 0 at io.druid.segment.loading.OmniDataSegmentKiller.<init>(OmniDataSegmentKiller.java:36)
  while locating io.druid.segment.loading.OmniDataSegmentKiller
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:142)
  while locating io.druid.segment.loading.DataSegmentKiller
    for parameter 4 at io.druid.indexing.common.TaskToolboxFactory.<init>(TaskToolboxFactory.java:76)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:131)
  while locating io.druid.indexing.common.TaskToolboxFactory
    for parameter 0 at io.druid.indexing.overlord.ThreadPoolTaskRunner.<init>(ThreadPoolTaskRunner.java:71)
  at io.druid.cli.CliPeon$1.configure(CliPeon.java:157)
  while locating io.druid.indexing.overlord.ThreadPoolTaskRunner
  while locating io.druid.query.QuerySegmentWalker
    for parameter 3 at io.druid.server.QueryResource.<init>(QueryResource.java:90)
  while locating io.druid.server.QueryResource
Caused by: com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
at io.druid.storage.s3.S3StorageDruidModule.getRestS3Service(S3StorageDruidModule.java:107)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.google.inject.internal.ProviderMethod.get(ProviderMethod.java:105)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:90)
at com.google.inject.spi.ProviderLookup$1.get(ProviderLookup.java:90)
at com.google.inject.multibindings.MapBinder$RealMapBinder$2.get(MapBinder.java:389)
at com.google.inject.multibindings.MapBinder$RealMapBinder$2.get(MapBinder.java:385)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:86)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.provision(InternalFactoryToInitializableAdapter.java:55)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:66)
at com.google.inject.internal.InternalFactoryToInitializableAdapter.get(InternalFactoryToInitializableAdapter.java:47)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.Scopes$1$1.get(Scopes.java:65)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1058)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at io.druid.guice.LifecycleScope$1.get(LifecycleScope.java:49)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:56)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:107)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:88)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:269)
at com.google.inject.internal.InjectorImpl$3$1.call(InjectorImpl.java:1005)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1051)
at com.google.inject.internal.InjectorImpl$3.get(InjectorImpl.java:1001)
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1036)
at io.druid.guice.LifecycleModule$2.start(LifecycleModule.java:134)
at io.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:71)
at io.druid.cli.CliPeon.run(CliPeon.java:211)
at io.druid.cli.Main.main(Main.java:91)

I don't use AWS at all so I am not sure why I am getting this issue. Must be an issue with the build. Here is my spec: 
{
  "type" : "index_hadoop",
  "spec" : {
    "dataSchema" : {
      "dataSource" : "api_created_trips_02",
      "parser" : {
        "type" : "string",
        "parseSpec" : {
          "format" : "json",
          "timestampSpec" : {
            "column" : "SecondsSinceEpoch",
            "format" : "posix"
          },
          "dimensionsSpec": {
            "dimensions": [
              "uuid",
              "country_id",
              "city_id",
              "request_at",
              "status"
            ],
            "dimensionExclusions": [],
            "spatialDimensions": []
          }
        }
      },
      "metricsSpec": [
        {
          "type": "doubleSum",
          "name": "duration",
          "fieldName": "duration"
        }
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "intervals" : ["2015-12-05/2015-12-06"]
      }
    },
    "ioConfig" : {
      "type" : "hadoop",
      "inputSpec" : {
        "type" : "static",
        "paths" : "hdfs://nameservice1/user/bwhite/api_created_trips_1day_json"
      }
    }
  }
}

Thanks!

Gian Merlino

unread,
Feb 17, 2016, 1:06:19 AM2/17/16
to druid...@googlegroups.com
Hey bwhite,

If you aren't using aws at all, try removing the "io.druid.extensions" % "druid-s3-extensions" and "com.amazonaws" % "aws-java-sdk" lines from that build.sbt linked in the docs (along with any other extensions you aren't using).

Also, are you using a special build of Hadoop, or the stock Apache one? If it's stock Apache could you give some more details about the Jackson related exception you saw? We are using Jackson 2.4.6 because we have seen it work well with most versions of Apache Hadoop, so it would be good to know if there's somewhere that's not working there.

Gian

--
You received this message because you are subscribed to the Google Groups "Druid User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to druid-user+...@googlegroups.com.
To post to this group, send email to druid...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/druid-user/427b62bf-0702-4bff-bdab-2863aab24be9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages