Hadoop indexing task fail with java.lang.IncompatibleClassChangeError (sometimes)

228 views

Skip to first unread message

Alexander Kasper

unread,

Oct 28, 2016, 6:04:25 AM10/28/16

to Druid User

Hi,

I'm running Druid 0.9.2-rc2 on Cloudera CDH 5.8.2 using the ORC extension. The first part of the indexing job consists of 23 map tasks, some of which fail with the following error:

ERROR [main] org.apache.hadoop.mapred.YarnChild: Error running child : com.google.common.util.concurrent.ExecutionError: java.lang.IncompatibleClassChangeError: Implementing class
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3934)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938)
	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821)
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4827)
	at com.google.inject.internal.FailableCache.get(FailableCache.java:48)
	at com.google.inject.internal.MembersInjectorStore.get(MembersInjectorStore.java:68)
	at com.google.inject.internal.InjectorImpl.getMembersInjector(InjectorImpl.java:993)
	at com.google.inject.internal.InjectorImpl.getMembersInjector(InjectorImpl.java:1000)
	at com.google.inject.internal.InjectorImpl.injectMembers(InjectorImpl.java:986)
	at io.druid.initialization.Initialization$ModuleList.addModule(Initialization.java:397)
	at io.druid.initialization.Initialization$ModuleList.addModules(Initialization.java:416)
	at io.druid.initialization.Initialization.makeInjectorWithModules(Initialization.java:320)
	at io.druid.indexer.HadoopDruidIndexerConfig.<clinit>(HadoopDruidIndexerConfig.java:99)
	at io.druid.indexer.HadoopDruidIndexerMapper.setup(HadoopDruidIndexerMapper.java:48)
	at io.druid.indexer.DetermineHashedPartitionsJob$DetermineCardinalityMapper.setup(DetermineHashedPartitionsJob.java:224)
	at io.druid.indexer.DetermineHashedPartitionsJob$DetermineCardinalityMapper.run(DetermineHashedPartitionsJob.java:282)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.IncompatibleClassChangeError: Implementing class
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
	at java.lang.Class.getDeclaredMethods(Class.java:1975)
	at com.google.inject.spi.InjectionPoint.getInjectionPoints(InjectionPoint.java:688)
	at com.google.inject.spi.InjectionPoint.forInstanceMethodsAndFields(InjectionPoint.java:380)
	at com.google.inject.internal.MembersInjectorStore.createWithListeners(MembersInjectorStore.java:93)
	at com.google.inject.internal.MembersInjectorStore.access$000(MembersInjectorStore.java:37)
	at com.google.inject.internal.MembersInjectorStore$1.create(MembersInjectorStore.java:45)
	at com.google.inject.internal.MembersInjectorStore$1.create(MembersInjectorStore.java:42)
	at com.google.inject.internal.FailableCache$1.load(FailableCache.java:37)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3524)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2317)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2280)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2195)
	... 23 more

I tracked this down and found that it fails when adding the module 'JettyServerModule', more specifically as can be seen in the exception during the injectMembers call. I suspected it might be a node specific problem, but I'm 99% sure they are all set up the same and I'm seeing

failed map tasks on nodes that have successful tasks as well.

Ingestion spec:

{
  "type": "index_hadoop",
  "spec": {
    "ioConfig": {
      "type": "hadoop",
      "inputSpec": {
        "type": "static",
        "inputFormat": "org.apache.hadoop.hive.ql.io.orc.OrcNewInputFormat",
        "paths": "/user/hive/warehouse/druid.db/adops/year=2016/month=10/day=01"
      }
    },
    "dataSchema": {
      "dataSource": "AdOpsVolume",
      "parser": {
        "type": "orc",
        "parseSpec": {
          "format": "timeAndDims",
          "timestampSpec": {
            "column": "ts",
            "format": "y-M-d H:m:s"
          },
          "dimensionsSpec": {
            "dimensions": [
              "feed",
              "country",
              "siteuuid",
              "sitename",
              "siteisapp",
              "sitetags",
              "deals"
            ],
            "dimensionExclusions": [],
            "spatialDimensions": []
          }
        },
        "typeString": "struct<ts:string,feed:string,country:string,siteuuid:string,sitename:string,siteisapp:boolean,sitetags:array<string>,deals:array<string>,requests:bigint>"
      },
      "metricsSpec": [{
          "type": "longSum",
          "name": "requests",
          "fieldName": "requests"
      }],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "DAY",
        "queryGranularity": "none",
        "intervals": ["2016-10-01/2016-10-02"]
      }
    },
    "tuningConfig": {
      "type": "hadoop",
      "partitionsSpec": {
        "type": "hashed",
        "targetPartitionSize": 5000000
      },
      "jobProperties": {
        "mapreduce.job.user.classpath.first": "true",
        "mapreduce.map.memory.mb": "2048",
        "mapreduce.reduce.memory.mb": "4092",
        "mapreduce.map.java.opts": "-server -Duser.timezone=UTC -Dfile.encoding=UTF-8",
        "mapreduce.reduce.java.opts": "-server -Duser.timezone=UTC -Dfile.encoding=UTF-8"
      },
      "leaveIntermediate": true
    }
  },
  "hadoopDependencyCoordinates": ["org.apache.hadoop:hadoop-client:2.6.0-cdh5.8.2"]
}

So at this point I'm lost for causes and would be grateful for every hint!

Thanks,

Alex

Fangjin Yang

unread,

Nov 4, 2016, 6:02:24 PM11/4/16

to Druid User

Hi Alex, do these docs help?

http://druid.io/docs/0.9.2-rc2/operations/other-hadoop.html

Alexander Kasper

unread,

Nov 7, 2016, 5:43:46 AM11/7/16

to Druid User

Hi,

I've read these and just today found the issue. The problem was caused by a dependency coming through the ORC extension. It pulled in a jetty-all-7.6.0.jar which was messing with the dependency injection. I removed it from the classpath and the indexing job then ran without any failed tasks. So if you're using this extension, be aware that the dependencies from it might cause issues.

Thanks,

Alex

Reply all

Reply to author

Forward

0 new messages