EMR updates + slf4j IllegalAccessErrors

[Ben Linsay]

In the middle of a funny issue - the Amazon AMI update to Hadoop 1.0 (woo!) appears to have rolled out in between two test runs of a Cascading job I was writing. I went from working code to getting the following stack trace:

Exception in thread "main" java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory

at org.slf4j.LoggerFactory.<clinit>(LoggerFactory.java:60)

at cascading.property.AppProps.<clinit>(AppProps.java:46)

Based on the slf4j FAQ (http://slf4j.org/faq.html#IllegalAccessError) it looks like an unfortunate dependency problem somewhere. Taking a look at the classpath, there are definitely multiple versions of slf4j hanging out:

hadoop@ip-10-58-47-161:~$ ll /home/hadoop/lib/*slf*

lrwxrwxrwx 1 hadoop   65 Aug  7 04:04 /home/hadoop/lib/slf4j-api-1.4.3.jar -> /home/hadoop/.versions/1.0.3/share/hadoop/lib/slf4j-api-1.4.3.jar

-rw-r--r-- 1 hadoop  26K Jul 31 22:57 /home/hadoop/lib/slf4j-api-1.6.4.jar

lrwxrwxrwx 1 hadoop   69 Aug  7 04:04 /home/hadoop/lib/slf4j-log4j12-1.4.3.jar -> /home/hadoop/.versions/1.0.3/share/hadoop/lib/slf4j-log4j12-1.4.3.jar

-rw-r--r-- 1 hadoop 9.6K Jul 31 22:57 /home/hadoop/lib/slf4j-log4j12-1.6.4.jar

Hanging out in IRC while I slog through this. Whee!

[Ben Linsay]

Removing the symlinks to the 1.4.3 version did the trick. I've been poking around for a cleaner fix, but no dice. It seems like having both versions in the classpath is the problem, so that's the problem that has to be solved.

[Chris K Wensel]

its probably safe to leave it out of your job jar since its in the hadoop classpath.

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/xZrbycvwlYEJ.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com

[Ben Linsay]

i'll give that a shot. seems like the issue is that the hadoop classpath has both the 1.4.x and the 1.6.x versions, so I didn't think that would fix anything. might as well try though. :)

On Tuesday, August 7, 2012 9:21:25 AM UTC-7, Chris K Wensel wrote:

its probably safe to leave it out of your job jar since its in the hadoop classpath.

ckw

On Aug 6, 2012, at 11:40 PM, Ben Linsay <bli...@gmail.com> wrote:

Removing the symlinks to the 1.4.3 version did the trick. I've been poking around for a cleaner fix, but no dice. It seems like having both versions in the classpath is the problem, so that's the problem that has to be solved.

On Monday, August 6, 2012 10:37:49 PM UTC-7, Ben Linsay wrote:

In the middle of a funny issue - the Amazon AMI update to Hadoop 1.0 (woo!) appears to have rolled out in between two test runs of a Cascading job I was writing. I went from working code to getting the following stack trace:

Exception in thread "main" java.lang.IllegalAccessError: tried to access field org.slf4j.impl.StaticLoggerBinder.SINGLETON from class org.slf4j.LoggerFactory

at org.slf4j.LoggerFactory.<clinit>(LoggerFactory.java:60)

at cascading.property.AppProps.<clinit>(AppProps.java:46)

Based on the slf4j FAQ (http://slf4j.org/faq.html#IllegalAccessError) it looks like an unfortunate dependency problem somewhere. Taking a look at the classpath, there are definitely multiple versions of slf4j hanging out:

hadoop@ip-10-58-47-161:~$ ll /home/hadoop/lib/*slf*

lrwxrwxrwx 1 hadoop   65 Aug  7 04:04 /home/hadoop/lib/slf4j-api-1.4.3.jar -> /home/hadoop/.versions/1.0.3/share/hadoop/lib/slf4j-api-1.4.3.jar

-rw-r--r-- 1 hadoop  26K Jul 31 22:57 /home/hadoop/lib/slf4j-api-1.6.4.jar

lrwxrwxrwx 1 hadoop   69 Aug  7 04:04 /home/hadoop/lib/slf4j-log4j12-1.4.3.jar -> /home/hadoop/.versions/1.0.3/share/hadoop/lib/slf4j-log4j12-1.4.3.jar

-rw-r--r-- 1 hadoop 9.6K Jul 31 22:57 /home/hadoop/lib/slf4j-log4j12-1.6.4.jar

Hanging out in IRC while I slog through this. Whee!

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/xZrbycvwlYEJ.

To post to this group, send email to cascading-user@googlegroups.com.
To unsubscribe from this group, send email to cascading-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

[Chris K Wensel]

if its in the hadoop classpath, the file a bug with AWS. or get the latest ruby emr client and peg it to a working hadoop version.

ckw

To view this discussion on the web visit https://groups.google.com/d/msg/cascading-user/-/7Zr8idNffwYJ.
To post to this group, send email to cascadi...@googlegroups.com.
To unsubscribe from this group, send email to cascading-use...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/cascading-user?hl=en.

[Ben Linsay]

Yeah, sorry for not being clear in the midst of that. Both 1.4.x and 1.6.x are included in the Hadoop classpath by way of /home/hadoop/lib/. When you run an EMR jar-step they both get included. I didn't find any solutions other than removing the symlinks in /home/hadoop/lib that point to the 1.4.x jars.

Already filed a bug with Amazon. I'll let you guys know what happens.

[Vikram Kadi]

I believe we are observing the same issue, is there a link to the filed bug, so we can watch it.

[Ben Linsay]

It's a private EMR support ticket. They're not past the "is this a real problem?" phase yet. I'll post any updates here.

Did you try removing the slf4j-1.4.3 symlinks?

[Vikram Kadi]

We are using lemur to launch a EMR cluster, so we really do not have the required control to modify the hadoop distribution.

Thanks for the reply.

[Ben Linsay]

My plan was to set up a simple bootstrap action that just rm's the links. I've been occupied with other work, so I haven't had a chance to try it yet.

[Jeroen van Dijk]

Op dinsdag 7 augustus 2012 23:39:25 UTC+2 schreef Vikram Kadi het volgende:

We are using lemur to launch a EMR cluster, so we really do not have the required control to modify the hadoop distribution.

Thanks for the reply.

Vikram,

I came across the above issue as well and, so it happens, Lemur was patches yesterday and now supports the :ami-version option. I've set it to 

:ami-version "2.1.3", previous it was also on "latest", hence the issue.

For details see https://github.com/mlimotte/lemur/commit/47a7f42172c136b8beeb64e775c8da4d68a91373 and http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/EnvironmentConfig_AMIHadoopPatches.html. This solved the issue for me.

And if you would like to use the latest hadoop and want to do some bootstrap actions, Lemur also has some options for that, but I haven't used that though.

Cheers,

Jeroen

[slojo404]

Just wanted to post a quick reply saying thank you as setting up a simple bootstrap action removing the links worked for me.

[Ben Linsay]

Got an update from AWS Support. They've "identified the issue in the latest AMI and are working on the solution". No word on what that solution is, but they're working on it. :P

[Vikram Kadi]

Ben,

Did Amazon ever get around to resolving  this?

regards,

Vikram

[Vikram Kadi]

Looks like its working! Just ran a test with latest AMI.