Re: Jedis with Hadoop: ClassNotFoundException: org.apache.commons.pool.impl.GenericObjectPool$Config

[George London]

Hi everyone!

Sorry to pester, but could someone please help me with this? I feel like Redis and Hadoop could play very nicely, but I'm blocked without Jedis. I tried visiting the Jedis IRC channel, but automatically got kicked out and banned.

Alternatively, could someone suggest another resource that would help or a way that I can debug this myself?

Thanks!

-George

On Saturday, November 24, 2012 1:21:21 PM UTC-8, George London wrote:

Hi All,

I just discovered Jedis today (it seems great!) and am already trying to put it to a non-standard use. 

I want to use Redis to coordinate among mappers in a Hadoop job. Hadoop creates "Mapper" classes on each machine in the cluster and then has those classes repeatedly call the "map" method to process each line of data. So I'm trying to create a connection pool when the Mapper initializes and then have each call of the map method get a connection from the pool and return it when it's done.

This is giving me an error:

2012-11-24 20:53:25,087 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoClassDefFoundError: org/apache/commons/pool/impl/GenericObjectPool$Config
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:247)
	at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:1510)
	at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1475)
	at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1569)
	at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:191)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:605)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.pool.impl.GenericObjectPool$Config
	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
	... 13 more

Here's the basic code I'm trying to run:

static class Accretor

extends TableMapper<ImmutableBytesWritable, Writable> {

private JedisPool jPool = null;

@Override

    protected void setup(Context context)

      throws IOException, InterruptedException {

JedisPoolConfig j = new JedisPoolConfig();

jPool = new JedisPool(j, "ec2-23-20-255-48.compute-1.amazonaws.com");

System.out.println("Opened jedis connection");

    }

    @Override

    public void map(ImmutableBytesWritable row, Result r, Context context) 

    throws IOException {

    // basic setup -- get conf, create logging table, start timer

    Jedis jedis = jPool.getResource();

    jPool.returnResource(jedis);

}

I have Jedis included via Maven:

            

<dependency>

    <groupId>redis.clients</groupId>

    <artifactId>jedis</artifactId>

    <version>2.0.0</version>

    <type>jar</type>

    <scope>compile</scope>

</dependency>

   

And I tried including the Maven artifact for commons-pool just in case:

<dependency>

<groupId>commons-pool</groupId>

<artifactId>commons-pool</artifactId>

<version>1.4</version>

</dependency>

I'm using Shade to bundle a "fat jar" for Hadoop that should include all the dependencies. When I run the same basic code but on my local machine, it works fine. So I'm guessing it has something to do with the dependency packaging...

Any ideas?

Thanks!

-George

[Sam Hendley]

This looks like a dependency issue. I have no experience with shade so
I can't be of much help there. What I would do in your shoes though is
to unzip the "fat jar" file and make sure it has all of the classes
you need. If not you know the problem is the fat jar builder. If they
are there I would guess you may have a multi jar problem on your
deployment system and its finding some other version of the pooling
library thats causing issues.

Sam

[George London]

I found a different approach instead of figuring this out :(

But I am still excited to try using Redis with Hadoop, so please let us know if you find a solution!

-------------------------------------------

George London

E: george....@gmail.com

T: @rogueleaderr

B: rogueleaderr.com

-------------------------------------------

On Apr 12, 2013, at 10:43 AM, Екатерина Красильникова <e.krasi...@owox.com> wrote:

Hi!

You found a solution?

I have hadoop single node and the same problem.

I think it's because, map run in a separate thread (look code below).

import java.io.IOException;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

import org.apache.hadoop.util.GenericOptionsParser;

import redis.clients.jedis.Jedis;

import redis.clients.jedis.JedisPool;

import redis.clients.jedis.JedisPoolConfig;

public class RedisTest {

protected static Jedis getJedis() {

JedisPoolConfig config = new JedisPoolConfig();

JedisPool pool = new JedisPool(config, "localhost");

return pool.getResource();

}

static class RedisMapper extends Mapper<IntWritable, IntWritable, IntWritable, IntWritable> {

 

protected Jedis jedis;

public void map(IntWritable key, IntWritable value, Context context) 

throws IOException, InterruptedException {

if (jedis == null) getJedis();

}

}

public static void main(String[] args) throws Exception {

// if this uncomment, error will not be

// getJedis();

// System.exit(1);

Configuration conf = new Configuration();

String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs();

if (otherArgs.length < 2) {

System.err.println("Usage params: <input_dir> <output_dir>");

System.exit(2);

}

Job job = new Job(conf, "Test Jedis");

job.setJarByClass(RedisTest.class);

job.setMapperClass(RedisMapper.class);

job.setOutputKeyClass(IntWritable.class);

job.setOutputValueClass(IntWritable.class);

FileInputFormat.addInputPath(job, new Path(otherArgs[0]));

FileOutputFormat.setOutputPath(job, new Path(otherArgs[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

   

}

}

суббота, 24 ноября 2012 г., 23:21:21 UTC+2 пользователь George London написал:

--
You received this message because you are subscribed to a topic in the Google Groups "Jedis" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jedis_redis/ziPNqJTtBes/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to jedis_redis...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Екатерина Красильникова]

Hi! I find a solution :)

If you want to use external lib in Map-Reduce process, you should specify to hadoop the path to this lib.

For the time being, i write in hadoop-env.sh following:

export HADOOP_CLASSPATH="/var/www/test/target/libs/jedis-2.1.0.jar:/var/www/test/target/libs/commons-pool-1.5.5.jar:$HADOOP_CLASSPATH"

and restarted hadoop.

Surely, there is more elegant solution. When I have it - I will write.

пятница, 12 апреля 2013 г., 20:50:27 UTC+3 пользователь George London написал:

[Екатерина Красильникова]

http://blog.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job/

понедельник, 15 апреля 2013 г., 13:01:05 UTC+3 пользователь Екатерина Красильникова написал:

[sudhee...@gmail.com]

Thi is great information and it is useful for hadoop learners.123trainings provides<a href="http://123trainings.com/it-hadoop-bigdata-online-training.html">hadoop online training</a>

[Jorge Arias]

I solved it by adding the library to the gradle (or maven) configuration.. the missing library is org.apache.commons:commons-pool2:2.2 from https://commons.apache.org/proper/commons-pool/dependencies.html.