Hipi + Hdfs

[Priyo Chatterjee]

I am using hipi with hadoop 2.7.1, with gradle build. I am not getting where to put hdfs information.



:examples:helloWorld:compileJava
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:1: error: modifier static not allowed here
 public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
               ^
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:1: error: modifier static not allowed here
public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
              ^
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:3: error: package org.hipi.image does not exist
import org.hipi.image.FloatImage;
                     ^
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:4: error: package org.hipi.image does not exist
import org.hipi.image.HipiImageHeader;
                     ^
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:5: error: package org.hipi.imagebundle.mapreduce does not exist
import org.hipi.imagebundle.mapreduce.HibInputFormat;
                                     ^
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:25: error: cannot find symbol
  public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                      ^
  symbol:   class HipiImageHeader
  location: class HelloWorld
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:25: error: cannot find symbol
  public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                                       ^
  symbol:   class FloatImage
  location: class HelloWorld
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:25: error: cannot find symbol
  public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                                                                ^
  symbol:   class FloatImage
  location: class HelloWorld
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:26: error: cannot find symbol
    public void map(HipiImageHeader key, FloatImage value, Context context)
                    ^
  symbol:   class HipiImageHeader
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:26: error: cannot find symbol
    public void map(HipiImageHeader key, FloatImage value, Context context)
                                         ^
  symbol:   class FloatImage
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:31: error: cannot find symbol
  public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
                                                                     ^
  symbol:   class FloatImage
  location: class HelloWorld
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:32: error: cannot find symbol
    public void reduce(IntWritable key, Iterable<FloatImage> values, Context context)
                                                 ^
  symbol:   class FloatImage
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:1: error: cannot find symbol
 public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                              ^
  symbol: class Mapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:1: error: cannot find symbol
 public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                     ^
  symbol: class HipiImageHeader
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:1: error: cannot find symbol
 public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                                      ^
  symbol: class FloatImage
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:1: error: cannot find symbol
 public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                                                  ^
  symbol: class IntWritable
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:1: error: cannot find symbol
 public static class HelloWorldMapper extends Mapper<HipiImageHeader, FloatImage, IntWritable, FloatImage> {
                                                                                               ^
  symbol: class FloatImage
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:3: error: cannot find symbol
    public void map(HipiImageHeader key, FloatImage value, Context context)
                    ^
  symbol:   class HipiImageHeader
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:3: error: cannot find symbol
    public void map(HipiImageHeader key, FloatImage value, Context context)
                                         ^
  symbol:   class FloatImage
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:3: error: cannot find symbol
    public void map(HipiImageHeader key, FloatImage value, Context context)
                                                           ^
  symbol:   class Context
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:4: error: cannot find symbol
        throws IOException, InterruptedException {
               ^
  symbol:   class IOException
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:1: error: cannot find symbol
public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
                                              ^
  symbol: class Reducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:1: error: cannot find symbol
public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
                                                      ^
  symbol: class IntWritable
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:1: error: cannot find symbol
public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
                                                                   ^
  symbol: class FloatImage
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:1: error: cannot find symbol
public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
                                                                               ^
  symbol: class IntWritable
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:1: error: cannot find symbol
public static class HelloWorldReducer extends Reducer<IntWritable, FloatImage, IntWritable, Text> {
                                                                                            ^
  symbol: class Text
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:3: error: cannot find symbol
    public void reduce(IntWritable key, Iterable<FloatImage> values, Context context)
                       ^
  symbol:   class IntWritable
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:3: error: cannot find symbol
    public void reduce(IntWritable key, Iterable<FloatImage> values, Context context)
                                                 ^
  symbol:   class FloatImage
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:3: error: cannot find symbol
    public void reduce(IntWritable key, Iterable<FloatImage> values, Context context)
                                                                     ^
  symbol:   class Context
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:4: error: cannot find symbol
        throws IOException, InterruptedException {
               ^
  symbol:   class IOException
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:47: error: cannot find symbol
    job.setInputFormatClass(HibInputFormat.class);
                            ^
  symbol:   class HibInputFormat
  location: class HelloWorld
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorld.java:54: error: cannot find symbol
    job.setMapOutputValueClass(FloatImage.class);
                               ^
  symbol:   class FloatImage
  location: class HelloWorld
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:29: error: cannot find symbol
        FloatImage avg = new FloatImage(1, 1, 3, avgData);
        ^
  symbol:   class FloatImage
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:29: error: cannot find symbol
        FloatImage avg = new FloatImage(1, 1, 3, avgData);
                             ^
  symbol:   class FloatImage
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldMapper.java:35: error: cannot find symbol
        context.write(new IntWritable(1), avg);
                          ^
  symbol:   class IntWritable
  location: class HelloWorldMapper
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:7: error: cannot find symbol
      FloatImage avg = new FloatImage(1, 1, 3);
      ^
  symbol:   class FloatImage
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:7: error: cannot find symbol
      FloatImage avg = new FloatImage(1, 1, 3);
                           ^
  symbol:   class FloatImage
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:11: error: cannot find symbol
      for (FloatImage val : values) {
           ^
  symbol:   class FloatImage
  location: class HelloWorldReducer
/home/priyo/software/hipi-release/examples/helloWorld/src/main/java/org/hipi/examples/HelloWorldReducer.java:23: error: cannot find symbol
        context.write(key, new Text(result));
                               ^
  symbol:   class Text
  location: class HelloWorldReducer
39 errors
:examples:helloWorld:compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':examples:helloWorld:compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

[Jason Lawrence]

I'm not sure I understand your question. Can you elaborate on what you mean by you can't figure out where to put hdfs information. Are you able to build and run the tools?

Jason

[Priyo Chatterjee]

I got that fixed. I want to store videos in file sequence format in hdfs and want to run map-reduce from there. I am not getting how to do this with the audio part.

[Priyo Chatterjee]

I dind't edit build.gradle file. hipi knows the path of hdfs. make sure you run your hdfs first.

On Wed, Oct 28, 2015 at 9:23 PM, sri harsha <sriharsha...@gmail.com> wrote:

Dear Priyo,

Can you please tell me how did you rectify this error? Did you edit the build.gradle file? 

Simple helloworld is running fine for me.. But when I try to run the map reduce, It is creating problems.. PFA for the error. 

[sri harsha]

thanks for the response priyo. MY HDFS is up..

actually i changed my build.gradle as below and it worked

jar {

manifest {

        attributes("Main-Class": "org.hipi.examples.HelloWorld")

    }

dependencies{

    compile project(':core')  }

}

-----------------------------------------------

Now when I run the program, At map job it fails as below..i have included the jar hipi-0.1.0.jar 

I am using hadoop-2.7.1 pseudonode cluster.

When I give

[root@localhost helloWorld]# hadoop jar build/libs/helloWorld.jar /input.hib /ohipi

15/10/29 08:38:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

15/10/29 08:38:40 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

15/10/29 08:38:41 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.

15/10/29 08:38:41 INFO input.FileInputFormat: Total input paths to process : 1

Spawned 1map tasks

15/10/29 08:38:41 INFO mapreduce.JobSubmitter: number of splits:1

15/10/29 08:38:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1446128864466_0008

15/10/29 08:38:42 INFO impl.YarnClientImpl: Submitted application application_1446128864466_0008

15/10/29 08:38:42 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1446128864466_0008/

15/10/29 08:38:42 INFO mapreduce.Job: Running job: job_1446128864466_0008

15/10/29 08:38:55 INFO mapreduce.Job: Job job_1446128864466_0008 running in uber mode : false

15/10/29 08:38:55 INFO mapreduce.Job:  map 0% reduce 0%

15/10/29 08:38:59 INFO mapreduce.Job: Task Id : attempt_1446128864466_0008_m_000000_0, Status : FAILED

Error: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.hipi.imagebundle.mapreduce.HibInputFormat not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)

at org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:174)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Caused by: java.lang.ClassNotFoundException: Class org.hipi.imagebundle.mapreduce.HibInputFormat not found

at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)

... 8 more

Is anyone facing the same problem?

My bashrc is as below

---------------------------------------------------------

export JAVA_HOME=/usr/java/jdk1.8.0_60

export PATH=$PATH:/usr/java/jdk1.8.0_60

export HADOOP_HOME=/home/harsha/hadoop-2.7.1

export HADOOP_INSTALL=$HADOOP_HOME

export HADOOP_MAPRED_HOME=$HADOOP_HOME

export HADOOP_COMMON_HOME=$HADOOP_HOME

export HADOOP_HDFS_HOME=$HADOOP_HOME

export HADOOP_YARN_HOME=$HADOOP_HOME

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native

export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

#### MAVEN 3.3.3 #########################

export MAVEN_HOME=/home/harsha/apache-maven-3.3.3

export PATH=$PATH:$MAVEN_HOME/bin

#### MAVEN 3.3.3 #########################

#### GRADLE 2.4 ###########################

export GRADLE_HOME=/home/harsha/gradle-2.4

export PATH=$PATH:$GRADLE_HOME/bin

#### GRADLE 2.4 ###########################

export PIG_HOME=/home/harsha/pig-0.15.0

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/harsha/hipi-release/core/build/libs/hipi-2.1.0.jar

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/home/harsha/hipi-release/release/hipi-0.1.0.jar

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_COMMON_LIB_NATIVE_DIR:/home/harsha/hipi-release/release/hipi-0.1.0.jar:/home/harsha/hipi-release/core/build/libs/hipi-2.1.0.jar

---------------------------------------------------------

[sri harsha]

BTW when I build using(core) etc.. its fine.. however when the program runs, It is not picking the org.hipi.imagebundle.mapreduce.HibInputFormat !! I dont know why this is happening

[Ankur Dubey]

I got same problem and then I have added manually all folders from core project hipi-2.1.0.jar and then on running I got yarn container launch error.

Do anyone know where the error could be? I post my error http://stackoverflow.com/questions/33365876/hipi-average-no-of-pixels-program-fails-due-to-yarn-containers-error-in-cloudera but there is no reply....... 

[akshay mamulwar]

i know where the error is, but dont knw how to correct it. here we go....

i downloaded hipi-release.zip from github.
i extracted the folder and i made this folder my java project in eclipse (kepler).
one this is done.....i got all the classes in hipi in my project but with lots of error.

so i added hadoop-core-1.1.2.jar in my build path...so all the errors are gone except one in class "HibInputFormat"

error is at: InputSplit split (-->split<--)

@Override
  public RecordReader<HipiImageHeader, HipiImage> createRecordReader(InputSplit split,
    TaskAttemptContext context)
  throws IOException, InterruptedException {
    return new HibRecordReader();
  }

it is showing error for splits as : The type org.apache.commons.logging.Log cannot be resolved. It is indirectly referenced from required .class files

so even with the error i built jar for this project but as excepted i havent gor HibInputFormat.class in my jar.

similarly i tried using built-in jar provided on github.....but if you carefully observe...these jars also not containing "HibInputFormat"


so thats the problem

[Niki Praha]

Thanks harsha :) I was struck at merely the same error. But after I changed the build.gradle as u mentioned, mine worked too :) Thank u

[Nandini Bommepally]

Hi Priyo Chatterjee,

I am working on image processing on hadoop using hipi can plz help me i am getting similar error which were posted above in this group..
and i need to work on  searching  similar images ... plz share your ideas on this ...

[shah.av...@vit.ac.in]

Can you Please Tell me how did you got this error solved ?? m also getting the same error "error: package org.hipi.image does not exist
import org.hipi.image.FloatImage; "

[majdouline meddad]

i also get the same problem , can someone help us