installation of hadoop

[Rajasri]

hi all...I tried installing Hadoop-0.20.1 in Ubuntu 12.10 using the instructions provided in http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ .. I completed installation..Now i am facing problem with "running the mapreduce job" ie

 $ bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
 --> The output folder is not getting created.
 -->i am using hadoop-0.20.1-examples.jar file which has many sub folders.But the site shows hadoop-examples.jar alone.(hadoop-1.0.3)
  Nothing occurs in the screen if i run this command. Can someone here help me with this ?? Thanks in advance!
  details:
  /user/hduser/gutenberg -- > has one .txt file
  hadoop-0.20.1-examples.jar --> has META-INF , ORG (-->apache-->hadoop-->examples-->(many .java files)

[Sai Sivam]

$ bin/hadoop dfs -ls /user/hduser

A screenshot of the problematic command please. 
Please post some context too. Like the commands before. 
I have not tried this example. I can do that sometime tomorrow. 

[Rajasri janakiraman]

sir,
1st image --> i have already stored the input file in
/user/hduser/gutenberg as pg5000.txt. Wen i run the command
$ bin/hadoop jar /usr/local/hadoop/hadoop*examples*.jar wordcount -D
mapred.reduce.tasks=16 /user/hduser/gutenberg
/user/hduser/gutenberg-output i am not getting any response.

2nd image--> i am using hadoop-0.20.1. so i changed the .jar file as
hadoop-0.20.1-examples.jar. Even then there is no response.

3rd image--> shows the content of folder "hadoop" which contain
"hadoop-0.20.1-examples.jar". I also changed that .jar file to
"wordcount.jar". But still no result. The "gutenberg-output" is also
not created.

Regards,
Rajasri

[Senthil Kumar]

Hi RajaSri

Are you sure that the  job runs??? did u check the jobtracker webUI??

Do you have any data in pg5000.txt?? from photo1, i think the size of the file is 0.

Can you send me the screenshot of dfs -cat /<your file>?

If even some bytes contents , can you please send me your screenshot of jobtracker UI??

Thanks

Senthil

--

[Rajasri janakiraman]

i didn't get any response for giving cat command to see the content of
that file. When i give jps command it shows the trackers also...
I have attached the screenshots for your further reference.
Regards,
Rajasri.J

[Senthil Kumar]

RajaSri,

 

Where is your datanode?? Its not running !!!!!!

 

Send me the logs for the datanode as well as namenode. secondary namenode??

 

Thanks

Senthil

[Pappala Rajasekhar]

RajaSri,

Check the *.xml file and give the correct configuration and have you changed the JAVA_HOME in hadoop-env.sh file. Make sure about the environmental variables..

--
 
 

--
Warm Regards
P. Raja Sekhar
Business Analyst
Affine Analytics Pvt Limited
Bangalore
Ph No.: +91 9391912223

This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. You are also notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.

[Senthil Kumar]

Rajasekhar,

Issues seen clearly..if JAVA_HOME is not set properly, other daemons could not have run.!!!!!

[Sai Sivam]

Going to look into this. If you have resolved this, please share that. WIth your resolution. Thanks.

[Rajasri janakiraman]

i have attached the localhost site for namenode and the webpage which
displays the namenode logs.I have set the JAVA_HOME environment
properly.
Regards,
Rajasri.J

On Mon, Dec 17, 2012 at 7:32 PM, Sai Sivam <s...@orzota.com> wrote:
> Going to look into this. If you have resolved this, please share that. WIth
> your resolution. Thanks.
>

> --
>
>

[Senthil Kumar]

its clear from the pic that datanode not running... please attach the datanode logs... not webpage...

waiting for your reply..

Thanks

Senthil

[Rajasri janakiraman]

how to get that datanode logs. I am new to hadoop.
Regards,
Rajasri.J

[Senthil Kumar]

Assuming you are running psuedo distributed cluster

Go to hadoop folder / logs folder / you will find all logs including datanode logs (ending with .log)

[Rajasri janakiraman]

i got the log file from /hadoop/logs/hadoop-hduser-datanode-ubuntu.log..

[Senthil Kumar]

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs

It clear states the error...follow the steps below

stop the cluster

Format the namenode... 

delete the datanode dir ( /app/hadoop/tmp/dfs)

restart the cluster

put the file in hdfs

run cat or ls 

run MR wordcount program..

Sorry i am in urgent to catch my train... will reply from mobile.

[Sai Sivam]

It worked for me. The wordcount example

hadoop 1.0.4 and JRE 1.6.0update37

It counted the number of words.
I see the result in the specified HDFS directory.

Please connect with me on Skype. My Skype id is sai.orzota.

We can help resolve your thing. We ran into ssh issues prior to this. But that was resolved.

[Sai Sivam]

org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /app/hadoop/tmp/dfs/data: namenode namespaceID = 550551875;

You will see datanode shutting node.

I recommend a "jps" to verify all services are running prior to running your example.

Thanks.

If you probably fix that, you should see it work.

Stepping out for dinner. Hopefully you will have resolved it soon.

[Rajasri janakiraman]

sir,
It is working now :) Thank u so much!!!:)
Regards,
Rajasri.J

[Sai Sivam]

Great! Awesome!

Now you are the expert! You could also post your resolution to the group!

Sasi and I had fun doing this. We will also try 0.20.1 to get a feel for it. Thanks for sharing your problem with the group.

Sai Sivam and Sasikumar C

[Senthil Kumar]

Guys,

Stick with 1.1.1 (or any stable versions) and not 0.20.XXX - legacy versions

Your complete APIs differs for development.

Senthil

[Rajasri janakiraman]

hi all.. I am trying to set up a single node cluster in my Desktop
(Ubuntu 12.10, RAM-1GB). When i run the wordcount program i am
getting error stating " Cannot run bash file. Cannot allocate memory"
. I have attached the screen shot fot further details...Can some one
help me with this problem. I am using
"http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/"
for reference. Thanks in advance. I have successfully installed
another cluster in my laptop. But this problem persists in my Desktop.
Is this because of RAM capacity? I have set "export
HADOOP_CLIENT_OPTS="-Xmx2048m $HADOOP_CLIENT_OPTS" hadoop-env.sh as
suggested by another site ( to handle java heap space exception ) .
Thanks in advance!:)

[Sivakumar Rajasundaram]

Hi,

I am guessing this issue may be because of JVM memory allocation. As u said ur system having 1 GB of RAM but u allocated 2GB(-Xmx2048m) to JVM.

Can u allocate lesser than 1GB (-Xmx512m or -Xmx1024m) & try? 

Regards,
-Siva
91 9500145827

--

[Subu]

Hi,

Rightly said by Siva, it is the issue related to memory allocation.

Here bit explanation on this two parameter-

-Xmx512m  - It is the minimum memory is allocated to the specific JVM from available physical RAM, when it get launched.

-Xmx1024m - This is max memory can be allocated to the specific JVM from the Phisical RAM.

 

More explanation- let's say we have a system having 8GB phisycal RAM

and we have set the parameter as above ..now what happen when the JVM requires more than 1GB RAM..will it be allocated to the JVM from Physical RAM as we have 8GB.

Answer is no. when it reaches to 1GB and requirres more memory..it calls the gurbage collector...to free up some memory..then if free up memory is enough to hold the new object it is good..else we will get Memoty out of exception error.

 

If you want to observe it through a graph..please grab a copy of HP Java Diagnostics Profiler and plug it into your JVM...you can observe all memory allocation....in pictorial view...

 

Regards

Subrata

[Rajasri janakiraman]

Thank u sir...Mine is only 512MB...Heard that wont be enuf to set up a
node in hadoop...
Regards,
Rajasri