"bin/hadoop: No such file or directory"

[Julia Lundin]

Lecture 8 step 4- Stand Alone Mode

jdk 1.8.0_11

jre 1.8.0_11

Ubuntu 64 Mac

Hi!

Setting up Hadoop has been problem-free until step 4 when I was going to test if the Stand Alone Mode worked. I wrote:

bin/hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+' 

it wasn't successful and said: 

bash: bin/hadoop: No such file or directory

In the setup guide it says that you can skip "bin" so I also tried:

 hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+', but then it said: 

Not a valid JAR: /home/julu8216/hadoop/hadoop-examples-*.jar

Any suggestions what the problem can be?

Thanks,

Julia

[Nitesh Jain]

Hi Julia,

Thanks for reaching out, please find the explanation on the errors. 

Explanation for bash: bin/hadoop: No such file or directory

Linux says "No such directory found" when it is not able to find the script that we are asking it to run. In case we use bin/hadoop, we should be in hadoop installations directory at the time we are running this. (in the directory which we marked as $HADOOP_INSTALL). What happens is that by specify bin/hadoop, we are asking Linux to go and execute the script "hadoop" which is present in the bin folder of Hadoop installation. We are relatively pointing to the bin folder, so we should be at the folder above it at the time we run the command.

However, this complexity is resolved once we add the bin folder to the active path, which we do by adding the following two line in profile script:

HADOOP_INSTALL=/home/{user_name}/hadoop/hadoop-1.2.1

PATH=$PATH:$HADOOP_INSTALL/bin

By doing this scripts in hadoop/bin folder becomes accessible from any active directory as it gets included in the active path. 

Explanation for  Not a valid JAR:

Again concept is the same. Linux at this point of time is not able to find the jar file in the active path. So the idea is to be in the directory where the jar file is present, at the time when you run this command. So if you get into hadoop directory by 

cd $HADOOP_INSTALL                       (if this doesn't work, try explicitly giving the hadoop path)

and then run the command

mkdir input                                                (because the next command needs an input directory which can be empty for now) 

and then run the command

hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

This should give you the proper output. Please let me know if this works out good for you! 

Hope this helps!

Best,

Nitesh

Hadoop-Skills.com

[Julia Lundin]

Hi Nitesh,

Thanks for your quick reply. I have now tried to run the first command in the folder you mentioned, and then it did work successfully. I looked in the profile and it looks like I already have added the bin folder to the active path:

JAVA_HOME=/usr/local/java/jdk1.8.0_11

PATH=$PATH:$JAVA_HOME/bin

JRE_HOME=/usr/local/java/jre1.8.0_11

PATH=$PATH:$JRE_HOME/bin

HADOOP_INSTALL=/home/julu8216/hadoop/hadoop-1.2.1

PATH=$PATH:$HADOOP_INSTALL/bin

export JAVA_HOME

export JRE_HOME

export PATH

With this script it should be accessible from any directory, so why doesn't it work?

Cheers,

Julia

Den tisdagen den 5:e augusti 2014 kl. 07:14:56 UTC+2 skrev Nitesh Jain:

Hi Julia,

Thanks for reaching out, please find the explanation on the errors. 

Explanation for bash: bin/hadoop: No such file or directory

Linux says "No such directory found" when it is not able to find the script that we are asking it to run. In case we use bin/hadoop, we should be in hadoop installations directory at the time we are running this. (in the directory which we marked as $HADOOP_INSTALL). What happens is that by specify bin/hadoop, we are asking Linux to go and execute the script "hadoop" which is present in the bin folder of Hadoop installation. We are relatively pointing to the bin folder, so we should be at the folder above it at the time we run the command.

However, this complexity is resolved once we add the bin folder to the active path, which we do by adding the following two line in profile script:

HADOOP_INSTALL=/home/{user_name}/hadoop/hadoop-1.2.1

PATH=$PATH:$HADOOP_INSTALL/bin

By doing this scripts in hadoop/bin folder becomes accessible from any active directory as it gets included in the active path. 

Explanation for  Not a valid JAR:

Again concept is the same. Linux at this point of time is not able to find the jar file in the active path. So the idea is to be in the directory where the jar file is present, at the time when you run this command. So if you get into hadoop directory by 

cd $HADOOP_INSTALL                       (if this doesn't work, try explicitly giving the hadoop path)

and then run the command

mkdir input                                                (because the next command needs an input directory which can be empty for now) 

and then run the command

hadoop jar hadoop-examples-*.jar grep input output 'dfs[a-z.]+'

This should give you the proper output. Please let me know if this works out good for you! 

Hope this helps!

Best,

Nitesh

Hadoop-Skills.com

On Monday, 4 August 2014 20:24:03 UTC+10, Julia Lundin wrote:

[Nitesh Jain]

Hey Julia,

Yes with the the current setup, you need not put "bin/hadoop" before every command and you can just put "hadoop" and the command because all the scripts in bin folder are there in the active path. 

Please see that "hadoop" is actually a script on $HADOOP_INSTALL/bin folder. Other scripts in bin folder are start-all.sh and stop-all.sh and so on. It is an interesting exercise to go in $HADOOP_INSTALL/bin folder and see the scripts. You would understand a little more about how hadoop internally functions. 

So the "hadoop" script is accessible right now from anywhere but the jar file should be present in your active working directory. If jar is not in the current active directory, you would get the error not jar found. Why it worked right now is because the jar file was present in the folder where you were located.

Do let me know if I could explain this properly.

Cheers,

Nitesh

Hadoop-skills.com 

[Rendani Ramabulana]

Hi

I have the same problem on my Mac 

bash: bin/hadoop: No such file or directory