A question about hadoop.tmp.dir

[Hao Huang]

I built the hadoop-pseudo-cluster using CDH, now facing problem of looking for /tmp folder.

Here are what I did following CDH guide:

sudo -u hdfs hadoop fs -mkdir /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp


sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred


sudo -u hdfs hadoop fs -mkdir /user/hao
sudo -u hdfs hadoop fs -chown hao /user/hao

Now I run command

hao@rhadoop:~$ sudo -u hdfs hadoop fs -mkdir /tmp
mkdir: `/tmp': File exists

So I go to file /etc/hadoop/conf/hdfs-site.xml, it shows

<property>
 <name>hadoop.tmp.dir</name>
 <value>/var/lib/hadoop-hdfs/cache/${user.name}</value>
 </property>

And if I do:

hao@rhadoop:~$ ls /var/lib/hadoop-hdfs/cache/
hdfs/   mapred/ root/  

So there is no folder called hao/, which is different as what I thought.

I'm so new to hadoop, anyone can help me with this?

[Ashwanth Kumar]

From the core-default.xml documentation,

hadoop.tmp.dir - A base for other temporary directories.

Typically inside ${hadoop.tmp.dir}/hdfs/ Hadoop stores the DFS Blocks (not the actual files / directories), and If I understand it right your fs.default.name / fs.defaultFS (depending on the hadoop version 1 / 2 resp) still points to file:/// (refer above link for defaults). This means that your HDFS still runs out of your local filesystem. 

Say you do this - 

$ hadoop dfs -mkdir $HOME/mynewfolder

It will create mynewfolder in your home directory. 

You need to change the fs.default.name to point to hdfs://host@port/ or s3n://key:secret@buckname/ depending on your requirement. 

--
You received this message because you are subscribed to the Google Groups "Hadoop Users Group (HUG) Chennai" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chennaihug+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Ashwanth Kumar / ashwanthkumar.in

[Hao Huang]

Thanks for pointing out that inside ${hadoop.tmp.dir}/hdfs/ Hadoop stores the DFS Blocks. And here are what I'm concerning:

1) My fs.default.name is pointing to myhost:myport , not to local file system.

2) I don't understand why I should do $ hadoop dfs -mkdir $HOME/mynewfolder, what will this give to me?

2) CDH's core-default.xml is stored in a jar file instead of conf folder and it can not be modified, right? if I understand correctly, core-site.xml is supposed to replace some default settings in core-default.xml when you try to establish connection from outside to hdfs. In my case, I'm trying to copy file from local to remote pseudo-cluster's hdfs, and could not find where exactly I copied file to.

Say my java code is:

Configuration conf = new Configuration();

FileSystem fs = FileSystem.get(conf);  

 fs.copyFromLocalFile("C:/test.cvs", "hdfs://172.16.1.39:8020/tmp/test.csv");

Is there a folder named as "tmp" or the "tmp" folder is   ${hadoop.tmp.dir}? And am I able to find a file "test.csv" somewhere or only blocks of "test.cvs" in that folder?

2013/7/15 Ashwanth Kumar <ashwan...@googlemail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Hadoop Users Group (HUG) Chennai" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/chennaihug/-g4llEgxMUc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to chennaihug+...@googlegroups.com.

[Ashwanth Kumar]

On Wed, Jul 17, 2013 at 5:11 AM, Hao Huang <hhu...@gmail.com> wrote:

Thanks for pointing out that inside ${hadoop.tmp.dir}/hdfs/ Hadoop stores the DFS Blocks. And here are what I'm concerning:

1) My fs.default.name is pointing to myhost:myport , not to local file system.

2) I don't understand why I should do $ hadoop dfs -mkdir $HOME/mynewfolder, what will this give to me?

I said that so that you can check if the folder gets created on your home directory if the fs.default.name is file:/// 

 

2) CDH's core-default.xml is stored in a jar file instead of conf folder and it can not be modified, right? if I understand correctly, core-site.xml is supposed to replace some default settings in core-default.xml when you try to establish connection from outside to hdfs. In my case, I'm trying to copy file from local to remote pseudo-cluster's hdfs, and could not find where exactly I copied file to.

Ah, you want to see the HDFS files. Then do $ hadoop dfs -ls /hdfs/path/to/list

HDFS is a virtual filesystem, you can't browse the files from a file manager that is used for browsing the local file system. 

 

Say my java code is:

Configuration conf = new Configuration();

FileSystem fs = FileSystem.get(conf);  

 fs.copyFromLocalFile("C:/test.cvs", "hdfs://172.16.1.39:8020/tmp/test.csv");

Is there a folder named as "tmp" or the "tmp" folder is   ${hadoop.tmp.dir}? And am I able to find a file "test.csv" somewhere or only blocks of "test.cvs" in that folder?

You can only find the blocks and not the actual file. If you want to view the actual file use the $ hadoop dfs commands. 

Eg. $ hadoop dfs -cat /tmpt/test.csv

[Hao Huang]

Thank you, Kumar, thank you! You helped me alot!! 

2013/7/17 Ashwanth Kumar <ashwan...@googlemail.com>