Hack HDFS

[Esash]

Hi All,

I am learning Hadoop and the Map-Reduce techniques. I understood the overview of how HDFS works. I want to understand how the blk_* files in the tmp/data/current directory is used to construct the blk*.meta files and what purpose they serve. What is the code path I am supposed to venture into to understand this ? Is it possible to manually construct these files ?

Please guide.

Thanks a lot, in advance.

Esash

[Julien Muller]

Hello, 

This is a french speaking mailing list.

For help in english, you might try apache lists:

http://hadoop.apache.org/mailing_lists.html

Best Regards,

Julien

2013/10/12 Esash <theemer...@gmail.com>

--
Vous recevez ce message, car vous êtes abonné au groupe Google Groupes Hadoop User Group France.
Pour vous désabonner de ce groupe et ne plus recevoir d'e-mails le concernant, envoyez un e-mail à l'adresse hadoop-user-group-...@googlegroups.com.
Pour plus d'options, visitez le site https://groups.google.com/groups/opt_out .

[Guillaume Polaert]

Hi,

 

According to « the Definitive Guide »,

 

The other files in the datanode’s current storage directory are the files with the blk_ prefix. There are two types: the HDFS blocks themselves (which just consist of the file’s raw bytes) and the metadata for a block (with a .meta suffix). A block file just consists of the raw bytes of a portion of the file being stored; the metadata file is made up of a header with version and type information, followed by a series of checksums for sections of the block.

 

As far as I know, it’s not possible to rebuild the origin file from blk* without namenode metadata. I my opinion there is not value to rebuild .meta for each block.

As Julien said, you may find a better response on the Apache ML.

 

Guillaume

 

De : hadoop-user-...@googlegroups.com [mailto:hadoop-user-...@googlegroups.com] De la part de Julien Muller
Envoyé : samedi 12 octobre 2013 23:15
À : hadoop-user-...@googlegroups.com
Objet : Re: [HUG France] Hack HDFS