Hadoop format using puppet

[Bai Shen]

I'm using puppet to automate the creation and maintenance of my hadoop cluster.  However, I'm not sure how to handle this next step.  In order for the cluster to run, the namenode needs to be formatted.

su hdfs -c "/usr/bin/hadoop namenode -format"



I'm not sure what the best method to handle this is.  Do I just add an exec command?  How do I make sure that it only gets run during the initial setup?

Thanks.

[Denmat]

Hi,

Tricky one. In the past I have used a touch file to test against (if touched file there don't do it - but that is a little insecure). 

After the filesystem is formatted I believe you should be able to list it. However when I was using cloudera Hadoop you that could return your local filesystem! So your test would always pass.

So maybe a format and touching a file in the hdfs would work?  Sorry don't work with Hadoop any more so cant test. 

You can have a look at the following:

https://github.com/denmat/hadoop_in_a_box

That was last approach (can't remember how I did it off hand).

Cheers

Den

--
You received this message because you are subscribed to the Google Groups "Puppet Users" group.
To post to this group, send email to puppet...@googlegroups.com.
To unsubscribe from this group, send email to puppet-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/puppet-users?hl=en.

[fsalum]

I did my hadoop manifest as below:

  exec { "Namenode - Format dfs.name.dir":
    path    => '/usr/bin:/usr/sbin:/bin',
    command => 'echo "Y" | hadoop namenode -format',
    user    => 'hdfs',
    unless  => "test -d $dfs_name_dir/image",
    require => Exec['Namenode - Create dfs.name.dir'],
  }

Regards,
Felipe