Does version of Hadoop matter?

[Xiaobo.Gu]

Hi,

I am trying Nectar with a local 0.20.203 Hadoop cluster with two
datanodes, there is no output from sigmax(1)<<book.csv, and
sigmax(1) << book.csv does not return.

[madhukara phatak]

We tested the framework with 0.21.0 though we dint used any version specific features.Just make sure following things.

1. Your are running the modeler right from the user on which you installed Hadoop.

 

2. Configuration file points to the correct hadoop installation directory 

3. You placed Nectar-analytics jar inside hadoop home directory 

4. The file is available in HDFS input directory and it is tab separated.

--
Regards

Madhukara Phatak

[Xiaobo.Gu]

The same result with Hadoop 0.21.0, by the way, I am on a Solaris 10u8
X64 virtual machine with Oracle JDK 1.60u23.

[madhukara phatak]

Please can you see job tracker web UI to see running command creates any job or not  . Or any job is failing etc

--
Regards

Madhukara Phatak

[Xiaobo.Gu]

Yes to all your questions, from Hadoop UI there are not any running
jobs, and no clue in the jobtracker log indicating that there is ever
a job submitted.


On Jul 24, 9:04 pm, madhukara phatak

<madhukarapha...@zinniasystems.com> wrote:
> Please can you see job tracker web UI to see running command creates any job
> or not  . Or any job is failing etc
>

[Xiaobo.Gu]

-bash-3.00$ more configuration.cfg
HADOOP_HOME=/usr/local/hadoop/
NO_OF_PARALLEL_JOBS=2
-bash-3.00$

[madhukara phatak]

ok there is adhoc way of testing the setup . Run the following command in hadoop home

bin/hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob  input/book.csv(or ur file) output234233

--
Regards

Madhukara Phatak

[Xiaobo.Gu]

-bash-3.00$ hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
book.csv
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob.main(SigmaJob.java:
88)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
-bash-3.00$

On Jul 24, 9:13 pm, madhukara phatak

[madhukara phatak]

Sorry the command is 

bin/hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob  input/book.csv(or ur file) output234233 1

--
Regards

Madhukara Phatak

[Xiaobo.Gu]

I can see the output234233 directory created in HDFS, but it's a empty
directory after the command return.

On Jul 24, 9:17 pm, "Xiaobo.Gu" <guxiaobo1...@gmail.com> wrote:
> -bash-3.00$ hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
> com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
> book.csv
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
>         at

> com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob.main(SigmaJob.j­ava:

> 88)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39)
>         at

> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp­l.java:

[Xiaobo.Gu]

-bash-3.00$ bin/hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
book.csv output234233 1
11/07/24 21:55:18 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/07/24 21:55:18 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id
11/07/24 21:55:18 WARN mapreduce.JobSubmitter: Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
11/07/24 21:55:19 INFO mapreduce.JobSubmitter: Cleaning up the staging
area hdfs://solsvr/tmp/hadoop-gpadmin/mapred/staging/gpadmin/.staging/job_201107242151_0002
-bash-3.00$ bin/hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
book.csv output234233 1
11/07/24 21:55:42 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/07/24 21:55:42 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id
11/07/24 21:55:42 WARN mapreduce.JobSubmitter: Use
GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.
11/07/24 21:55:43 INFO mapreduce.JobSubmitter: Cleaning up the staging
area hdfs://solsvr/tmp/hadoop-gpadmin/mapred/staging/gpadmin/.staging/job_201107242151_0003
java.io.IOException: Target /tmp/output234233/output234233 is a
directory
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:377)
at org.apache.hadoop.fs.FileUtil.checkDest(FileUtil.java:379)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:226)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:216)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:167)
at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1543)
at
org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1524)
at
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob.createRunHadoop(SigmaJob.java:
82)
at
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob.main(SigmaJob.java:
91)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

^C-bash-3.00$

On Jul 24, 9:17 pm, "Xiaobo.Gu" <guxiaobo1...@gmail.com> wrote:

> -bash-3.00$ hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
> com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
> book.csv
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
>         at

> com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob.main(SigmaJob.j­ava:

> 88)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39)
>         at

> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp­l.java:

[madhukara phatak]

Ok u tried to run the command multiple times so you have to change the output directory every time and just make sure that user has a permission to read/write on /tmp.

--
Regards

Madhukara Phatak

[Xiaobo.Gu]

I think there is something wrong with my local hadoop

-bash-3.00$ bin/hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
book.csv output234233 1
11/07/24 22:00:35 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
cacheTimeout=300000
11/07/24 22:00:36 WARN conf.Configuration: mapred.task.id is
deprecated. Instead, use mapreduce.task.attempt.id
11/07/24 22:00:36 WARN mapreduce.JobSubmitter: Use

GenericOptionsParser for parsing the arguments. Applications should
implement Tool for the same.

11/07/24 22:00:36 INFO input.FileInputFormat: Total input paths to
process : 1
11/07/24 22:00:36 WARN conf.Configuration: mapred.map.tasks is
deprecated. Instead, use mapreduce.job.maps
11/07/24 22:00:36 INFO mapreduce.JobSubmitter: number of splits:1
11/07/24 22:00:36 INFO mapreduce.JobSubmitter: adding the following
namenodes' delegation tokens:null

The command hang for a while and can't proceed.

On Jul 24, 9:17 pm, "Xiaobo.Gu" <guxiaobo1...@gmail.com> wrote:

> -bash-3.00$ hadoop jar Nectar-analytics-0.0.1-SNAPSHOT.jar
> com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob input/
> book.csv
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
>         at

> com.zinnia.nectar.regression.hadoop.primitive.jobs.SigmaJob.main(SigmaJob.j­ava:

> 88)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
> 39)
>         at

> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImp­l.java:

[madhukara phatak]

Take a fresh copy of hadoop-0.21.0.tar.gz and setup the single node setup and then try to run the jar. So if you get any issue feel free to post here.

--
Regards

Madhukara Phatak

[Xiaobo.Gu]

I configed mapred-site.xml with this

<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx256m</value>
<!-- Not marked as final so jobs can include JVM debugging options --
>
</property>

Does it conflict with NO_OF_PARALLEL_JOBS=2



On Jul 24, 10:05 pm, madhukara phatak

[madhukara phatak]

Yeah.

Please increase the values above 2 and then test . Just see in UI , is map and reduce is completing successfully .

--
Regards

Madhukara Phatak

[David Jacobs]

Nice  Post