Named Outputs with CDH4.3 mr1 are not working properly for me

[Alexei Perelighin]

Hi,

I have code that uses Named Outputs just fine, here is how they are initialized:

 mr.setOutput(new Path(args[2]), this.getOutpuFormat(), ITuple.class, NullWritable.class);

//FAILURE.SDP_FSSG

mr.addNamedOutput(FailureSchema.outputFolderName, new HadoopOutputFormat(TextOutputFormat.class), ITuple.class, NullWritable.class);

//FAILURE.FSSG

mr.addNamedOutput(fssg_failure_dir, new HadoopOutputFormat(TextOutputFormat.class), ITuple.class, NullWritable.class);

//OUTPUT.FSSG

mr.addNamedOutput(fssg_output_dir, new PangoolSequenceOutputFormat("\u0001"), ITuple.class, NullWritable.class);//output decorated flume output

//OUTPUT.SDP_FSSG

for (String natco : NATCO) {

mr.addNamedOutput(natco, this.getOutpuFormat(), ITuple.class, NullWritable.class);//output per natco

}

When I look at the file names passed to the RecordWriters using FileOutputFormat.getDefaultWorkFile in CDH3 I get:

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/fssg/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/UK/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/DE/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/NL/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/HR/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/GR/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/AT/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local_0002_r_000000_0/RO/_temporary/_attempt_local_0002_r_000000_0/part-r-00000

But With CDH4.3:

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

2013-09-06 16:25:25,029 [main] INFO  org.apache.hadoop.mapred.JobClient -  map 100% reduce 0%

2013-09-06 16:25:30,637 [communication thread] INFO  org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce

2013-09-06 16:25:31,032 [main] INFO  org.apache.hadoop.mapred.JobClient -  map 100% reduce 67%

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

2013-09-06 16:25:45,638 [communication thread] INFO  org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce

2013-09-06 16:25:48,638 [communication thread] INFO  org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/DE/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/DE/_temporary/_attempt_local1331573145_0002_r_000000_0/UK/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/DE/_temporary/_attempt_local1331573145_0002_r_000000_0/UK/_temporary/_attempt_local1331573145_0002_r_000000_0/AT/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/DE/_temporary/_attempt_local1331573145_0002_r_000000_0/UK/_temporary/_attempt_local1331573145_0002_r_000000_0/AT/_temporary/_attempt_local1331573145_0002_r_000000_0/GR/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/DE/_temporary/_attempt_local1331573145_0002_r_000000_0/UK/_temporary/_attempt_local1331573145_0002_r_000000_0/AT/_temporary/_attempt_local1331573145_0002_r_000000_0/GR/_temporary/_attempt_local1331573145_0002_r_000000_0/RO/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

TRACE FILE file:/tmp/pangool/OUTPUT.SDP_FSSG/_temporary/_attempt_local1331573145_0002_r_000000_0/fssg/_temporary/_attempt_local1331573145_0002_r_000000_0/NL/_temporary/_attempt_local1331573145_0002_r_000000_0/DE/_temporary/_attempt_local1331573145_0002_r_000000_0/UK/_temporary/_attempt_local1331573145_0002_r_000000_0/AT/_temporary/_attempt_local1331573145_0002_r_000000_0/GR/_temporary/_attempt_local1331573145_0002_r_000000_0/RO/_temporary/_attempt_local1331573145_0002_r_000000_0/HR/_temporary/_attempt_local1331573145_0002_r_000000_0/part-r-00000

As a result of this, when the work files are commited I loose most of the named outputs.

Has anybody encountered such issues?

Thanks,

Alexei

[Pere Ferrera]

Hello Alexei,

Do you use the latest version of Pangool? We had to refactor the named outputs code for making it compatible with both MR1 and MR2... But maybe CDH4.3 has some particularity we need to take a look at... In this case I will address this issue on Monday and let you know.

--
Has recibido este mensaje porque estás suscrito al grupo "pangool-user" de Grupos de Google.
Para anular la suscripción a este grupo y dejar de recibir sus correos electrónicos, envía un correo electrónico a pangool-user...@googlegroups.com.
Para obtener más opciones, visita https://groups.google.com/groups/opt_out.

[Alexei Perelighin]

Hi Pere,

Here are my maven depnedencies:
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.11</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>joda-time</groupId>
            <artifactId>joda-time</artifactId>
            <version>1.6</version>
        </dependency>
        <dependency>           
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.1</version>
        </dependency>
        <dependency>           
            <groupId>org.apache.commons</groupId>
            <artifactId>commons-lang3</artifactId>
            <version>3.1</version>
        </dependency>
        <dependency>
            <groupId>org.apache.avro</groupId>
            <artifactId>avro</artifactId>
            <version>1.7.4</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-core</artifactId>
            <version>2.0.0-mr1-cdh4.3.0</version>
        </dependency>
        <dependency>
            <groupId>com.datasalt.pangool</groupId>
            <artifactId>pangool-core</artifactId>
            <version>0.60.3</version>
            <classifier>mr2</classifier>
        </dependency>

[Pere Ferrera Bertran]

Hello Alexei,

I could reproduce your issue... I will work on a fix today.

2013/9/8 Alexei Perelighin <alex...@googlemail.com>

--
Pere Ferrera

CTO & Co-founder

www.datasalt.com

[Alexei Perelighin]

Hi Pere,

Excellent news.

BTW, Are you using the same source code for CDH3 and CDH4?

Thanks,

Alexei

[Pere Ferrera Bertran]

Hello Alexei,

Yes, we use the same code because, as of now, to make Pangool compatible with MR1 and MR2 we just needed to implement some reflexion tricks. In runtime Pangool knows what kind of environment is running on and may execute slightly different business logic. This simplifies our code base, granted we won't need to deal with too many special cases.

I just found the issue... There was a place where a Configuration object was not properly cloned. For some reason this was not a problem on our MR1 and MR2 jenkins automated tests... But I will add CDH4.3 to the automated suite.

I will try to release a new artifact between today and tomorrow.

2013/9/9 Alexei Perelighin <alex...@googlemail.com>

[Pere Ferrera]

I am releasing a new version with this issue fixed, ... It will be 0.60.5, will tell you when it's available for download.

[Alexei Perelighin]

Would it be a stable release ready for production systems?

[Pere Ferrera]

Yes, it will be just as stable and production-ready as the current one, with this issue fixed (I just added a new Configuration(...) line).

[Pere Ferrera]

Hello Alexei,

Can you try new release 0.60.6 and confirm that it solves your problem?

Para anular la suscripción a este grupo y dejar de recibir sus correos electrónicos, envía un correo electrónico a pangool-user+unsubscribe@googlegroups.com.

Para obtener más opciones, visita https://groups.google.com/groups/opt_out.

--
Has recibido este mensaje porque estás suscrito al grupo "pangool-user" de Grupos de Google.

Para anular la suscripción a este grupo y dejar de recibir sus correos electrónicos, envía un correo electrónico a pangool-user+unsubscribe@googlegroups.com.

[Alexei Perelighin]

Hi Pere,

The NamedOutputs are working with 0.60.6 mr1 and CDH4.3. Thanks a lot for the quick fix :D

Thanks,

Alexei