Re: converting bin file to java

[Fady]

The file you are trying to read comes from the legstar PDI or Talend projects. It is not a character file and records do not end with line feeds. Therefore, you cannot use BufferedReader.

The file contains variable size records which makes it hard to read with regular java classes such as FileInputStream.

The legstar PDI and Talend projects have special classes meant to read such files. For an example, you can look at http://code.google.com/p/legstar-pdi/source/browse/trunk/legstar.pdi.zosfile/src/main/java/com/legstar/pdi/io/ZosVariableFileInputStream.java.

Also you seem to be using the PersonalData record where CustomerData is what describes a record in the file you attached.

[Fady]

Ok, attached is a working sample implemented as a JUnit test.

Give it a try and let us know if it fits your needs.

Fady

[aaparna...@gmail.com]

Thanks a lot Fady, This really helped.

 

We are now trying to run it in Map Reduce and presently we are conidering only 1 mapper and experimenting with it. we thought of writing a Custom Inputformat Class and read the entire file into the buffer in other words use the zosfileinputstream in the inputformat class and read the entire file into the record reader.

 

Do  you think this is the right approach?

 

Thanks,

Aparna

On Friday, December 21, 2012 1:09:09 PM UTC+5:30, aaparna...@gmail.com wrote:

We are converting bin file to Java and are using legstar,   we have tried to put byteArray of  .bin file ( which read through buffered reader) into method hostToJavaTransform(byte[] HostByte) , but it did not transform and give errors: Want to know if this is the right approach

This is the javaCode:

import java.io.BufferedReader;
import java.io.BufferedWriter;
import java.io.ByteArrayOutputStream;
import java.io.DataOutputStream;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Arrays;
import com.legstar.coxb.transform.HostTransformException;
import com.legstar.test.coxb.cobol.PersonalData;
import com.legstar.test.coxb.cobol.bind.PersonalDataTransformers;
import com.legstar.coxb.transform.AbstractTransformers;
import com.legstar.coxb.transform.HostTransformStatus;

public class ReadMainframe {
public  void hostToJavaTransform(final byte[] mainframe) throws HostTransformException
     {
        PersonalDataTransformers transformers = new PersonalDataTransformers();
        PersonalData personal = transformers.toJava(mainframe);
        personal.getCustomerName().trim();
        personal.getCustomerAddress().trim();
        personal.getCustomerPhone().trim();
     System.out.println(personal.getCustomerName().trim());
   
    }
    public static void main(String[]args) throws Exception
    {
         BufferedReader br = new BufferedReader(new FileReader("/home/hadoop/Desktop/cobol/ZOS.FCUSTDAT.bin"));
        try (BufferedReader reader = new BufferedReader(br))
        {  byte[] mainframeOut =null;
            String line = null;
            while ((line = reader.readLine()) != null)
            {
                System.out.println(line);
                mainframeOut = line.getBytes();
                System.out.println(Arrays.toString(mainframeOut));
            }
            ReadMainframe read = new ReadMainframe();
            read.hostToJavaTransform(mainframeOut);
            }catch (IOException x)
            { System.err.format("IOException: %s%n", x);}
       
       
    }
}

 

Thanks, Aparna

[Fady]

Hello Aparna,

The sample file you are using is characteristic of what you would get if you did a raw FTP of a z/OS variable record VSAM file's content. This is explained here: http://code.google.com/p/legstar-pdi/wiki/GetStarted#How_the_sample_z/OS_files_were_created.

Because that file does not contain any record delimiters, it is hard to use it with a Map Reduce inputformat that would allow multiple mappers to process different parts of the file simultaneously. 

So I think I understand why you are stuck to reading the entire file in a single mapper. I guess from a Hadoop perspective the file is unsplittable.

Is that the right approach? I don't think so. It kind of defeats the whole idea of the Mapping phase which is to split the work between multiple mappers.

I am not familiar enough with Hadoop but I would assume anyone with a variable size record file would have a similar problem. Have you looked around for solutions?