Marshalling nested unbounded sized collections to fixed length flat file

[James Elsey]

Hi,

I am trying to marshall some objects into a fixed length flat file, but am having some difficulties understanding how I can marshal beans that have nested, unbounded sized collections.

For the purposes of this question, I've simplified my domain structure to the following : 

public class GarageWrapper {

    private Header header;
    private List<Car> cars;
    private Footer footer;
     
     // getters & setters omitted
}

public class Header {

    private String garageName = "Moes";
     // getters & setters omitted
}

public class Car {
    private String carName = "Mondeo";
    private List<Colour> colours;
     // getters & setters omitted
}

public class Colour {
    private String colourName = "red";
     // getters & setters omitted
}

public class Footer {
    private String locationName;
     // getters & setters omitted
}


This is my beanio.xml file

<beanio xmlns="http://www.beanio.org/2012/03"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.beanio.org/2012/03 http://www.beanio.org/2012/03/mapping.xsd">

    <stream name="garageStream" format="fixedlength">
        <parser>
            <!--Set recordTerminator to null, this prevents new lines between records-->
            <property name="recordTerminator" value=""/>
        </parser>

        <record name="garageWrapper" class="GarageWrapper">

            <segment name="header" class="Header">
                <field name="garageName" length="4"/>
            </segment>

            <segment name="cars" class="Car" collection="list" >
                <field name="carName" length="6"/>

                <segment name="colours" class="Colour" collection="list" minOccurs="1" maxOccurs="unbounded">
                    <field name="colourName" length="5"/>
                </segment>

            </segment>

            <segment name="footer" class="Footer">
                <field name="locationName" length="7"/>
            </segment>

        </record>

    </stream>

</beanio>

I want to pass the marshaller a single garageWrapper, this contains a header and a footer, and between a list of cars. The number of cars in this list can vary. Each individual car will have a carName field that will always appear. Each car also has a varying list of colours.

If I execute the above with some sample data for 2 cars, I get the following output:

MoesMondeoWhiteBlackGreenIpswich

This has correctly marshalled the header, footer, but has only included a single car element, and has missed out "EscortCreamBeige". I believe this is because the segment named "cars" has no maxOccurs="unbounded" set, so it defaults to just a single element.

If I put maxOccurs="unbounded" onto the cars segment, I get the following exception:

Caused by: org.beanio.BeanIOConfigurationException: A segment of indeterminate size may not follow another component of indeterminate size

Having followed the documentation guidelines for record grouping (http://www.beanio.org/2.0/docs/reference/index.html#RecordGrouping) I have come up with the following alternative

<beanio xmlns="http://www.beanio.org/2012/03"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.beanio.org/2012/03 http://www.beanio.org/2012/03/mapping.xsd">

    <stream name="garageStream" format="fixedlength">
        <parser>
            <!--Set recordTerminator to null, this prevents new lines between records-->
            <property name="recordTerminator" value=""/>
        </parser>

        <record name="header" order="1" minOccurs="1" maxOccurs="1" class="Header">
            <field name="garageName" length="4"/>
        </record>
        <group name="cars" order="2" minOccurs="0" maxOccurs="unbounded">
            <record name="car" order="1" minOccurs="1" maxOccurs="1" class="Car">
                <field name="carName" length="6"/>
                <segment name="colours" class="Colour" collection="list" minOccurs="1" maxOccurs="unbounded">
                    <field name="colourName" length="5"/>
                </segment>
            </record>
        </group>
        <record name="footer" order="3" minOccurs="1" maxOccurs="1" class="Footer">
            <field name="locationName" length="7"/>
        </record>

    </stream>

</beanio>

Which gives the following 
Exception in thread "main" org.beanio.BeanWriterException: Bean identification failed: No record or group mapping for class 'class GarageWrapper' at the current stream position

Adding in a top level group for the wrapper, and tweaking the cars group, I use the following:

<beanio xmlns="http://www.beanio.org/2012/03"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://www.beanio.org/2012/03 http://www.beanio.org/2012/03/mapping.xsd">

    <stream name="garageStream" format="fixedlength">
        <parser>
            <!--Set recordTerminator to null, this prevents new lines between records-->
            <property name="recordTerminator" value=""/>
        </parser>

        <group name="garageWrapper" class="GarageWrapper">
            <record name="header" order="1" minOccurs="1" maxOccurs="1" class="Header">
                <field name="garageName" length="4"/>
            </record>
            <group name="cars" order="2" minOccurs="0" maxOccurs="unbounded">
                <record name="cars" order="1" minOccurs="0" maxOccurs="unbounded" class="Car" collection="list">
                    <field name="carName" length="6"/>
                    <segment name="colours" class="Colour" collection="list" minOccurs="1" maxOccurs="unbounded">
                        <field name="colourName" length="5"/>
                    </segment>
                </record>
            </group>
            <record name="footer" order="3" minOccurs="1" maxOccurs="1" class="Footer">
                <field name="locationName" length="7"/>
            </record>

        </group>
    </stream>

</beanio>

Which then gives :
Exception in thread "main" org.beanio.BeanWriterException: Record groups not supported by Marshaller

What am I doing wrong here? 


For the record, this is the main method I'm using

import org.beanio.Marshaller;
import org.beanio.StreamFactory;

import java.util.Arrays;

public class BeanioRunner {

    public static void main(String[] args) {
        new BeanioRunner().run();
    }

    private void run() {
        StreamFactory factory = StreamFactory.newInstance();
        factory.loadResource("beanio.xml") ;
        Marshaller marshaller = factory.createMarshaller("garageStream");
        GarageWrapper garageWrapper = new GarageWrapper();

        //Header
        Header header = new Header();
        garageWrapper.setHeader(header);

        // Cars
        Car mondeo = new Car("Mondeo", Arrays.asList(new Colour("White"), new Colour("Black"), new Colour("Green")));
        Car escort = new Car("Escort", Arrays.asList(new Colour("Cream"), new Colour("Beige")));
        garageWrapper.setCars(Arrays.asList(mondeo, escort));

        // Footer
        Footer footer = new Footer();
        footer.setLocationName("Ipswich");
        garageWrapper.setFooter(footer);

        String fileRecord = marshaller.marshal(garageWrapper).toString();
        System.out.println(fileRecord);
    }
}

The output I'm after is:

MoesMondeoWhiteBlackGreenEscortCreamBeigeIpswich

    
I was working with version 2.0.2, but have also tried this on the latest 2.1.0.M1

[James Elsey]

@Kevin,

I'm half expecting you to reply suggesting that I shouldn't be using nested unbounded collections, as the reader of the stream would not know where one collection ends and the other starts.

In my scenario each record will be prepended with a constant (such as HDR, FTR, CAR, CLR), so that the consumer can read the line and know where to split based on the constants. Also, the actual values are of fixed length, so after CAR they would know that the first 5 characters are the carName etc.

Is "A segment of indeterminate size may not follow another component of indeterminate size" a preventative measure that beanio implies? Is there a way of overriding this?

[Kevin]

Hi James,

You cannot have an unbounded segment nested in another unbounded segment without another field that explicitly controls the number of occurrences.  And you cannot use a Marshaller or Unmarshaller for processing groups of records (since a Marshaller/Unmarshaller is by nature used to process only a single record).  Instead, a BeanReader / BeanWriter must be used for reading and writing record groups.

Your post is rather confusing.  You mention each record will be prepended with HDR, FTR, CAR, etc but then say the output should be "MoesMondeoWhiteBlackGreenEscortCreamBeigeIpswich" which doesn't include any of the identifiers, so I'm not sure what you are truly after.

Thanks,

Kevin

[James Elsey]

Apologies, I should have included an example on my 2nd post

HDRMoesCARMondeoCLRWhiteCLRBlackCLRGreenCAREscortCLRCreamCLRBeigeFTRIpswich 

The presence of a constant code before each segment enables the reader of this flat file to understand which collection a field belongs to, if it starts with CLR it will recognise it belongs to the inner collection. This requirement is imposed by the consumer of the flat file.

Do you have any examples of how I can marshal nested unbounded collections?

Is it possible to relax the "A segment of indeterminate size may not follow another component of indeterminate size" and put full responsibility on the reader of the file to understand where separate collections are?

[Kevin]

Hi James,

Do you only care about marshalling / writing that format, and not unmarshalling / reading it?

If so, your mapping file with record groups might work with a BeanWriter, although you have an extra group layer.

    <stream name="garageStream" format="fixedlength">
        <parser>
            <!--Set recordTerminator to null, this prevents new lines between records-->
            <property name="recordTerminator" value=""/>
        </parser>
        <group name="garageWrapper" class="GarageWrapper">
            <record name="header" order="1" minOccurs="1" maxOccurs="1" class="Header">
                <field name="garageName" length="4"/>
            </record>

            <record name="cars" order="2" minOccurs="0" maxOccurs="unbounded" class="Car" collection="list">

                <field name="carName" length="6"/>
                <segment name="colours" class="Colour" collection="list" minOccurs="1" maxOccurs="unbounded">
                    <field name="colourName" length="5"/>
                </segment>
            </record>

            <record name="footer" order="3" minOccurs="1" maxOccurs="1" class="Footer">
                <field name="locationName" length="7"/>
            </record>
        </group>
    </stream>

Thanks,
Kevin

[James Elsey]

I managed to find a work around, figured I'd post it here as it may help others.

Instead of having collections in the beanio.xml file under one giant stream, I split them up into separate streams, so a stream for the header, another for the trailer, and a separate one for each of the collection items.

Then I created a service class that iterates over the collections and marshalls them separately, concatenating them at the end. This moves the complexity away from the beanio config and into a class where you have more control over those nested collections.

Thanks