Error parsing EDIFACT with UNA header and comma as decimal separator

[Ulrich Schuster]

I am trying to parse an EDIFACT interchange that uses syntax level 3 and the comma as decimal separator, and where the interchange has the following UNA header segment:

UNA:+,? '

My smooks (2.0.0-M1) setup is as follow (Kotlin code):

val schemaUri = "d04b/EDIFACT-Messages.dfdl.xsd"

val smooks = Smooks()
val reader = EdifactReaderConfigurator(schemaUri).setMessageTypes(myMessageTypes)
reader.cacheOnDisk = true
reader.setDecimalSign(",")
smooks.setReaderConfig(reader)
val filterSettings = FilterSettings.newSaxNgSettings().setReaderPoolSize(5)
smooks.setFilterSettings(filterSettings)
smooks.setExports(Exports(StringResult::class.java))

When I try to parse an interchange with the above UNA header, I get the following error:

org.smooks.api.SmooksException: Failed to filter source

[...]

Caused by: org.smooks.api.SmooksException: Schema Definition Error: Non-distinct property ',' found in: textStandardGroupingSeparator, textStandardDecimalSeparator
Schema context: E0017 Location line 3334 column 14 in jar:file:.m2/repository/org/smooks/cartridges/edi/edi-schemas/2.0.0-RC1/edi-schemas-2.0.0-RC1.jar!/EDIFACT-Common/EDIFACT-Service-Segments-4.1.dfdl.xsd

Parsing does work fine if I omit the UNA segment and if I use the default UNA:+.? ' with decimal point instead of comma – even though my configuration explicitly sets the comma as default decimal separator.

Is there any particular configuration I am missing?

[Ulrich Schuster]

Here is a first hypothesis, from analyzing the sources and the error trace:

It seems that apache Daffodil uses both a decimal separator (textStandardDecimalSeparator) and a thousand group separator (textStandardGroupingSeparator). It seems that the group separator by default is the comma, and that setting the decimal separator to be the comma as well leads to a clash - hence the Daffodil error message that group separator and decimal separator are not distinct.

However, EDIFACT does not know about a group separator; hence, it cannot be set explicitly. So the question is: How can I configure smooks to configure Daffodil to use another group separator, or no group separator at all?

[Ulrich Schuster]

Ok, the smooks reader configuration is called traidSeparator. Hence, I now use 

reader.setDecimalSign(",").setTriadSeparator("~")

I use the tilde (~) just to get the separator out of the way. Proper handling would be to disable it, because EDIFACT does not support a triad separator. Is this possible?

[Claude]

Yeah, I don't think it should be allowed for EDIFACT. Can you create an issue for this? We'll drop it in the next release.

Claude

[Reshma Kumari]

Hi @Claude,

Is this fixed? I am using Smooks - 2.0.0-RC3.
Is there any quick workaround that i can take to fix it?

Thanks,

Reshma

[Reshma Kumari]

As suggested by following post,
https://github.com/smooks/smooks-edi-cartridge/issues/200

I also tried setting following

ResourceConfig edifactUnparser = new DefaultResourceConfig("*", new Properties());
edifactUnparser.setResource("");
edifactUnparser.setParameter("schemaUri", "/"+version+"/EDIFACT-Messages.dfdl.xsd");edifactUnparser.setParameter("cacheOnDisk", "true");

edifactUnparser.setParameter("decimalSign", ",");
edifactUnparser.setParameter("triadSeparator", "~");
edifactUnparser.setResourceType("dfdl");

if(ediVersion == null)
edifactUnparser.setParameter("dataProcessorFactory", "org.smooks.cartridges.edifact.EdifactDataProcessorFactory");
else
edifactUnparser.setParameter("dataProcessorFactory", "com.dpw.bridgeService.services.helper.CustomEdifactDataProcessorFactory");

for (String mesaage:messageType) {
edifactUnparser.setParameter("messageType", mesaage);
}

// Build pipeline resource config programmatically
ResourceConfig pipeline = new DefaultResourceConfig("/Interchange", new Properties());
pipeline.setResource("org.smooks.engine.resource.visitor.smooks.NestedSmooksVisitor");
pipeline.setParameter("action", "REPLACE");
pipeline.setParameter("smooksResourceList", "<smooks-resource-list xmlns=\"https://www.smooks.org/xsd/smooks-2.0.xsd\">" + edifactUnparser.toXml() + "</smooks-resource-list>");

String encoding = "UTF-8";
Smooks smooks = new Smooks(new DefaultApplicationContextBuilder().setClassLoader(JsonToEDITemplateBuilderServiceImpl.class.getClassLoader()).build());
StringResult stringResult = new StringResult();
try {
smooks.addConfiguration(pipeline);
ExecutionContext executionContext = smooks.createExecutionContext();
executionContext.setContentEncoding(encoding);
System.out.println(interchange.toString());
long startTime = System.currentTimeMillis();
smooks.filterSource(executionContext, new ByteSource(interchange.toString().getBytes(encoding)), stringResult);
long elapsedTime = System.currentTimeMillis() - startTime;
log.info("Time taken to finish filter source jsonToEdi : {}" , elapsedTime);
NewRelic.addCustomParameter("jsonToEdiTime", elapsedTime);
}catch (Exception e){
e.printStackTrace();
}

but still getting the same error.

The interchange that i am trying to parse is 

<D01B:Interchange
        xmlns:D01B="http://www.ibm.com/dfdl/edi/un/edifact/D01B"
        xmlns:srv="http://www.ibm.com/dfdl/edi/un/service/4.1"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<UNA><CompositeSeparator>:</CompositeSeparator><FieldSeparator>+</FieldSeparator><DecimalSeparator>,</DecimalSeparator><EscapeCharacter>?</EscapeCharacter><RepeatSeparator> </RepeatSeparator><SegmentTerminator>'</SegmentTerminator></UNA>
<UNB><S001><E0001>UNOC</E0001><E0002>2</E0002></S001><S002><E0004>XYZ</E0004></S002><S003><E0010>KIK</E0010></S003><S004><E0017>190410</E0017><E0019>1012</E0019></S004><E0020>222</E0020></UNB>
    <D01B:Message>
    <UNH><E0062>1</E0062><S009><E0065>INVOIC</E0065><E0052>D</E0052><E0054>01B</E0054><E0051>UN</E0051></S009></UNH>
        <D01B:INVOIC>
          <BGM><C002><E1001>331</E1001></C002><C106><E1004>D-IN25000049</E1004></C106><E1225>9</E1225></BGM><DTM><C507><E2005>3</E2005><E2380>20250417</E2380><E2379>102</E2379></C507></DTM><RFF><C506><E1153>ON</E1153><E1154>PO123456</E1154></C506></RFF><NAD><E3035>BY</E3035><C082><E3039>123456789</E3039><E3055>16</E3055></C082><C080><E3036>Buyer Company Ltd</E3036></C080><C059><E3042>123 Buyer Street</E3042><E3042>Buyer City</E3042></C059><E3251>12345</E3251><E3207>DE</E3207></NAD><NAD><E3035>SE</E3035><C082><E3039>987654321</E3039><E3055>16</E3055></C082><C080><E3036>Seller Company Inc</E3036></C080><C059><E3042>456 Seller Road</E3042><E3042>Seller City</E3042></C059><E3251>54321</E3251><E3207>DE</E3207></NAD><CUX><C504><E6347>2</E6347><E6345>EUR</E6345><E6343>9</E6343></C504></CUX><LIN><E1082>1</E1082><C212><E7140>123456789</E7140><E7143>IN</E7143></C212><PIA><E4347>1</E4347><C212><E7140>ABC123</E7140><E7143>BP</E7143></C212></PIA><IMD><E7077>F</E7077><C273><E7008>Product Description 1</E7008></C273></IMD><QTY><C186><E6063>47</E6063><E6060>10</E6060></C186></QTY><MOA><C516><E5025>203</E5025><E5004>250,00</E5004></C516></MOA><PRI><C509><E5125>AAA</E5125><E5118>25,00</E5118></C509></PRI></LIN><LIN><E1082>2</E1082><C212><E7140>987654321</E7140><E7143>IN</E7143></C212><PIA><E4347>1</E4347><C212><E7140>XYZ987</E7140><E7143>BP</E7143></C212></PIA><IMD><E7077>F</E7077><C273><E7008>Product Description 2</E7008></C273></IMD><QTY><C186><E6063>47</E6063><E6060>5</E6060></C186></QTY><MOA><C516><E5025>203</E5025><E5004>125,00</E5004></C516></MOA><PRI><C509><E5125>AAA</E5125><E5118>25,00</E5118></C509></PRI></LIN><ALC><E5463>C</E5463><C552><E1230>FC</E1230></C552></ALC><MOA_ALC><C516><E5025>8</E5025><E5004>20,00</E5004></C516></MOA_ALC><UNS><E0081>S</E0081></UNS><MOA><C516><E5025>39</E5025><E5004>395,00</E5004></C516></MOA><MOA><C516><E5025>77</E5025><E5004>75,00</E5004></C516></MOA><MOA><C516><E5025>9</E5025><E5004>470,00</E5004></C516></MOA><TAX><E5283>7</E5283><C241><E5153>VAT</E5153></C241><C243><E5278>19</E5278></C243><E5305>S</E5305></TAX>
        </D01B:INVOIC>
    <UNT><E0074>26</E0074><E0062>1</E0062></UNT>
    </D01B:Message>
<UNZ><E0036>1</E0036><E0020>000000001</E0020></UNZ>
</D01B:Interchange>

Thanks,

Reshma

[Reshma Kumari]

Hi Team,

Can someone help?

Thanks,

Reshma