Performance issue for "host to xml" conversions

[Benjamin]

Hello,

We are currently facing a performance issue while using Legstar within the ESB WebMethods. We convert copycobol messages directly into xml strings by doing the following steps:

1) Instantiate a HostToXmlTransformer object using the default constructor.
2) Then call the transform method with the byte payload, a StringWriter object, the encoding.

After few analyses, it appears that the transform method runs efficiently but that the transformer's instantiation is time consuming. For example, in our worst case (message related to a copybook of more than 300 lines), it takes around 500ms while the transformation takes less than 10ms.

As we are using Legstar in a basic/default way (no tuning/configuration), we are wondering if you could see a possibility to improve our performance. The version we are using is 1.5.5.

Thanks for your help!

Regards,

Benjamin.

[Fady]

Hello Benjamin,

This seemed like a sufficiently serious matter to justify a wiki page.

A lot of load testing has gone in the Java to Host transformations in the past but XML to Host hasn't been covered yet.

Let us know if you were able to implement the proposed solution and any other thoughts.

Thanks for bringing that up.

Fady

[Benjamin]

Hello Fady,

Thanks for your quick answer.

Indeed, we have already thought other solutions and we are currently focusing on the following two:

- implement a pool of transformers object : i confirm it improves drastically the performances but it has an impact on the memory, as we need a pool per type of COBOL message (and we use plenty of them).
- use HostToJava Transformers and then uses the java code to map the fields values directly into the WebMethods "document types" (which are built-in structures to handle messages in WebMethods). Very efficient as well but requires more development on our side.

For batch file processing, we could also use bundles : instantiate a transformer once and calls the transform method consecutively on every message of the bundle. If the bundle contains enough lines, it is also efficient as well.

Even if we haven't chosen definitely, there is a high chance we will go for the solution using HostToJava transformers.

Regards,

Benjamin.

[Fady]

Yes, bypassing XML altogether will get you the best performance.

This being said, if you are still interested in an XML alternative, I can propose the following:

When the LegStar COBOL binding classes were initially written, back in 2005, JAXB was not part of the JDK and there were several companies claiming to have their own implementations. For that reason, LegStar did not make any assumption regarding JAXBContext thread safety.

I think today it is safe to consider that most everyone uses the JDK's JAXB implementation. 

With that in mind, I can change the LegStar binding classes to store the JAXBContext instances in a static hashmap and reuse them. This can be done in legstar-coxbapi and therefore does not necessitate to regenerate or recompile the binding classes.

I ran a quick test and can achieve 220 TPS with this change on the same environment described previously.

Does not address the memory issue you are mentioning though, as there will be as many instances of JAXBContext as you have copybooks (assuming all copybooks are used). I can run more tests to figure out how much memory is allocated per JAXBContext.

Let me know if this can be of interest.

Fady

[Benjamin]

Fady,

Thanks for your help.

Just to be sure I am understanding well. In the case there are several threads processing messages related to the same copybook, does it mean the first new (the one creating the JAXBContext) will be expensive and the other new will reuse the JAXBContext and therefore will be faster?

Indeed, it would be interesting to know if the JAXBContext objects are huge or not in memory. However, do not spend too much time on this as we are analyzing different solutions at the moment (transformer pool, bundle, host to java).

Regards,

Benjamin.