Proposal: Support Pipelines

[Claude Mamo]

As a Smooks application developer, I want a general way to define multiple pipelines (i.e., segregated visitor chains) in a given execution and direct their output to different streams, including the execution output stream. For example, we could have pipeline A processing fragment X and streams the output to the filesystem. We could then have pipeline B that also processes fragment X but streams the output to a network socket. Both pipelines A and B are working on separate copies of fragment X. Transformations to fragment X will only be visible to the pipeline which made the transformations. In other words, pipeline B transformations are visible to pipeline B visitors but invisible to pipeline A whereas pipeline A transformations are visible to pipeline A visitors but  invisible to pipeline B. Furthermore, visitors outside the pipelines won't see either of the pipeline transformations.

The above behaviour is already supported to a certain extent in the Template cartridge. The FreeMarker visitor can either:

1. Direct its output to a stream distinct from the execution output stream
2. Bind the output to a Java bean, or
   
3. Write the output to the execution output stream. 

I'm proposing that we generalise and extend this behaviour to all the visitors. This can be done without needing to create new primitives. Smooks itself is a pipeline so all we require is the ability run instances of Smooks within a Smooks execution in order to mimic pipelines. The idea of nesting Smooks is not a new one because there is an existing implementation of it in Smooks 1.7.1. However, I suspect it's still a prototype since it isn't at all documented and has several constraints like being limited to run only a single instance of NestedExecutionVisitor in a Smooks execution.

From an XML API perspective, the nested Smooks visitor configuration should be composed of two parts: (a) the action applied on the output (e.g., directing the output to a stream different than the execution output stream ), and (b) the visitor chain which is essentially the smooks-resource-list of the nested Smooks:

<smooks-resource-list xmlns="https://www.smooks.org/xsd/smooks-2.0.xsd"
                      xmlns:core="https://www.smooks.org/xsd/smooks/smooks-core-1.6.xsd">

    ...
    ...
    <core:smooks filterSourceOn="a">
        <core:action>
            <core:outputTo outputStreamResource="MyStream"/>
        </core:action>
        <core:visit>
            <smooks-resource-list>
                <resource-config selector="a">
                    <resource>org.smooks.visitors.smooks.BarVisitor</resource>
                </resource-config>
                <resource-config selector="c">
                    <resource>org.smooks.visitors.smooks.QuxVisitor</resource>
                </resource-config>
                <resource-config selector="d">
                    <resource>org.smooks.visitors.smooks.QuuzVisitor</resource>
                </resource-config>
            </smooks-resource-list>
        </core:visit>
    </core:smooks>
    ...
    ...
</smooks-resource-list>

I welcome any thoughts on ways to improve this.

Claude

[Tom Fennelly]

Hi Claude.

Sorry for the late reply. I see this went out during the xmas break.

Hmmm I'm not so sure. Of course this kind of thing had crossed my mind before as something one "could do", but I always avoided it. I'd see this as a bit of a change in philosophy for Smooks, heading more in the direction of Apache Camel or the like. I'd think smooks configs are potentially complicated enough as they are without nesting them inside each other. In all the years I was involved with Smooks I can't recall anyone asking for this capability and I'd think only do this if there's a lot of demand for it. For this kind of use case I'd use the likes of Apache Camel since these guys have already solved that kind of problem nicely already. There used to be a Smooks+Camel example.

T.

[Claude Mamo]

Hi Tom,

I hope you enjoyed the holidays.

For sure, I'm not suggesting that Smooks replaces Apache Camel when it comes to transport integration or even data mapping though there might be some overlap with this feature. I might have not given the best example in the proposal. Hopefully, the blog post draft I emailed you provides a better use case. I can't say if there's a "lot" of demand for pipelines but it's been requested such as from a comment in https://www.infoq.com/news/2008/11/smooks-events/?itm_source=presentations_about_Smooks&itm_medium=link&itm_campaign=Smooks:

Smooks would be much more acceptable for me if

...

...

2. some sort of more controlled way over whole transformation. Not only visitor pattern where are you limited only to current event and some custom made context. In stax you can chain easily chain readers and writers and make nice pipeline on one stream of events ...

Broadly speaking, the notion of a pipeline was already implemented in a very limited context within the Template cartridge from before Smooks 2:

     <ftl:freemarker applyOnElement="order-item">  
        <ftl:template>target/classes/orderitem-split.ftl</ftl:template>  
        <ftl:use>  
            <!-- Output the templating result to the "orderItemSplitStream" file output stream... -->  
            <ftl:outputTo outputStreamResource="orderItemSplitStream"/>  
        </ftl:use>  
    </ftl:freemarker>  

What is being proposed is to widen the context. I completely agree that multiple levels of nesting will make the config harder to agree so we should find a solution for that (perhaps importing the nested config?) but I don't think we should drop the proposal.

Claude

--
You received this message because you are subscribed to the Google Groups "Smooks Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smooks-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/smooks-dev/2863c8ea-20e6-4983-9806-ef9dec3f8d54n%40googlegroups.com.

[Tom Fennelly]

On Mon, 15 Feb 2021 at 09:53, Claude Mamo <claud...@gmail.com> wrote:

but I don't think we should drop the proposal.

Okidoki 

[Claude Mamo]

Minor typo fix:

I completely agree that multiple levels of nesting will make the config harder to understand so we should find a solution for that (perhaps importing the nested config?) ...

Claude

--

You received this message because you are subscribed to the Google Groups "Smooks Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to smooks-dev+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/smooks-dev/CA%2BbPaoKY9-fveMJBPGVrn3fNaWSdb-7T_qUP9sqGQiYgfdR1TA%40mail.gmail.com.