String interpolation and macro support for jPOS config files.

[Victor Salaman]

Hi:

I've recently made some enhancements which might be of interest to the jPOS community and would like your feedback. I've added the concept of config file "decoration" to Q2.

What this means is that your config files can contain markup which would be resolved at runtime. The implementation i've made is based on Freemarker (www.freemarker.org)

For example, in your "deploy" directory you'd have a file called "deploy-properties.ftl". This file is "auto imported" each time a config file is read from Q2's deploy directory.

Let's say it contains something like:

[#-- ********* Transaction Manager properties ********* --]
[#assign tmSessions = '1']
[#assign tmDebug = 'true']
[#assign tmDebugParticipant = 'true']


Now in my file '80_txnmgr.xml' I'd have something like:

    <property name="space" value="transient:default"/>
    <property name="queue" value="txnmgr"/>
    <property name="persistent-space" value="jdbm:db/TxnMgrSpace"/>
    <property name="sessions" value="${tmSessions}"/>

    <property name="debug" value="${tmDebug}"/>

    [#if tmDebugParticipant == 'true']
    <participant class="org.jpos.transaction.Debug" logger="Q2" realm="debug"/>
    [/#if]

Although right now you could accomplish string resolution using jPOS' build.xml file, doing a conditional inclusion of a participant as above would be difficult. This also becomes useful in channel adaptor host and port definitions, url's, etc... Anything that can usually change from different environments you're working with. For example, my config files usually have different parameters for development, test and production, but the files are the same.

Another advantage of the enhancements is being able to create macros, a feature specially useful in transaction manager config files. For example:
 
[#macro endpointQuery name endpoint itc mti pcode]

    <group name="${name}">
        <participant class="xxxx.common.SetITC">
            <property name="itc" value="${itc}"/>
        </participant>
        <participant class="xxxx.common.FindDuplicate">
            <property name="itc" value="${itc}"/>
            <property name="checkpoint" value="find-duplicate"/>
        </participant>
        <participant class="org.jpos.transaction.participant.HasEntry">
            <property name="name" value="DUPLICATE_ENTRY"/>
            <property name="yes" value="DuplicateResponse"/>
            <property name="no" value="${name}_Response"/>
        </participant>
    </group>

    <group name="${name}_Response">
        <participant class="xxxx.yyyy.Create${endpoint}Request">
            <property name="mti" value="${mti}"/>
            <property name="pcode" value="${pcode}"/>
            <property name="space" value="jdbm:db/${endpoint}-stan"/>
            <property name="template" value="cfg/${endpoint}-template.xml"/>
        </participant>
        <participant class="xxxx.common.ExecuteGroups">
            <property name="groups" value="Query${endpoint} CreateCCResponse LogAndReply"/>
        </participant>
    </group>
 
[/#macro]

Now you can invoke the macro in your transaction manager config file:

[@endpointQuery name='AthSale' endpoint='Ath' itc='05300' mti='0200' pcode='000000'/]
[@endpointQuery name='AthOnlineRefund' endpoint='Ath' itc='03000' mti='0200' pcode='200000'/]
[@endpointQuery name='AmexSale' endpoint='Amex' itc='05300' mti='1100' pcode='004000'/]
[@endpointQuery name='AmexOnlineRefund' endpoint='Amex' itc='03000' mti='1420' pcode='009003'/]

In the previous example, I have taken some ideas from Andy Orrock's blog as avid readers will notice!

In summary I'd like to know wether someone else besides me would find this useful so I can commit the changes to Q2 (support for config decoration) and create an optional module with the Freemarker implementation of the config decoration SPI, so that everyone can use it.

/V

[Alejandro Revilla]

I think this is a great idea. My early comments without looking at the code:

• Can we make this optional (so we can give it a spin without forcing every jPOS system to require this new library)? It's OK to make some changes to Q2/QFactory in order to expose some methods that you could override.
• Would it be feasible to make the template engine pluggable so we could use velocity or stringtemplate instead?
  
• Would this operate on protected QBeans (I think yes if you process them in memory)
  

[Victor Salaman]

Hi Alejandro:

Yes, it's completely optional and requires no new jar dependencies. The changes are very simple. We basically just introduce a new interface:

package org.jpos.q2;

import java.io.File;

public interface ConfigDecorationProvider
{
    public void initialize(File deployDir) throws Exception;
    public void uninitialize();
    public String decorateFile(File f) throws Exception;
}

The modifications in Q2 are trivial, they just:
     *  add a method to read a config from the classpath (META-INF/org/jpos/config/Q2.properties) and activate whatever config decorator you have there. 
     *  make several changes to run(): to invoke decoration initialization and uninitialization events following Q2s lifecycle.
     *  make several changes in deploy(File): to invoke the decorator everytime a deploy occurs.

The implementation can be anything desired and can therefore be put in an optional jPOS module where no additional depency is required except if so watned by the jPOS user.

I have many years of experience with Velocity and velocimacros and chose Freemarker because:
     * It's only 1 jar, has no extra dependencies
     * Syntax looks and feels good to a velocity user and has no side effects.
     * More powerful IMHO.

StringTemplate looks good to me but does not support macro's, but the possibility of using any templating engine is there.

In summary, we'd be making these changes to Q2.java, and we'd be introducing a new optional module "freemarker-decorator".
 
Let me know if I should commit :)

/V

[Alejandro Revilla]

Got it. Looks nice and clean to me, specially the fact that everything is optional.

We don't have an 'opt' directory in jPOS, should we add the freemarker-decorator to jposee/opt or we should go ahead and add an 'opt' directory? (your call). We could temporarily add it to jposee/opt and then move it to jpos6/modules after some initial tests.

Ok about freemarker, I started with it and will be glad to give it another try. I heard good things about StringTemplate, but I don't have much experience with it (other than my PoC on the STLogListener). One jar is good and Freemarker is proven and stable, and probably better than Velocity.

Feel free to commit it at your convenience. If you can, you may want to update http://jpos.org/wiki/ChangeLog (if you don't have an account there, ping me and I'll create one for you).

[Victor Salaman]

I'll add it to jposee/opt for now.

/V

[Victor Salaman]

Committed to jposee/opt ....

Note to self: Never name a jPOS module with a dash as part of the name.... It will make you waste 2 hours of precious time before you figure out the build is silently ignoring files because it doesn't like your module name :)

[Alejandro Revilla]

Great! we'll give it a shot.

Regarding your note to self, I got caught with that dash problem all the time ... we need a regexp expert to fix that one (I'm really bad with regexps).

Questions:

• Why did you have to add this check for file.name != LOGGER_CONFIG in line 440 ?
• If you change the decorator config template and touch the QBean descriptor, would it take the new values?

[Mark Salter]

Alejandro Revilla wrote:
> Regarding your note to self, I got caught with that dash problem all the
> time ... we need a regexp expert to fix that one (I'm really bad with
> regexps).

I tried with a module folder of mark-test and also mark_test each only
contained a deploy folder with a single (empty) file called test.xml and
test2.xml respectively.

Both files were deployed.

So, do I have the wrong 'dash' or a newer version of ant (Apache Ant
version 1.7.1 compiled on June 27 2008) or is it an o/s difference?

--
Mark

[Alejandro Revilla]

Thank you for the test; Victor, what is your ant version?

Interesting enough, mine is version 1.7.0 compiled on May 21 2009 (older version newer compile).

I think I'll have to fetch a new version and get rid of the OSX provided one.

[Victor Salaman]

Apache Ant version 1.7.0 compiled on May 21 2009 on MacOS X 10.5.7

[Mark Salter]

Victor Salaman wrote:
> Apache Ant version 1.7.0 compiled on May 21 2009 on MacOS X 10.5.7

Matches Alejandro's, but we don't know if ant is not helping here...

Which character is causing you both brief - can I get an example module
name please?

--
Mark

[Victor Salaman]

Hi Alejandro:

Why did you have to add this check for file.name != LOGGER_CONFIG in line 440 ? I din't want the logger file decorated, initially when coding this I noticed that if your decorator code is wrong and you have no way of seeing what is wrong, you're in trouble.

If you change the decorator config template and touch the QBean descriptor, would it take the new values? Yes indeed.

/V

[Victor Salaman]

Mark:

To try the problem out, you could get the same ant version and do a "svn co" of jposee on a different directory, and rename decorator_freemarker to decorator-freemarker ... then do "ant clean jar" and you'll see that the "src" files are nowhere to be found...

If you do a "ant -v" instead, you'll get some messages saying "skipping XXX - don't know how to handle it."

Maybe it's the ant version we have....

/V

[Mark Salter]

Victor Salaman wrote:
> Mark:
>
> To try the problem out, you could get the same ant version and do a "svn co"
> of jposee on a different directory, and rename decorator_freemarker to
> decorator-freemarker ... then do "ant clean jar" and you'll see that the
> "src" files are nowhere to be found...

Ok, I had the right 'dash' (-), but only check deploy - not src...

>
> If you do a "ant -v" instead, you'll get some messages saying "skipping XXX
> - don't know how to handle it."

Ok, I'll take a look.

>
> Maybe it's the ant version we have....

Perhaps not, let me check the same thing happens here.

--
Mark

[Mark Salter]

Victor Salaman wrote:
> To try the problem out, you could get the same ant version and do a "svn co"
> of jposee on a different directory, and rename decorator_freemarker to
> decorator-freemarker ... then do "ant clean jar" and you'll see that the
> "src" files are nowhere to be found...
>
> If you do a "ant -v" instead, you'll get some messages saying "skipping XXX
> - don't know how to handle it."
>
> Maybe it's the ant version we have....

Nope, same here - now comparing apples to apples 8).

I will take a look.

--
Mark

[Alejandro Revilla]

Mark, don't spend more than 30 seconds on this, we can live with the work around of not using dashes in the modules names.

[Alejandro Revilla]

Why did you have to add this check for file.name != LOGGER_CONFIG in line 440 ? I din't want the logger file decorated, initially when coding this I noticed that if your decorator code is wrong and you have no way of seeing what is wrong, you're in trouble.

Got it. Makes sense.
 

If you change the decorator config template and touch the QBean descriptor, would it take the new values? Yes indeed.

Perfect.
 

[Mark Salter]

Alejandro Revilla wrote:
> Mark, don't spend more than 30 seconds on this, we can live with the work
> around of not using dashes in the modules names.

No worries, I am happy to waste some today.

This is ant 'mapper' related, I am interested to find out why we :-

<copy todir="${build.dir}" filtering="true">
<mapper type="regexp" from="^\w*[/|\\](.*)" to="\1" />
<fileset dir="${build.modules}"
excludesfile="${modules.ignore.list}">
<include name="**/*.java"/>
<include name="**/*.xml"/>
<include name="**/*.properties"/>
<include name="**/*.cfg"/>
<include name="**/cfg/**"/>
<exclude name="**/cfg/_*" />
</fileset>
</copy>

I added a '-' to get :-

<mapper type="regexp" from="^\w*[/-|\\](.*)" to="\1" />

and the dash-module src is covered by the copy.

I now want to understand why and what we are doing with this (and other)
mappers.

I see why you mentioned regex now 8).

--
Mark

[Alejandro Revilla]

I now want to understand why and what we are doing with this (and other)
mappers.

We are basically trying to flatten the directory tree up one level and merging it into the build directory.

 

I see why you mentioned regex now 8).

:)

If that added dash doesn't break other things, we can add it to build.xml

[Mark Salter]

Alejandro Revilla wrote:
>
> If that added dash doesn't break other things, we can add it to build.xml

Indeed, I am just checking that it doesn't 8).

--
Mark

[Mark Salter]

It is not that simple...

... please keep avoiding '-''s for now.

8)

PS, I am away for a week - without internet access so I will have to
'talk' later.

--
Mark

[Alejandro Revilla]

If I recall correctly, I came to your very same conclusion a while ago, no "-" it is.

Enjoy your week offline, when back, please tell us how it feels the real life :) :)