WTA - jpos mux load-balance proper configuration

[A Haris Kurniawan]

Hi all,


Ok in my understanding after i read about mux and muxpool, i conclude that jpos has a feature that can be used somewhat similar to load-balancer/HA using the muxpool.

so i set an environment like this :
Q2(
ISORequestListener -> space.out() -> TxManager -> GroupSelector -> TransactionParticipant -> mux.request() -> MuxPool -> (mux-1(key) ^ mux-2(key)) -> ChannelAdapter(mux-n) -> client-n
)

now the question is,
1. is this the right way to do it?

2. the key for the ISOmsg is held by individual mux-n... now is it possible if ISOmsg with key(7,11) is dispatched to mux-1 and then couple of hours later ISOmsg also with the same value of key(7,11) is dispatched to mux-2... is mux-2 will raise an error or is it going to pass just like any normal ISOmsg.
do it need to place a logic for checking unique ISOmsg somewhere, maybe in the TransactionParticipant using as simple Set<7,11>.

3. how do i get response message from client-n, is it by ISOmsg response=mux.request() ?

Thanks,
Haris

[chhil]

Q2(ISORequestListener -> space.out() -> TxManager -> GroupSelector -> TransactionParticipant -> mux.request() -> MuxPool -> (mux-1(key) ^ mux-2(key)) -> ChannelAdapter(mux-n) -> client-n)

​

MuxPool and Qmux both implement the MUX interface, so your transaction participant needs to be aware of the muxpool name and pull it from the nameregistrar and call the muxpool.request(...) and the muxpool load balances the request across the muxes in its configuration and call the individual MUX's.request.

The QMUX configuration has a strategy property , if you define it as "round-robin" else it will use the PRIMARY_SECONDARY strategy. It uses the is  mux connected value and determines which next one to choose from the pool based on the strategy.

now is it possible if ISOmsg with key(7,11) is dispatched to mux-1 and then couple of hours later ISOmsg also with the same value of key(7,11) is dispatched to mux-2… is mux-2 will raise an error or is it going to pass just like any normal ISOmsg.

​

No, its not possible if you are using a timeout value in your request and your timeout is less than the time the key is repeated. Its more to do with not repeating transaction requests in flight through the same mux. The key uses your fields but it also uses the MTI and the out queue name of your mux (see here https://github.com/jpos/jPOS/blob/master/jpos/src/main/java/org/jpos/q2/iso/QMUX.java#L179-L208). So if the new request has the same key while the previous request is in flight for the same mux it will throw a duplicate key exception. Now this behavior will vary in a muxpool scenario (someone correct me if I am wrong). If the reuse-space property of the QMUX is configured to true (default is false), this implies the keys are in the same global space used by all muxes and a duplicate across any mux will get flagged for transaction s in flight. If the reuse-space property is false, each mux has a local space instance and only duplicates will get detected if they hit the same mux. The entry is removed on timeout or a response is received (basically the synchronous request call completes).

how do i get response message from client-n, is it by ISOmsg response=mux.request() ?

​

the mux.request call returns a response or null (when it times out)

    /**
     * @param m
     *            message to send
     * @param timeout
     *            amount of time in millis to wait for a response
     * @return response or null
     */
    public ISOMsg request(ISOMsg m, long timeout) throws ISOException {

​

If you haven't done so already, do go through the excellent programmers guide at  http://jpos.org/doc/proguide-draft.pdf

-chhil

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/10824430-8d68-49f2-b45a-0ebd2d59a1f5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andy Orrock]

Here's a further internal write-up I did on muxes:

http://screencast.com/t/yL5ZXpBCLqh 

I assume where you say:

"...after i read about mux and muxpool" that you're referring to my online pieces.

For the record, here's a "split-by-divisor" implementation:

<?xml version="1.0" ?>

<mux class="org.jpos.q2.iso.MUXPool" logger="Q2" name="chase-mux">

 <muxes>chase-mux-0 chase-mux-1</muxes>

 <strategy>split-by-divisor</strategy>

 <split-field>42</split-field>

</mux>

That's a split by MID (Merchant External ID).  That jPOS contribution is by Martin Muguiro.  He also contributed "round-robin-with-override" -- to be used if you need to force completions and reversals to go out the same route as their corresponding originals.  Here's what that looks like:

 <mux class="org.jpos.q2.iso.MUXPool" logger="Q2" name="chase-mux">

  <muxes>chase-mux-0 chase-mux-1</muxes>

 <strategy>round-robin-with-override</strategy>

 <follower-override>1400 1420</follower-override>

 <original-channel-field>1000</original-channel-field>

 </mux>

Where:

follower-override = The MTIs of the transactions which can't be assigned round-robin

original-channel-field = You pass the mux pool the original channel in field 1000 and it uses that to force the route (if the MTI in flight is a follower)  

--------

Andy Orrock

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAPazefDcNGswp6jjYzFX6gv9bqqKAALtRbLXf%3DGa7pO_LPHvEA%40mail.gmail.com.

[A Haris Kurniawan]

thanks chill for the clear explanation,

i've read the proguide but i'm not smart enough to understand many of things written there :D,

thanks andy for the explanation on the stategy variations on muxpool.

so i did some googling about reuse-space on qmux,
http://jpos.org/blog/2014/04/qmux-internal-space/

apparently Alejandro made it default to false because on high transaction if using the same space for the Qmux keystorage it can effect the overall performance as Alejandro said "..to improve performance, intend to discourage such use in the future.."

so setting reuse-space to true is a bad practice...

but what if i need to make sure that the entire isomessage spread accross the qmux on the pool is unique

and then i think of this:

on the isorequestlistener, setConfiguration part, i do this:
Set<String> isoMsgKeyHolder = Collections.synchronizedSet(new HashSet<String>());
NameRegistrar.register("ISOMSG_KEYHOLDER", isoMsgKeyHolder);

and on the transactionparticipant i do this:

String msgId = isomsg.getString(7) + isomsg.getString(11);
Set keyHolder = (HashSet<String>) NameRegistrar.get("ISOMSG_KEYHOLDER");
boolean keyIsUnique;
synchronized (keyHolder) {
   keyIsUnique = keyHolder.add(msgId);
   }
if (keyIsUnique) {
   QMUX mux = (QMUX) NameRegistrar.get("mux." + muxPool);
   ISOMsg response = mux.request(isomsg, timeOut);
   isosrc.send(response);
} else {
   isomsg.set(39, ErrorRc.ERR_DUPLICATE_ISODATA);
   isosrc.send(isomsg);
}

is that ok performance wise?

or disaster can happen?

Thanks,

Haris

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAPZRy5GX4XL7R6xT8Jk7bG7eWitkvL8S-3xKdR9UT7KE8xCbGg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Best regards,
A. Haris Kurniawan

[chhil]

You are welcome.

I will defer the reuse-space performance question to Alejandro or other souls on this list.
To the best of my knowledge, the performance was related to threads of all queues in the space waking up on notification even when a single queue name in the default global space was modified.

My feeling with your current code may add a similar overhead using a synchronized collection. 

You may want to consider persistence of the key if its important enough to decline across restarts and of course removal of a key from that ever growing data structure you add to :). Spaces have this nice timeout property which expire entries.

There is a persistent je space that you can use, that persists the data to a file and is aware of the keys across restarts or simply have a duplicate check in a participant that queries your database.

-chhil

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAL8bJ0G0OFOtA1jmE0m5V-7fkjGuJLw79rBnvAhv1JNfqLGYiQ%40mail.gmail.com.

[A Haris Kurniawan]

"that ever growing data structure you add to"... hohoho i was just about to think of this impact,

workaround is somekind of watchdog using dailytaskadapter(which a cannot find any example of how to use it :p)

that will reset the keyholder list every midnight

but this will add more overhead and i'm like reinventing the not so pretty egg-shaped-wheel :D

the constraint is only to make sure that the IsoMsg is unique for a single day

anyway on that dailytaskadapter,

further reading the source code....
    public void run () {
        while (running()) {
            waitUntilStartTime();
            if (running()) {
                Thread taskThread = new Thread(task);
                taskThread.setDaemon (true);
                taskThread.start();
                ISOUtil.sleep (1000);
            }
        }
    }
when it tick (waitUntilStartTime is finished) it create new thread of the task. is this correct?

wouldnt that mean create a new thread every single day?

and how do you use this class? is there any xml config?

i'm learning to maximize every tools provided in jpos framework, so please dont suggest to use quartz :D

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAPazefD3zuE2OwUz5NoHswideZsY5KjOkJgVzMWLaV7PRLF-AQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[chhil]

I still feel you might be better off doing a dup check in the db. Consider you have multiple servers for HA and load balancing, and each connecting to clients. Your strategy will only work on one box. So if req 1 comes on box 1 and a dup of request comes on box 2 you will not be able to decline it.

Anyways...

You could do something like

package org.jpos.q2.iso;

import java.util.Calendar;

public class TestDailyTask extends DailyTaskAdaptor {

    @Override
    public void run() {
        while (running()) {
            waitUntilStartTime();
            if (running()) {

                System.out.println(
​"Will get invoked when configured start time is breached"
);

            }
        }
    }

}

​

Deploy file 

<myDailyTask logger="Q2" class="org.jpos.q2.iso.TestDailyTask">

    <property name="start" value="13:26:00" />

</myDailyTask>

​

There is  is a QuartzAdaptor in jposee. Look at the test case if interested 

https://github.com/jpos/jPOS-EE/blob/master/modules/quartz/src/main/resources/META-INF/q2/installs/deploy/90_cron.xml

-chhil

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAL8bJ0HPp%2BEsLcC2LcExU8M9azRBc7EHY%3DRZmO3whYotYSgZ1A%40mail.gmail.com.

[A Haris Kurniawan]

ok that code was simple, i was so desperate to made it work that i made another class that extend Qbeansupport and an interface MBean :p

Thanks

hmm... using db for dup check would guarantee a consistent keyIsoMsg, i'm just not sure about the overhead cost.

what would be awesome though is using mux(oneshotchanneladapter) to send current keyIsoMsg update between Servers.

imagine 3server with 3mux-nodes each,

2 on DC, 1 on DRC

server-1 actively receive transaction, dispatch the transaction to muxpool which contains 6mux, and on the same time send the current KeyIsoMsg to other 2 Servers.

if server-1 is down, the network people only need to route the packets to server-2, then server-2 dipatch the transaction to any mux available on DC.

if server-2 is also down, the network people route the packet to server-3, server-3 dipatch the transaction to any mux available on DRC.

and if the DRC is down everyone should just go home :D

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAPazefBkg7Zq%2BepBMGYutjit_qy_WnEZnq2rOBkzhLTKAyzL9w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[chhil]

jpos is opensource, feel free to contribute :)

My personal opinion, there is no need to tie duplicate detection at the mux level. 
You should do your on performance tests for the approaches and come to a conclusion.

There is a replicated space for distributed environments in the jposee project that uses jgroups. Personally never used it, but you may find it useful. There may be some reference in the group emails.
http://jpos.org/jposee/jPOS-EE.pdf (See chapter 6)

-chhil

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAL8bJ0EnDr8dNujyLYZvOnUE%2BPxTv5%3DzMyEufk0pe1EV2tN6-A%40mail.gmail.com.

[Andrés Alcarraz]

Hi Haris. I Think we could be more helpful if you tell us the reasons why you need that overcontrol on duplicate message keys.

One problem I see with your workaround:

String msgId = isomsg.getString(7) + isomsg.getString(11);

is that your reverses would have been marked as duplicated and you need to processes them as well. Also, if you are trying to check for duplicates at this level, and given that at midnight you plan to forget keys, I guess you endpoints synchronize at that point.

If this is the case you need to make sure, not only that reverses are being processed, but also you need to send the reverses to the original mux you sent the request, because the other end would have not knowledge of that request and would reject that reverse since it would not be able to handle it. This is where the round/robin with override mentioned by Andy takes place. You need to store somewhere, the mux to which message was sent so that in the process of the reverse you can override the destination mux with that.

That all said, I see all this setup overcomplicated, are you sure you need to deal with duplicates at this level, wouldn't it be responsability for the next level of authentication to have their data syncronized and check duplicates in the db or whatever storage meda they are usind?

For all of this, as I feel I'm answering blindly, I would like to have more information on the context of why you need those checks. May be you just missunderstood the real cuase you need to approach.

Best regards.

Andrés Alcarraz

El 30/12/16 a las 05:40, A Haris Kurniawan escribió:

-- -- jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org --- You received this message because you are subscribed to the Google Groups "jPOS Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com. To post to this group, send email to jpos-...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAL8bJ0EnDr8dNujyLYZvOnUE%2BPxTv5%3DzMyEufk0pe1EV2tN6-A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.

[A Haris Kurniawan]

Hi Andrés

The overcontrol on message keys was based on my experience on one of transaction system that i've handled. Where there are multiple parties connected with a single host and its all communicated using iso8583. At first it dont really seems to be/ would be an issue. But then when the data was being reconciled apparently theres a lot of duplicate data being sent by those parties. And for each of those duplicate data a mac was being issued that was later on used the business process. 1 unique data can have up to 10 mac.

And the biggest problem was all those duplicate data was being served to the managenent in the dashboard report and they wont understand that those data was duplicated which mean theres only one data that should've been considered valid. They insist if theres 10 mac then theres 10data

So we decided to filtered out those duplicate data on transaction level.

No problem after that.

Anyway this interest me when you talk about reversals, 1st of all we dont send reversal messages but those parties send reversal message in case theres an error on the data that they send. It also implemented slightly different.

The reversal has no constraint about bit7 and bit11 has to be the same(0200 and 0400) during transaction. But instead bit7 and bit11 was part of original data message(also consist of several other bit) to be reversed that put on bit90.

Just in case you're wondering, its not really an Electronic financial transaction system but more of a electronic data transaction system that has several financial infos in it. Thats why iso8583 was choosed

-- -- jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org --- You received this message because you are subscribed to the Google Groups "jPOS Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com. To post to this group, send email to jpos-...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAL8bJ0EnDr8dNujyLYZvOnUE%2BPxTv5%3DzMyEufk0pe1EV2tN6-A%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/cb7bb064-90a8-34b3-286d-c1d71cda8508%40gmail.com.