Handling multiple 'weird' ISOmessage on a single packet

68 views
Skip to first unread message

A Haris Kurniawan

unread,
Dec 15, 2016, 3:30:52 AM12/15/16
to jPOS Users
Hi All,

So this is like a month of learning jpos, and things are starting to be clear... and then i stumble upon a case,

this is the situation

the ISO spec is a little tricky
its [HEADER][MTI][BODY]
the real spec might be like this ISO1111111110800822000000000000000000000000000001215150308000005001
so there's no total message length header it's just [HEADER][MTI][BODY]

the client can send those messages very rapidly
now because of latency,etc... it can arrive on the server socket like this: ISO1111111110800822000000000000000000000000000001215150308000005001ISO1111111110800822000000000000000000000000000001215150308000006001ISO1111111110800822000000000000000000000000000001215150308000007001

on my experience using java and c# to handle iso8583 i've been dealing with this situation, and i can handle it pretty ok manually.
reading from the socket an amount of byte (because i dont know how much it's actualy is)
convert to string, trim it, split the message into an array using the header(ISO111111111),
on the sample message above i'll be getting 3 message which i can process later on

it could also be incomplete because remember i only read an amount of byte,
it could be just like this
ISO1111111110800822000000000000000000000000000001215150308000005001ISO1111111110800822000000000000000000000000000001215150308000006001ISO1111111110800822000000000
it got incomplete on the body part
which i still could handle just by appending the last message found with the next message that i'll pick up from the socket

hell could be loose, it can also be like this
ISO1111111110800822000000000000000000000000000001215150308000005001ISO1111111110800822000000000000000000000000000001215150308000006001ISO11111
it got incomplete on the header part
but i still could handle that, by appending the detail message that did't get parsed.with the next message that i'll pick up from the socket

now there's more,
the primary bitmap was actualy binary written instead of hexa, the sample that i give above already converted, because it would be too long to put here
this one (primary bitmap written on binary) already got handle, i extend x25channel and modify the primary bitmap into hexa so jpos could read it,
vice versa when responding those data. i override streamReceive() and send()

then i try to simulate multiple iso on a single packet....
the channel that i made only receive the 1st iso message, the rest of it(the 2nd and the 3rd iso message on the packet) got lost

does this ever happened to any of you guys, and how do you work it out

Thanks in advance,
Haris



Victor Salaman

unread,
Dec 15, 2016, 4:34:10 AM12/15/16
to jpos-...@googlegroups.com
Hi:

I feel that you need to sit down and align your packager with the spec, and make sure your channel is able to handle your message. Do you know what product you are connecting to? Does it go over a NAC? You mention X.25, is it connecting to a Pad or is it over TCP/IP? Please be very detailed :)

/V

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/b96cdbd6-eb9a-4864-a415-d2eb8ee99599%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

A Haris Kurniawan

unread,
Dec 15, 2016, 4:54:28 AM12/15/16
to jpos-...@googlegroups.com
it's over tcp/ip
the extended channel can only handle single ISOmsg at a single packet.
but if i emulate more than one ISOmsg(append several isomessage string), only the 1st ISOmsg got handled, the rest of it got discarded

i test it using an application written manually (not using jpos libs), pack iso string message and send it over the socket to the server running in q2

btw i'm using x25 because i think it's most suitable with the case,
the iso message has no message-length-header, no-tail, just [TEXTHEADER][MTI][BODY]


For more options, visit https://groups.google.com/d/optout.



--
Best regards,
A. Haris Kurniawan

Victor Salaman

unread,
Dec 15, 2016, 5:12:53 AM12/15/16
to jpos-...@googlegroups.com
Hi:

In that case, you need to make sure that everything is aligned. To clarify, your channel is responsible for reading interpreting your header and extracting an ISOMsg, so your packager needs to be able to unpack your [MTI][BODY] without a problem.

Take your [MTI][BODY] and feed it to your packager... does it pack/unpack neatly without a problem? If so, you may continue. If it doesn't (you have extra bytes not consumed by the packager, etc) then anything you do afterwards will be frustrating.

After your packager is 100% up to the spec,

You will need to extend X25Channel's setHeader(String) so it just converts the string to a byte array and avoid the hex2byte conversion (or use the setHeader(byte[]) method instead. When initializing your channel make sure you set the header to "ISO111111111" so the channel knows the size of the header (very important).

With this in mind, the channel will now be able to read message, extract it , and be ready to read the next message as it appears in the stream.

/V 

Alejandro Revilla

unread,
Dec 15, 2016, 1:19:45 PM12/15/16
to jPOS Users
Actually jPOS has the ability to read messages from a stream without knowing its length. For instance PADChannel solves EXACTLY your problem. (PAD acronym comes after old X.25/X.3 packet assembler/disassembler)

Implementations that expect the "ISO" literal as a message boundary delimiter are kind of brittle and can be tricked with a card with a name "ISO Smith" or a merchant called "ISO Pizza" :)

Beware that PADChannel expects a header in hex, so instead of "ISOXXX" you'd have to use ISOUtil.hexString("ISOXXX".getBytes()), that's the only caveat.


A Haris Kurniawan

unread,
Dec 15, 2016, 11:45:14 PM12/15/16
to jpos-...@googlegroups.com
Thanks Alejandro for the insight
But i opted for a different approach 

I tried PADchannel and indeed it can read all the isomessages in the socket, but not the way i expected. Because the primary bitmap is binary-written (010001010000...) And the padchannel expect it to be hexa.

So i stay with extending x25channel but now instead of using read-until-finished approach i use read-as-much-as-needed approach.

what i did was i read a certain amount of byte from the stream according to bit length from the isospec. Its like unpacking data manually from the stream and then stop when i'm at the last bitdata, append each data into byte[] and then return it. 
So the rest of the isomessages(iso2 and iso3) stays in the socket stream and the channel will pick it up, do the same thing again untill theres no more left.

Did a lot of marshalling byte to string vice versa. Performance might be hurt 

So thats how i make it work.

I wonder if there's a more elegant solutions than that.

Cheers,
Haris






chhil

unread,
Dec 16, 2016, 12:42:06 AM12/16/16
to jpos-...@googlegroups.com
It would be a good test to compare the performance of your current approach to...

Your channel has read x bytes of data (header + bitmap+ arbitrary amount available).
You do a ISOMsg.unpack on it.
Your unpack will succeed or fail. 
If it succeeds , use the offset returned by the unpack method to know till where you are in the byte array that was passed to it so you know where the next message has started.
If your unpack fails with an exception, you will have a ISOMsg which partially unpacked, you gather the bytes from the tcp stream and try again and retry till it succeeds and go to the succeed case above.

Context switches by exception may not be the best approach, but its well worth trying to see the performance.

-chhil

Victor Salaman

unread,
Dec 16, 2016, 12:54:31 AM12/16/16
to jpos-...@googlegroups.com
Yes, it's more elegant to the let the channel and packager do the work for you.

Did you miss my last message? You're overcomplicating things. Please read/re-read my last email.

:)

Alejandro Revilla

unread,
Dec 16, 2016, 5:27:00 PM12/16/16
to jPOS Users
Please see my comments inline, below:

​​
Thanks Alejandro for the insight
But i opted for a different approach 

I tried PADchannel and indeed it can read all the isomessages in the socket, but not the way i expected. Because the primary bitmap is binary-written (010001010000...) And the padchannel expect it to be hexa.


​This is not a channel thing, it's a packager thing. Please review the packager you're using along with PADChannel, and make sure you've set the appropriate header. Please pay special attention to my comments regarding the header being in hex format instead of ASCII.

A Haris Kurniawan

unread,
Dec 17, 2016, 12:11:58 AM12/17/16
to jpos-...@googlegroups.com
Ok so i dig jpos source code to see how the packager actually works...

Cmiiw, 
So the elegant way is..
Using padchannel and set the header to hex representation of the string or extend padchannel with overriding the setheader() method, and then implement my own IsoBitmapPackager to be used on field[1] in the isopackager. 
Is that correct?

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.

chhil

unread,
Dec 17, 2016, 12:23:23 AM12/17/16
to jpos-...@googlegroups.com
You would write your packager (xml file) and define the fields appropriately (including the bitmap).
You would extend the channel as Victor has pointed out earlier.
You would configure this channel to use the appropriate packager by pointing to the packager xml file you created.

-chhil

Victor Salaman

unread,
Dec 17, 2016, 12:23:50 AM12/17/16
to jpos-...@googlegroups.com

Hi:

Check this out:


import org.jpos.core.ConfigurationException;
import org.jpos.core.SimpleConfiguration;
import org.jpos.iso.BaseChannel;
import org.jpos.iso.ISOException;
import org.jpos.iso.ISOMsg;
import org.jpos.iso.ISOPackager;
import org.jpos.iso.ISOServer;
import org.jpos.iso.packager.ISO87APackager;
import org.jpos.util.LogEvent;
import org.jpos.util.Logger;

import java.io.EOFException;
import java.io.IOException;
import java.io.InterruptedIOException;

public class VVDemo
{
    private ISOServer server;
    private Thread t;

    private static final int port = 10000;

    public static void main(String[] args) throws Exception
    {
        VVDemo vv = new VVDemo();
        vv.startServer();
        vv.process();
    }

    private void process() throws ISOException, IOException, InterruptedException
    {
        ISO87APackager packager = new ISO87APackager();

        MyChannel channel = new MyChannel("localhost", port, packager);
        channel.connect();

        String s = "ISO1111111110800822000000000000000000000000000001215150308000005001" +
                   "ISO1111111110800822000000000000000000000000000001215150308000006001" +
                   "ISO1111111110800822000000000000000000000000000001215150308000007001";

        channel.send(s.getBytes());
        ISOMsg r;
        r = channel.receive(); // Receive 1st response
        r.dump(System.out, "--> RSP: ");
        r = channel.receive(); // Receive 2nd response
        r.dump(System.out, "--> RSP: ");
        r = channel.receive(); // Receive 3rd response
        r.dump(System.out, "--> RSP: ");
        server.shutdown();
        t.join();
    }

    /**
     * Creates simple server that produces a fixed response.
     *
     * @throws IOException
     * @throws ConfigurationException
     */
    private void startServer() throws IOException, ConfigurationException
    {
        ISO87APackager packager = new ISO87APackager();
        MyChannel channel = new MyChannel(packager);

        server = new ISOServer(port, channel, null);
        server.setConfiguration(new SimpleConfiguration());
        server.addISORequestListener(
            (source, m) ->
            {
                try
                {
                    ISOMsg r = (ISOMsg) m.clone();
                    r.setResponseMTI();
                    r.set(39, "00");
                    source.send(r);
                    return true;
                }
                catch (Exception e)
                {
                    e.printStackTrace();
                    return false;
                }
            });

        t = new Thread(server);
        t.start();
    }

    /**
     * Custom channel with a 12 byte header and 3 byte trailer which streams input.
     */
    public static class MyChannel extends BaseChannel
    {
        public MyChannel(String host, int port, ISOPackager p)
        {
            super(host, port, p);
        }

        public MyChannel(ISOPackager p) throws IOException
        {
            super(p);
        }

        @Override
        protected int getHeaderLength()
        {
            return 12;
        }

        /**
         * Completely lifted from PADChannel. Just adds "trailer" support
         * @return
         * @throws IOException
         * @throws ISOException
         */
        @Override
        public ISOMsg receive() throws IOException, ISOException
        {
            byte[] header = null;
            ISOMsg m = new ISOMsg();
            m.setPackager(packager);
            m.setSource(this);
            int hLen = getHeaderLength();
            LogEvent evt = new LogEvent(this, "receive");
            try
            {
                synchronized (serverInLock)
                {
                    if (hLen > 0)
                    {
                        header = new byte[hLen];
                        serverIn.readFully(header);
                    }
                    m.unpack(serverIn);
                    getMessageTrailer(m);
                }
                m.setHeader(header);
                m.setDirection(ISOMsg.INCOMING);
                m = applyIncomingFilters(m, evt);
                m.setDirection(ISOMsg.INCOMING);
                evt.addMessage(m);
                cnt[RX]++;
                setChanged();
                notifyObservers(m);
            }
            catch (ISOException e)
            {
                evt.addMessage(e);
                throw e;
            }
            catch (EOFException e)
            {
                evt.addMessage("<peer-disconnect/>");
                throw e;
            }
            catch (InterruptedIOException e)
            {
                evt.addMessage("<io-timeout/>");
                throw e;
            }
            catch (IOException e)
            {
                if (usable)
                {
                    evt.addMessage(e);
                }
                throw e;
            }
            catch (Exception e)
            {
                evt.addMessage(e);
                throw new ISOException("unexpected exception", e);
            }
            finally
            {
                Logger.log(evt);
            }
            Logger.log(evt);
            return m;
        }

        @Override
        protected void getMessageTrailer(ISOMsg m) throws IOException
        {
            byte[] bytes = new byte[3];
            serverIn.read(bytes);
        }

        @Override
        protected void sendMessageTrailer(ISOMsg m, byte[] b) throws IOException
        {
            serverOut.write("001".getBytes());
        }
    }
}

A Haris Kurniawan

unread,
Dec 17, 2016, 3:01:17 AM12/17/16
to jpos-...@googlegroups.com
thanks chill, n thanks victor for the code,

but you guys seems to missed some things on my 1st post:
"...the primary bitmap was actualy binary written instead of hexa, the sample that i give above already converted, because it would be too long to put here..."

the bitmap is actually ASCII written "10010010111000000..." yes it's written one-zero-zero-one-blah instead of the usual hex written "822000...."
that was the catch...

my current approach is by overriding the streamReceive() of x25channel, i read the stream to a complete single isomessage, this is possible by referencing the byte length on each bitdata to the iso-spec.
so i'm like pulling out from the socket stream and then stop when the iso-spec said its the last message, even though theres still available bytes on the socket

from the bytes that i've pulled, its actually a complete isomessage (MTI+BODY), i convert the value of the bitmap from the ASCII binary to ASCII hexa so the isomessage can be handled by the packager that i've configured, that is field[1] using IFA_BITMAP

now the rest of the bytes that was still on the socket will be picked up again by the overrided x25 streamReceive() untill theres no more byte on the socket

in this approach, actually the channel and the packager that did it, i only make sure they can work on the isomessage

now obviously the channel will send response, so i override the send() method so that isomessage-response can be read by the other party by convert the ascii hexa bitmap to ascii binary

that was it.

now hints from Alejandro,
the more appropriate way is by using PADchannel, adjust the header and then use packager that can handle ASCII binary
which i can't found on default jpos
the options is IFA_BITMAP, IFB_BITMAP, IFE_BITMAP

so i think i should make my own IFA_BITMAP_BINARY





For more options, visit https://groups.google.com/d/optout.



--

Victor Salaman

unread,
Dec 17, 2016, 3:08:53 AM12/17/16
to jpos-...@googlegroups.com
Hi:

Yes, make your own bitmap and use it in your custom packager, basing yourself on iso87ascii. 

All I have to say is that whoever created your spec should be sent to the lions :)

/V

Alejandro Revilla

unread,
Dec 17, 2016, 7:52:14 AM12/17/16
to jpos-...@googlegroups.com
I'm really not sure why you would need your own channel, you can use PADChannel, and why you would need your own bitmap packager, you just need to use IFB_BITMAP, I don't think you need to write a single line of code, just use proper XML config.

A Haris Kurniawan

unread,
Dec 17, 2016, 9:36:45 AM12/17/16
to jpos-...@googlegroups.com
Yes Alejandro i could use padchannel. But IFB_BITMAP won't work. Because the bitmap is not in binary, its 10101000... written in ascii(1bit represent as 1byte), instead of the usual hexa 82200... Trust me i'm not delutioning.

So create my own IsoBitMapPackager means writing my own pack(), unpack(), 
And also my own something like asciiBinary2BitSet 

Or i could just be lazy and convert the bitmap in the byte thats being fed to hex2BitSet from ascii-binary to ascii-hexa :p

The elegant way does not look pretty T_T





On Dec 17, 2016 7:52 PM, "Alejandro Revilla" <a...@jpos.org> wrote:
I'm really not sure why you would need your own channel, you can use PADChannel, and why you would need your own bitmap packager, you just need to use IFB_BITMAP, I don't think you need to write a single line of code, just use proper XML config.

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.

Victor Salaman

unread,
Dec 17, 2016, 10:22:21 AM12/17/16
to jpos-...@googlegroups.com
Hi Alejandro 

He can't use PADChannel since has a 3 byte trailer (ASCII) as part of the message.

Sent from my iPhone

On Dec 17, 2016, at 7:51 AM, Alejandro Revilla <a...@jpos.org> wrote:

I'm really not sure why you would need your own channel, you can use PADChannel, and why you would need your own bitmap packager, you just need to use IFB_BITMAP, I don't think you need to write a single line of code, just use proper XML config.

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.

To post to this group, send email to jpos-...@googlegroups.com.

A Haris Kurniawan

unread,
Dec 17, 2016, 6:12:06 PM12/17/16
to jpos-...@googlegroups.com
Victor, theres no trailer, the last 3byte(001) is bit70 value. 

I finally tried padchannel, i just set the header into a hex representation of "ISO111..." 

Built a 'lazy' IFA_BITMAP_BINARY, it only wraps around IFA_BITMAP, override unpack() copy the byte being used and convert the bitmap part into hexa, and then work as usual. Override pack() again just converting hexa to ascii-bin.

Now its working as expected. 
But theres something that made me pulling my hair all night.  
The log got printed twice. 
If i send 1 isomessage it shows the log 2 isomessages, if i send 2 isomessages it shows in the log 4 iso messages.

I check my client-iso-tester multiple times and damn sure that i send it correctly

After digging into padchannel source apparently it was the channel behaviour

It print log twice on receive() T_T
In the finally part and just before returning ISOmsg.

May i know why Alejandro?



On Dec 17, 2016 10:22 PM, "Victor Salaman" <vsal...@gmail.com> wrote:
Hi Alejandro 

He can't use PADChannel since has a 3 byte trailer (ASCII) as part of the message.

Sent from my iPhone

On Dec 17, 2016, at 7:51 AM, Alejandro Revilla <a...@jpos.org> wrote:

I'm really not sure why you would need your own channel, you can use PADChannel, and why you would need your own bitmap packager, you just need to use IFB_BITMAP, I don't think you need to write a single line of code, just use proper XML config.

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.

To post to this group, send email to jpos-...@googlegroups.com.

Victor Salaman

unread,
Dec 17, 2016, 7:53:03 PM12/17/16
to jpos-...@googlegroups.com
Your bitmap doesn't contain a bit 70.

/V

A Haris Kurniawan

unread,
Dec 17, 2016, 8:11:32 PM12/17/16
to jpos-...@googlegroups.com
whoa apparently i just pasted the wrong hexa in my 1st post, so sorry :D

the isomessage in the stream is like this:
ISO1111111110800100000100010000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000000000000000000000001215150308000005001

so i should've pasted:
ISO1111111110800822000000000000004000000000000001215150308000005001

sorry for the typo

i still don't get it why padchannel need to print the log twice, is it for debugging or auditing purpose?

Cheers,
Haris


For more options, visit https://groups.google.com/d/optout.



--

Alejandro Revilla

unread,
Dec 17, 2016, 11:35:12 PM12/17/16
to jpos-...@googlegroups.com
So IFB_BITMAP should work. I'll check the double-logging, probably a bug in PADChannek. 

@apr
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.

To post to this group, send email to jpos-...@googlegroups.com.

Alejandro Revilla

unread,
Dec 18, 2016, 7:13:46 PM12/18/16
to jPOS Users
FYI, I've checked PADChannel and I don't see a reason for double logging.

Can you show us your logging configuration?



A Haris Kurniawan

unread,
Dec 18, 2016, 7:56:30 PM12/18/16
to jpos-...@googlegroups.com
the logging is default jpos xml,

i debug it and it print the log twice, i don't think its a bug
but the logger.log() is being called two times,
it's line 126 and 128 :)



For more options, visit https://groups.google.com/d/optout.

Alejandro Revilla

unread,
Dec 18, 2016, 8:28:33 PM12/18/16
to jPOS Users
Oh... you're right!

Will remove the last one right away.



Alejandro Revilla

unread,
Dec 18, 2016, 8:30:51 PM12/18/16
to jPOS Users

A Haris Kurniawan

unread,
Dec 18, 2016, 10:01:25 PM12/18/16
to jpos-...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages