ISOMUX (or variants) with FSDISOMsg

580 views
Skip to first unread message

antiguy

unread,
Mar 2, 2011, 1:53:29 AM3/2/11
to jPOS Users
Hello,
I was wondering if it would be possible to use ISOMUX (or QMUX, etc)
with FSDISOMsg/FSDMsg. If I recall correctly, the mux may be
configured to match response to sending thread using some fields.
Would that be possible using FSDISOMsg, since it is not a regular
ISOMsg?

Thanks in advance.

Mark Salter

unread,
Mar 2, 2011, 2:57:21 AM3/2/11
to jpos-...@googlegroups.com
The MUX's should work just find with FSDISOMsg as that class provides an
Integer route into the fields it contains - which the MUXs need.

It won't work with FSDMsg as the access to it's field values is via
String names/ids.

It should be straight forward to enhance the MUX to use a string
(perhaps if 'non-ints' are specified in the key of it's config.
But there will probably be other changes too, as FSDMsg is not of ISO*
flavour.

So if you use FSDISOMsg all should be well, let us know how you get on.

--
Mark

antiguy

unread,
Mar 2, 2011, 4:57:19 AM3/2/11
to jPOS Users
Thanks Mark. Already wrapping the FSDMsg instances into FSDISOMsg for
sending on an ISOChannel.

Mark Salter

unread,
Mar 2, 2011, 6:46:56 AM3/2/11
to jpos-...@googlegroups.com
On 02/03/2011 09:57, antiguy wrote:
> Thanks Mark. Already wrapping the FSDMsg instances into FSDISOMsg for
> sending on an ISOChannel.

So you should all is working well I hope?

--
Mark

'Dipo Odumosu

unread,
Mar 3, 2011, 6:36:59 AM3/3/11
to jpos-...@googlegroups.com
Hello everyone,
Just found something that will make my using any ISOMUX variant impossible at my current level of understanding. Here's the code for my test ThalesAdaptor class (newly added lines are commented  - // NEW:


public class ThalesAdaptor {
    private FSDChannel channel;

    public ThalesAdaptor(String host, int port) {
        channel = new FSDChannel();
        channel.setHost(host, port);
        channel.setPackager(new FSDPackager());
    }

    public void connect() throws Exception {
        channel.connect();
    }

    private FSDMsg createRequest(String command) {
        FSDMsg req = new FSDMsg("file:cfg/hsm-");

        if (command != null)
            req.set("command", command);

        return req;
    }

    private FSDMsg createResponse(String response) {
        FSDMsg resp = new FSDMsg("file:cfg/hsm-resp-");

        if (response != null)
            resp.set("response", response);

        return resp;
    }

    public FSDMsg diagnostics() {
        return createRequest("NC");
    }

    public FSDMsg generateDoubleLengthKey() {
        FSDMsg req = createRequest("A0");
        req.set("mode", "0");
        req.set("key-type", "001");
        req.set("key-scheme-lmk", "U");

        return req;
    }

    public FSDMsg command(FSDMsg request) throws Exception {
        StringBuffer sbuffer = new StringBuffer(request.get("command"));
        sbuffer.setCharAt(1, (char) (sbuffer.charAt(1) + 1));
        FSDMsg resp = createResponse(sbuffer.toString());

        FSDISOMsg msg = new FSDISOMsg(request); // NEW
        msg.setHeader(new byte[] { (byte) 1, (byte) 2, (byte) 3, (byte) 4});

        channel.send(msg);

        byte [] buffer = new byte [1024];
        byte [] buffer2 = new byte[1024];

        //channel.getBytes(buffer);
        FSDISOMsg rep = new FSDISOMsg(resp); // NEW
        rep.merge(channel.receive()); // NEW

        // message length (network byte order) and header account for the
        // first 6 received bytes. remove them for the actual message
        //System.arraycopy(buffer, 6, buffer2, 0, buffer.length - 6);

        //resp.unpack(buffer2);

        // return resp;

        return rep.getFSDMsg(); // NEW
    }
}


Here's the code for the driver class:

public class Program {
    public static void main(String [] args) throws Exception {
        ThalesAdaptor adaptor = new ThalesAdaptor("127.0.0.1", 9998);
        adaptor.connect();

        FSDMsg req = adaptor.diagnostics();

        req.dump(System.out, " ");

        FSDMsg resp = adaptor.command(req);

        resp.dump(System.out, " ");

        req = adaptor.generateDoubleLengthKey();

        req.dump(System.out, " ");

        resp = adaptor.command(req);

        resp.dump(System.out, " ");
    }
}



And here's the output:

<fsdmsg schema='file:cfg/hsm-base'>
   command: 'NC'
 </fsdmsg>
Exception in thread "main" org.jpos.iso.ISOException: java.lang.NullPointerException (java.lang.NullPointerException)
    at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
    at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
    at com.projxchange.magikstik.qudit.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:75)
    at com.projxchange.magikstik.qudit.Program.main(Program.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Nested:java.lang.NullPointerException
    at java.lang.StringBuilder.<init>(StringBuilder.java:92)
    at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:576)
    at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:572)
    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:193)
    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:210)
    at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
    at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
    at com.projxchange.magikstik.qudit.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:75)
    at com.projxchange.magikstik.qudit.Program.main(Program.java:22)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)

But if i remove the new code and uncomment the relevant lines in the command() method, it works as expected. Might anyone be able to guide me in properly unpacking the received data? Thanks in advance.

Mark Salter

unread,
Mar 3, 2011, 6:54:33 AM3/3/11
to jpos-...@googlegroups.com
You really need to debug this and help yourself learn why your code
change is not working. It is a great way to learn jPOS (and java).

On 03/03/2011 11:36, 'Dipo Odumosu wrote:
> | FSDISOMsg rep = new FSDISOMsg(resp); // NEW
> rep.merge(channel.receive()); // NEW

HINT: Are you *certain* that what you get back from the channel is
*always* valid input into a FSDISOMsg.merge call?

>
> // message length (network byte order) and header account for the
> // first 6 received bytes. remove them for the actual message
> //System.arraycopy(buffer, 6, buffer2, 0, buffer.length - 6);
>
> //resp.unpack(buffer2);
>
> // return resp;
>
> return rep.getFSDMsg(); // NEW
> }
> }|

What data is being received through the channel during this run...

> Exception in thread "main" org.jpos.iso.ISOException:
> java.lang.NullPointerException (java.lang.NullPointerException)
> at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)

...I have to suspect is was null?

> But if i remove the new code and uncomment the relevant lines in the
> command() method, it works as expected. Might anyone be able to guide me
> in properly unpacking the received data? Thanks in advance.

Make sure you have valid data before trying to incur an unpack.

8)

--
Mark

Mark Salter

unread,
Mar 3, 2011, 6:59:10 AM3/3/11
to jpos-...@googlegroups.com
On 03/03/2011 11:36, 'Dipo Odumosu wrote:
> | // message length (network byte order) and header account for the
> // first 6 received bytes. remove them for the actual message
> //System.arraycopy(buffer, 6, buffer2, 0, buffer.length - 6);|
Also, is this change related?
Are you uncommenting this 'System.arraycopy' code to make 'it works as
expected'?

If not, how are you handling the header, or isn't there one?

--
Mark

'Dipo Odumosu

unread,
Mar 3, 2011, 7:09:08 AM3/3/11
to jpos-...@googlegroups.com
In the 'original' code, I use the System.arraycopy() call to ensure that the message length header (2 bytes, nbo) and also the 4-byte header I set (the line with the call to  msg.setHeader()) is removed (in total, 6 bytes). You might have noticed that I'm copying the entire contents of buffer to buffer2 except for the first 6 bytes of buffer in that System.arraycopy() call.

I'm not sure how I'm supposed to remove the header if I'm to use channel.receive() (I reckon it should be in the received (FSD)ISOMsg anyway, before I begin fiddling with it). Furthermore, I was actually hoping to use the header the way you can configure QMUX to use field 11, 41 and other fields to tag the retrieved ISOMsg instance and match it to its correct recipient (on a thread level).

You're probably right on my needing to debug this myself. Was about to download jPOS off github when your email came in.

'Dipo Odumosu

unread,
Mar 3, 2011, 7:38:39 AM3/3/11
to jpos-...@googlegroups.com
Looking at the code, the channel (BaseChannel) delegates unpacking to
the ISOMsg. FSDISOMsg also delegates unpacking to its internal FSDMsg
instance. In this case, I might need to override the receive() method
and tell it to forward the actual byte data to an FSDMsg instance of the
required schema to unpack (I'd initially run into this problem, and
that's how I started getting the received bytes and unpacking them as my
'original' code shows). Until I do that, it's pretty definite that I
won't be able to use FSDMsg with any MUX variant. Ouch!


Mark Salter

unread,
Mar 3, 2011, 7:42:46 AM3/3/11
to jpos-...@googlegroups.com
On 03/03/2011 12:09, 'Dipo Odumosu wrote:
> In the 'original' code, I use the |System.arraycopy()| call to ensure
> that the message length header (2 bytes, nbo) and also the 4-byte header
> I set (the line with the call to |msg.setHeader()|) is removed (in

> total, 6 bytes). You might have noticed that I'm copying the entire
> contents of |buffer| to |buffer2| except for the first 6 bytes of
> |buffer| in that |System.arraycopy() |call.
I'm not sure I follow this, but as long as you are dealing with any
header somewhere, all might be well.

>
> I'm not sure how I'm supposed to remove the header if I'm to use
> channel.receive() (I reckon it should be in the received (FSD)ISOMsg
> anyway, before I begin fiddling with it).

The Channel will return an ISOMsg (*if* it has one (HINT)), the with the
right packager and arrangement, it might be ok - it is hard to be sure
from here.

> Furthermore, I was actually
> hoping to use the header the way you can configure QMUX to use field 11,
> 41 and other fields to tag the retrieved ISOMsg instance and match it to
> its correct recipient (on a thread level).

I would hope ISOSource or similar would be useful and available? I.e.
the connection to your system on which the request arrived and response
should be returned?

>
> You're probably right on my needing to debug this myself. Was about to
> download jPOS off github when your email came in.

Very good, check for a null back on the channel.receive in the first
instance 8).

--
Mark

Mark Salter

unread,
Mar 3, 2011, 7:48:05 AM3/3/11
to jpos-...@googlegroups.com
I'm not following this exactly, but I think you are trying to wrap the
response bytes back from your HSM as an (FSD)ISOMsg so that you can pass
it back somewhere and a MUX can match this response to a request?

As long as you can unpack the right bytes (that form a message that will
unpack with your Channel and packager) then it 'might just work'.

I still think you can use an FSDISOMsg (which really is an ISOMsg
because it extends that class!) through a MUX...


... please bear in mind that I can't really see what you are attempting
to do (yet?).

--
Mark

'Dipo Odumosu

unread,
Mar 3, 2011, 8:39:18 AM3/3/11
to jpos-...@googlegroups.com
I'll try to explain the problem.

Top-level 'want': To send an FSDISOMsg over the wire and receive an ISOMsg (which I can cast to FSDISOMsg). I hope this makes sense.

tl;dr version of problem:
Nothing in the 'regular' ISOMUX-ISOChannel-ISOPackager stream 'knows' how to 'unserialize' a byte stream (array) to an FSDMsg instance, not even FSDPackager.

Longer analysis (we'll assume the packager is an FSDPackager instance, for sanity/clarity's sake):
  • FSDChannel extends NACChannel, which extends BaseChannel.
  • Neither FSDChannel nor NACChannel override BaseChannel's unpack() method.
  • BaseChannel's unpack() method creates an ISOMsg using createMsg(). This calls the packager's createISOMsg method().
  • FSDPackager's createISOMsg() only contains this: return new ISOMsg;
  • BaseChannel's unpack() tells the ISOMsg to unpack() the byte stream. We're still dealing with just a vanilla ISOMsg here, not an FSDISOMsg.
  • FSDISOMsg's unpack() tells its internal FSDMsg instance to unpack the byte stream. ISOMsg's version tells the packager to unpack() using itself (it's an ISOComponent).
  • The packager tells the ISOComponent to unpack it. FSDMsg does not implement ISOComponent.
  • In the entire scenario above, there's no FSDMsg doing any unpacking. Unpacking fails.
I downloaded the master from github, so I guess I'm up to date on this one. Maybe Andy Orrock could throw some light on this, since he's used to using FSDMsg.

Summary: With the current source, I think (but I'll be glad to be proven wrong) it's impossible to get an FSDISOMsg from a channel.receive() call. The binary data won't be properly unpacked into an ISOMsg instance.

David Bergert

unread,
Mar 3, 2011, 9:18:06 AM3/3/11
to jpos-...@googlegroups.com
"Top-level 'want': To send an FSDISOMsg over the wire and receive an ISOMsg (which I can cast to FSDISOMsg). I hope this makes sense."

1) Prepare a FSDMsg and convert to FSDISO to send:

protected ISOMsg methodName throws Exception {

FSDMsg fsd = new FSDMsg ("file:cfg/fsd-");
fsd.set("41",xxxx) //MUX key field
fsd.set("x,"xxx);
fsd.set("x,"xxx);
fsd.set("x,"xxx);
fsd.set("x,"xxx);
fsd.set("x,"xxx);
fsd.set("x,"xxx);

return new FSDISOMsg (fsd);
}

2) Send it as an ISOMsg via normal jpos channels.

Note the outbound channel must use the Dummy Packager

packager="org.jpos.iso.packager.DummyPackager"

and your channel with want to create an outbound FSDISO message with the proper fsd schema and likely override like so.

public ISOMsg createMsg() {
return new FSDISOMsg (new FSDMsg (schema));
}


3) Receive the ISO msg response and cast to FSDISO then get the fsdmsg inside of it,

//Take the FSDISO convert to FSD, and get the meth-trans-replay value, and set in response.

FSDISOMsg riso = (FSDISOMsg) response;
if (riso != null) {
FSDMsg rfsd = riso.getFSDMsg();
...
...
}


David Bergert, CISSP, CISA, CPISM/A
www.paymentsystemsblog.com
>

'Dipo Odumosu

unread,
Mar 3, 2011, 9:27:22 AM3/3/11
to jpos-...@googlegroups.com
Domo arigato gozaimasu, Bergert-san, Salter-san. There had to be a much
less painless way to get it done than what I envisioned myself needing
to do, and this seems to be it. Thanks once again.

AAO

unread,
Mar 3, 2011, 9:30:04 AM3/3/11
to jpos-...@googlegroups.com
Excellent.

Some additional info is this post:

http://www.andyorrock.com/2007/07/as-may-you-know.html

...but my piece doesn't cover channel mechanics as Dave has done here.

Andy


>

--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage.
Please support jPOS, contact: sa...@jpos.org

You received this message because you are subscribed to the  "jPOS Users" group.
Please see http://jpos.org/wiki/JPOS_Mailing_List_Readme_first
To post to this group, send email to jpos-...@googlegroups.com
To unsubscribe, send email to jpos-users+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/jpos-users

Mark Salter

unread,
Mar 3, 2011, 9:39:18 AM3/3/11
to jpos-...@googlegroups.com
On 03/03/2011 13:39, 'Dipo Odumosu wrote:
> I'll try to explain the problem.
Thanks 8).

>
> *Top-level 'want':* To send an FSDISOMsg over the wire and receive an


> ISOMsg (which I can cast to FSDISOMsg). I hope this makes sense.

It does to me and I think it should be straightforward, once everything
'aligns' (including my mind with your intent 8))...

> *
> tl;dr version of problem:* Nothing in the 'regular'
> ISOMUX-ISOChannel-ISOPackager stream 'knows' how to 'unseriali (ze' a byte


> stream (array) to an FSDMsg instance, not even FSDPackager.

Here I think I disagree, but I also think you might have a mix of items
that is confusing this/me ...

I thought you wanted to send an (FSD)ISOMsg to an HSM, get a response
back, perhaps even MUX'ing the HSM connection to match response to request?

So we started off talking about FSDISOMsg's and MUX (which I think will
work, but now we have jumped 'down' to handling the binary data passing
across a wire, which is a big drop - my ears popped - 8).


> *Summary:* With the current source, I think (but I'll be glad to be


> proven wrong) it's impossible to get an FSDISOMsg from a
> channel.receive() call. The binary data won't be properly unpacked into
> an ISOMsg instance.
>

I think you need to :-

override yourChannel.createMsg() :-

to return a FSDISOMsg that is 'pre-seeded' with an FSDMsg
that in turn 'knows' your config

or indeed override yourPackager.getISOMsg() to return the same.


Then, when the BaseChannel.unpack (ISOMsg, byte[]) is called from
BaseChannel.receive() (around line 669), I think it will just work as
the FSDISOMsg will be given the byte[] receive to unpack.

Obviously this is just on paper and success will depend on what you have
and if the 'pre-seeded' FSDISOMsg is possible for you to attain and
manage, I have not managed to check that.

I wonder if the FSDMsg config is the same on request and response?
Could you save the FSDMsg somewhere 'handy' to reuse it for the
response? You will also then be dependant on sending a request before
receiving a response, but this should be easily handled.

Does this help?


--
Mark

Mark Salter

unread,
Mar 3, 2011, 9:41:18 AM3/3/11
to jpos-...@googlegroups.com
On 03/03/2011 14:39, Mark Salter wrote:

> Does this help?
>
>


Arrrrrrgh, I took too long realising the ask and checking my suggestion!!!

I see Dave and Andy concur (and beat me to responding). That's what
experience gives you 8).

What a team!

8)

--
'Hugs'
Mark

'Dipo Odumosu

unread,
Mar 4, 2011, 3:35:54 AM3/4/11
to jpos-...@googlegroups.com
Hello again,

Please excuse my bumping the thread again. I'm implementing as Andy &
David have said:

public class ThalesChannel extends FSDChannel {
String basePath;
String schema;

public String getBasePath() {
return basePath;
}

public void setBasePath(String basePath) {
this.basePath = basePath;
}

public String getSchema() {
return schema;
}

public void setSchema(String schema) {
this.schema = schema;
}

@Override
public ISOMsg createMsg() {
if (basePath != null && schema != null)
return new FSDISOMsg(new FSDMsg(basePath, schema));

if (basePath != null)
return new FSDISOMsg(new FSDMsg(basePath));

return new FSDISOMsg(); // no-op. will generate
exceptions if used.
}
}

A (possibly academic) note: if I'm using a MUX, how does the MUX resolve
what schema to use? Let's say thread A wants to send a diagnostics (NC)
request, and thread B wants to send a key generate request (A0), unless
I subclass the MUX as well, there's no way to determine the schema
beforehand.

Thanks for all the help once again. Off to determine how MUXes work :)

Mark Salter

unread,
Mar 4, 2011, 4:28:54 AM3/4/11
to jpos-...@googlegroups.com
On 04/03/2011 08:35, 'Dipo Odumosu wrote:
> A (possibly academic) note: if I'm using a MUX, how does the MUX resolve
> what schema to use? Let's say thread A wants to send a diagnostics (NC)
> request, and thread B wants to send a key generate request (A0), unless
> I subclass the MUX as well, there's no way to determine the schema
> beforehand.
The MUX doesn't care or need to, it just handles ISOMsg instances,
asking them for the key fields for matching...

... I'm just wondering about the matching process and it's use of
ISOMsg.getMTI in this context - and if it will work...

Doesn't the hierarchy of scheme file(s), driven by key fields tagged
within the schema (key='true') ultimately produce the right schema for
all you message/command types?

I am recollecting something I have not looked at for a few years, so I
would suggest you check and see - let me know?

If it is more complex than I imagine, then you can also consider
overriding getDynamicPackager on your Channel and peek into the
header/raw to make your schema/base choices.


>
> Thanks for all the help once again. Off to determine how MUXes work :)
>

MUX's in jPOS are brilliantly simple 8), especially when combined with
using a space (QMUX) for in/out/unsolicited.

Have fun.


--
Mark

AAO

unread,
Mar 4, 2011, 1:24:06 PM3/4/11
to jpos-...@googlegroups.com, Mark Salter
"Doesn't the hierarchy of scheme file(s), driven by key fields tagged
within the schema (key='true') ultimately produce the right schema for
all you message/command types?"

Exactly.

We do things like this in our message request code:

        FSDMsg fsd = new FSDMsg ("file:cfg/verizon-");
        FSDMsg msg = (FSDMsg) ctx.tget (REQUEST);
        assertNotNull (msg, APPLERR_BADDATA,  "CreateVerizonRequest: invalid VG2 request");

        fsd.set ("0", mti);
        fsd.set ("request-source", "0");

        String storeNumber = msg.get ("store-number");
        fsd.set ("store-number", storeNumber);
        [etc.]

As Mark says, key fields indicate which part of the schema is in play. 

The full Verizon example I posted here:

http://www.andyorrock.com/2007/07/as-may-you-know.html

doesn't have 'key = true' in examples

Here's a simple example of that in action (on a batch file, but OLTP principles are the same):

sercoup-import-base.xml:

<?xml version="1.0" encoding="UTF-8"?>

<schema>

 <field id='record-id' type='A' length='1' key='true' />
</schema>

sercoup-import-H-.xml:

<?xml version="1.0" encoding="UTF-8"?>

<schema id='H'> <!-- Header Record -->
 <field id='file-name'                 type='A' length='7'  />
 <field id='vendor-name'               type='A' length='20' />
 <field id='file-date'                 type='A' length='8'  />
 <field id='file-seq-number'           type='N' length='6'  />
</schema>

sercoup-import-D-.xml:

<?xml version="1.0" encoding="UTF-8"?>

<schema id='D'> <!-- Detail Record -->
 <field id='message-id'                type='A' length='5'  />
 <field id='card-number'               type='A' length='12' />
 <field id='how-many'                  type='N' length='8'  />
 <field id='barcode'                   type='A' length='40' />
 <field id='start-issuance-date'       type='A' length='10'  ><!--
  Dates are in YYYY-MM-DD format
  --> </field>
 <field id='end-issuance-date'         type='A' length='10' />
 <field id='end-redemption-date'       type='A' length='10' />
</schema>

sercoup-import-T-.xml:

<?xml version="1.0" encoding="UTF-8"?>

<schema id='T'> <!-- Trailer Record -->
 <field id='trailer-counter'           type='N' length='7'  />
</schema>

Andy


'Dipo Odumosu

unread,
Mar 7, 2011, 5:57:01 AM3/7/11
to jpos-...@googlegroups.com
Hello again,

I've been working on the Thales simulator with FSDMsg (although I've not
yet switched my code to using the MUX as I want to be certain I
understand the points raised by Andy, David and Mark previously), and
I've run into issues with getting RuntimeException being thrown on
message unpacking.

However, if I add a hsm-resp-null.xml file with the same contents as my
hsm-resp-base.xml file, I don't run into issues.

I want to apologize for dumping the entire listing below. I thought it
might be easier to see the code in one page than upload an archive with
the sources. If the latter is more acceptable, please let me know.
thanks for your help in advance.

Here are my source files:

----- Program.java -----

public class Program {
public static void main(String [] args) throws Exception {
ThalesAdaptor adaptor = new ThalesAdaptor("127.0.0.1", 9998);
adaptor.connect();

FSDMsg req = adaptor.diagnostics();

req.dump(System.out, " ");

FSDMsg resp = adaptor.command(req);

resp.dump(System.out, " ");

req = adaptor.generateDoubleLengthKey();

req.dump(System.out, " ");

resp = adaptor.command(req);

resp.dump(System.out, " ");

// this returns a ZMK parity error (code 10)
req =
adaptor.importDoubleLengthKey("C15745088502B0D3A2EF68435EE6D075",
"5C2F0A5F099655FB9ED9905B5C6743CE");

req.dump(System.out, " ");

resp = adaptor.command(req);

resp.dump(System.out, " ");
}
}

----- ThalesAdaptor.java -----
public class ThalesAdaptor {
private ThalesChannel channel;

public ThalesAdaptor(String host, int port) {

channel = new ThalesChannel();
channel.setHost(host, port);
channel.setPackager(new DummyPackager());
}

public void connect() throws Exception {
channel.connect();
}

private FSDMsg createRequest(String command) {
FSDMsg req = new FSDMsg("file:cfg/hsm-");

if (command != null)
req.set("command", command);

return req;
}

private FSDMsg createResponse(String response) {
FSDMsg resp = new FSDMsg("file:cfg/hsm-resp-");

if (response != null)
resp.set("response", response);

return resp;
}

public FSDMsg diagnostics() {
return createRequest("NC");
}

public FSDMsg generateDoubleLengthKey() {
FSDMsg req = createRequest("A0");
req.set("mode", "0");
req.set("key-type", "001");
req.set("key-scheme-lmk", "U");

return req;
}

public FSDMsg importDoubleLengthKey(String zmk, String key) {
FSDMsg req = createRequest("A6");


req.set("key-type", "001");

req.set("zmk", "X" + zmk);
req.set("key-under-zmk", "X" + key);
req.set("key-scheme", "X");

return req;
}

public FSDMsg command(FSDMsg request) throws Exception {
StringBuffer sbuffer = new StringBuffer(request.get("command"));
sbuffer.setCharAt(1, (char) (sbuffer.charAt(1) + 1));
FSDMsg resp = createResponse(sbuffer.toString());

FSDISOMsg msg = new FSDISOMsg(request);

msg.setHeader(new byte[] { (byte) 1, (byte) 2, (byte) 3, (byte)
4});

channel.setBasePath(resp.getBasePath());
channel.setSchema(resp.getBaseSchema());
channel.send(msg);

FSDISOMsg response = (FSDISOMsg) channel.receive();

resp.merge(response.getFSDMsg());

return resp;
}
}

----- ThalesChannel.java -----

public class ThalesChannel extends FSDChannel {
String basePath;
String schema;

public String getBasePath() {
return basePath;
}

public void setBasePath(String basePath) {
this.basePath = basePath;
}

public String getSchema() {
return schema;
}

public void setSchema(String schema) {
this.schema = schema;
}

@Override
public ISOMsg createMsg() {
if (basePath != null && schema != null)
return new FSDISOMsg(new FSDMsg(basePath, schema));

if (basePath != null)
return new FSDISOMsg(new FSDMsg(basePath));

return new FSDISOMsg(); // no-op. will generate
exceptions if used.
}
}

----- hsm-base.xml -----

<?xml version='1.0' encoding='utf-8'?>
<schema>
<field id='command' type='A' length='2' key='true'/>
</schema>

----- hsm-resp-base.xml -----

<?xml version='1.0' encoding='UTF-8'?>
<schema>

<field id='response' type='A' length='2' key='true'/>
<field id='error' type='A' length='2'/>
</schema>

----- hsm-resp-null.xml has the same contents as the above -----

----- hsm-NC.xml -----


<?xml version='1.0' encoding='UTF-8'?>

<schema id='NC'>
</schema>

----- hsm-resp-ND-xml -----


<?xml version="1.0" encoding="UTF-8"?>

<schema id='ND'>
<!--this is the LMK check value-->
<field id="lmk-check-value" type="A" length="16" />
<!--this is the firmware number-->
<field id="firmware-number" type="A" length="9" />
</schema>

----- hsm-A0.xml -----
<?xml version='1.0' encoding='utf-8'?>
<schema id='A0'>
<field id='mode' type='N' length='1'/>
<field id='key-type' type='A' length='3'/>
<field id='key-scheme-lmk' type='A' length='1'/>
</schema>

----- hsm-resp-A1.xml -----
<?xml version='1.0' encoding='utf-8'?>
<schema id='A1'>
<field id='key-lmk' type='A' length='33'/>
<field id='check-value' type='A' length='6'/>
</schema>

----- hsm-A6.xml -----


<?xml version='1.0' encoding='UTF-8'?>

<schema id='A6'>
<field id='key-type' type='A' length='3'/>
<field id='zmk' type='A' length='33'/>
<field id='key-under-zmk' type='A' length='33'/>
<field id='key-scheme' type='A' length='1'/>
</schema>

----- hsm-A7.xml (i realized this file was incorrectly named, but
renaming it to hsm-resp-A7.xml didn't affect anything) -----


<?xml version='1.0' encoding='UTF-8'?>

<schema id='A7'>
<field id='key-under-lmk' type='AFS' length='33'/>
<field id='check-value' type='AFS' length='6'/>
</schema>

----- Output with hsm-resp-null.xml present: -----


<fsdmsg schema='file:cfg/hsm-base'>
command: 'NC'
</fsdmsg>

<fsdmsg schema='file:cfg/hsm-resp-base'>
response: 'ND'
error: '00'
lmk-check-value: '7B44AC1DDEE2A94B'
firmware-number: '0007-E000'
</fsdmsg>
<fsdmsg schema='file:cfg/hsm-base'>
command: 'A0'
mode: '0'
key-type: '001'
key-scheme-lmk: 'U'
</fsdmsg>
<fsdmsg schema='file:cfg/hsm-resp-base'>
response: 'A1'
error: '00'
key-lmk: 'U94D098BBE54537CC78BE9ED1DFC46F0A'
check-value: '02686B'
</fsdmsg>
<fsdmsg schema='file:cfg/hsm-base'>
command: 'A6'
key-type: '001'
zmk: 'XC15745088502B0D3A2EF68435EE6D075'
key-under-zmk: 'X5C2F0A5F099655FB9ED9905B5C6743CE'
key-scheme: 'X'
</fsdmsg>
<fsdmsg schema='file:cfg/hsm-resp-base'>
response: 'A7'
error: '10'
EOF: 'true'
</fsdmsg>

----- Output with hsm-resp-null.xml absent ----


<fsdmsg schema='file:cfg/hsm-base'>
command: 'NC'
</fsdmsg>
Exception in thread "main" org.jpos.iso.ISOException:

java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not
found (java.lang.RuntimeException:
C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not found)


at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
at

com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:96)
at com.projxchange.sandbox.Program.main(Program.java:22)


at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)

Nested:java.lang.RuntimeException:
C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not found
at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:602)
at org.jpos.util.FSDMsg.unpack(FSDMsg.java:433)


at org.jpos.util.FSDMsg.unpack(FSDMsg.java:193)
at org.jpos.util.FSDMsg.unpack(FSDMsg.java:210)
at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
at

com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:96)
at com.projxchange.sandbox.Program.main(Program.java:22)

Mark Salter

unread,
Mar 7, 2011, 6:17:55 AM3/7/11
to jpos-...@googlegroups.com
On 07/03/2011 10:57, 'Dipo Odumosu wrote:
> Exception in thread "main" org.jpos.iso.ISOException:
> java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not
> found (java.lang.RuntimeException:
> C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not found)


Here the field being used as a key contains null...

... which is being used as a String and key (and part of the name of the
target schema file).

So where is this null coming from?

Some guesses:-

Perhaps it is the end of a message left over from an incompletely parsed
previous message?

Perhaps the HSM really is returning a null/empty message?


Perhaps a network trace (wireshark?) will help you see the data that is
causing the problem...


How about a timeout?

--
Mark

'Dipo Odumosu

unread,
Mar 10, 2011, 10:11:10 AM3/10/11
to jpos-...@googlegroups.com
Hello,

I was able to create a very small sample to test, and ran smack into a NullPointerException. My guess is that on receiving, I'm still running into unpacking issues.

I think the problem is that when the ChannelAdapter is receiving the bytes, it still calls channel.receive(), which calls channel.unpack(). If I were handling the channel directly, I could tell it to set the schema of the FSDMsg needed to unpack the bytestream. In a situation using a MUX; however, any thread could send any FSDISOMsg with any schema through the mux, and short of pushing and popping some parameters on and off a stack, I cannot currently think of how to preserve the state (aka determine which thread sent what message, and which schema to use for unpacking the designated response).

I've thought of retrieving the schema from the FSDISOMsg when sending, in order to use it for unpacking the received message, but that depends responses being delivered in the order in which the original requests are sent.

I'd really appreciate it if anyone could throw light my way. Thank you in advance.

 Here's my source files:
  • ThalesChannel.java:


  • public class ThalesChannel extends FSDChannel {
        String basePath;

  •     public String getBasePath() {
            return basePath;
        }

        public void setBasePath(String basePath) {
            this.basePath = basePath;
        }

  •     public ISOMsg createMsg() {

  •         if (basePath != null)
                return new FSDISOMsg(new FSDMsg(basePath));

            return new FSDISOMsg();
        }
    }

  • ThalesAdaptor.java:

    public class ThalesAdaptor {
        public FSDMsg diagnostics() {
            FSDMsg msg = new FSDMsg("file:cfg/thales-diagnostics-");

            return msg;
        }

        public FSDMsg diagnostics2() {
            FSDMsg msg = new FSDMsg("file:cfg/thales-diagnostics-resp-");

            return msg;
        }
    }

  • thales-diagnostics-base.xml:


  • <?xml version='1.0' encoding='UTF-8'?>
    <schema>
  •     <field id='action' type='A' length='2'>NC</field>
        <field id='11' type='N' length='6'/>
        <field id='41' type='A' length='8'/>
    </schema>

  • thales-diagnostics-resp-base.xml:


  • <?xml version='1.0' encoding='UTF-8'?>
    <schema>
  •     <field id='response' type='A' length='2'>ND</field>
        <field id='error-code' type='A' length='2'/>

  •     <field id='lmk-check-value' type='A' length='16'/>
  •     <field id='firmware-number' type='A' length='9'/>
  •     <field id='11' type='N' length='6'/>
        <field id='41' type='A' length='8'/>
    </schema>

  • ThalesCore.java:

    public class ThalesCore extends QBeanSupport {
        QMUX multiplexer;

        public void init() {
            try {
                multiplexer = (QMUX) NameRegistrar.get("mux." + cfg.get("multiplexer"));
            }
            catch (NameRegistrar.NotFoundException ex) {
                log.error(ex);
                multiplexer = null;
            }
        }

        public void start() {
            if (multiplexer == null)
                return;

            ThalesAdaptor adaptor = new ThalesAdaptor();

            FSDMsg msg = adaptor.diagnostics();

            msg.set("11", RandomStringUtils.randomNumeric(6)); // from Apache Commons.Lang
            msg.set("41", RandomStringUtils.randomAlphanumeric(8));

            FSDISOMsg wrapper = new FSDISOMsg(msg);

            byte [] header = new byte [] { (byte) 0x41, (byte) 0x41, (byte) 0x41, (byte) 0x41};

            wrapper.setHeader(header);

            try {
                ISOMsg response = multiplexer.request(wrapper, 30000);

                if (response == null) {
                    log.error("Request timed out");
                    return;
                }

                FSDMsg msg2 = adaptor.diagnostics2();

                FSDISOMsg resp = new FSDISOMsg(msg2);

                resp.merge(response);
            }
            catch (ISOException ex) {
                log.error(ex);
            }
        }
    }
Outputs:
  • q2.log:

    <log realm="channel/127.0.0.1:9998" at="Thu Mar 10 15:43:58 WAT 2011.62" lifespan="31ms">
      <connect>
        127.0.0.1:9998
      </connect>
    </log>
    <log realm="channel/127.0.0.1:9998" at="Thu Mar 10 15:43:58 WAT 2011.62">
      <send>
        <fsdmsg schema='file:cfg/thales-diagnostics-base'>
          11: '242344'
          41: 'dtRjJD15'
        </fsdmsg>
      </send>
    </log>
    <log realm="channel/127.0.0.1:9998" at="Thu Mar 10 15:43:58 WAT 2011.187">
      <fsd-channel-debug>
        received message length: 33
      </fsd-channel-debug>
    </log>
    <log realm="channel/127.0.0.1:9998" at="Thu Mar 10 15:43:58 WAT 2011.187" lifespan="125ms">
      <receive>
        <iso-exception>
          java.lang.NullPointerException
          <nested-exception>
          java.lang.NullPointerException

  •     at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
        at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
        at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
  •     at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:302)
        at java.lang.Thread.run(Thread.java:662)
          </nested-exception>
          org.jpos.iso.ISOException: java.lang.NullPointerException (java.lang.NullPointerException)

  •     at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
        at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
        at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
  •     at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:302)
        at java.lang.Thread.run(Thread.java:662)
    Nested:java.lang.NullPointerException

  •     at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
        at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
        at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
  •     at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:302)
        at java.lang.Thread.run(Thread.java:662)
        </iso-exception>
        --- data ---
        0000  41 41 41 41 4E 44 30 30  37 42 34 34 41 43 31 44  AAAAND007B44AC1D
    0010  44 45 45 32 41 39 34 42  30 30 30 37 2D 45 30 30  DEE2A94B0007-E00
    0020  30                                                0

      </receive>
    </log>
    <log realm="org.jpos.q2.iso.ChannelAdaptor" at="Thu Mar 10 15:43:58 WAT 2011.265">
      <warn>
        channel-receiver-thales-receive
        <iso-exception>
          java.lang.NullPointerException
          <nested-exception>
          java.lang.NullPointerException

  •     at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
        at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
        at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
  •     at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:302)
        at java.lang.Thread.run(Thread.java:662)
          </nested-exception>
          org.jpos.iso.ISOException: java.lang.NullPointerException (java.lang.NullPointerException)

  •     at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
        at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
        at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
  •     at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:302)
        at java.lang.Thread.run(Thread.java:662)
    Nested:java.lang.NullPointerException

  •     at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
        at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
        at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
  •     at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:302)
        at java.lang.Thread.run(Thread.java:662)
        </iso-exception>
      </warn>
    </log>

  • Thales simulator log:

    Client from 127.0.0.1:1516 is connected
    Client: 127.0.0.1:1516
    Request: AAAANC242344dtRjJD15
    Parsing header and code of message AAAANC242344dtRjJD15...
    Searching for implementor of NC...
    Found implementor ThalesSim.Core.HostCommands.Runtime.HSMDiagnostics_NC, instantiating...
    Calling AcceptMessage()...
    Calling ConstructResponse()...
    Calling ConstructResponseAfterOperationComplete()...
    Attaching header/response code to response...
    Sending: AAAAND007B44AC1DDEE2A94B0007-E000
    Calling Terminate()...
    Implementor to Nothing
    Client disconnected.

'Dipo Odumosu

unread,
Mar 10, 2011, 10:26:47 AM3/10/11
to jpos-...@googlegroups.com
Hello,
To make the scenario understandable, let's assume two threads, A and B, want to send two FSD messages to the mux. Thread A is using schema 1, thread B uses schema 2. Unless there is guaranteed first-in, first out delivery of responses, the mux doesn't (and shouldn't) know what schema to use to unpack, nor does the channel adapter. How then can I successfully ensure every time I send an FSDISOMsg over the mux, the correct schema is used for unpacking responses? I will be using this (if it eventually matures) in a multithreaded environment and I think it's a valid concern. If I only was carrying out one operation, I could hardcode the schema for unpacking and move on my merry way. If I ever need to do more than one operation (read: use more than one unpacking schema), that approach would not work, and I've not been able to come up with an effective solution. I'm begging your pardon for what might pass for an awfully long-winded discourse.

Regards.

Mark Salter

unread,
Mar 10, 2011, 10:32:40 AM3/10/11
to jpos-...@googlegroups.com
On 10/03/2011 15:11, 'Dipo Odumosu wrote:

> I was able to create a very small sample to test, and ran smack into a
> NullPointerException. My guess is that on receiving, I'm still running
> into unpacking issues.

If it is during unpack then it must be during receiving.

>
> If I were handling the channel directly, I could tell it to set the
> schema of the FSDMsg needed to unpack the bytestream.

There were a few options given above, all of which give you this control.

You can control whatever processing you like by extending or overriding
a Channel with your own code?
See getDynamicPackager(header, b), here you have the chance to inspect
any header and the raw data and select a Packager to deal with the
message in hand.

Perhaps instead of using random strings in 11 and 41, concoct a way of
indicating the schema/packager to be use?

> In a situation
> using a MUX; however, any thread could send any FSDISOMsg with any
> schema through the mux, and short of pushing and popping some parameters
> on and off a stack, I cannot currently think of how to preserve the
> state (aka determine which thread sent what message, and which schema to
> use for unpacking the designated response).

As above, use 11+41 to do something useful?

BUT, I still don't understand though *why* just one schema - with key
fields to trigger schema qualification can't just work for you?

Perhaps you missed it, but I asked why a single set of schema files
won't work for you. My fear is you are making this *much* harder than
you need to.

So, can you explain why you think you need to vary the scheme base with
each response please?

> Here's my source files:
>

[snip]


> public ISOMsg createMsg() {
> if (basePath != null)
> return new FSDISOMsg(new FSDMsg(basePath));

Is basePath always being set ahead of being used?
>
> return new FSDISOMsg();
do you *ever* get to this return instead of a qualified schema? Would
this cause a NPE?

> }
> }
>
> * ThalesAdaptor.java:


>
> public class ThalesAdaptor {
> public FSDMsg diagnostics() {
> FSDMsg msg = new FSDMsg("file:cfg/thales-diagnostics-");

Why do you need a schema for requests...


>
> return msg;
> }
>
> public FSDMsg diagnostics2() {
> FSDMsg msg = new FSDMsg("file:cfg/thales-diagnostics-resp-");

and another one for responses???

This is *key* and the answer is important and will make you life much
easier, so please answer this time 8)


[snip]

> Sending: AAAAND007B44AC1DDEE2A94B0007-E000
Can you correctly/manually parse this response - ever?

--
Mark

Mark Salter

unread,
Mar 10, 2011, 10:35:14 AM3/10/11
to jpos-...@googlegroups.com
On 10/03/2011 15:26, 'Dipo Odumosu wrote:
> If I only was carrying out one operation, I could
> hardcode the schema for unpacking and move on my merry way.
It is this position I hope to get you to...


... having a single schema to handle requests *and* responses is what I
think you should have.

The fact you are dealing with requests differently and distinctly from
responses - and probably don't need to - is the root cause of your trouble.

Perhaps your HSM is odd in some way, but I suspect it isn't 8).


--
Mark

'Dipo Odumosu

unread,
Mar 10, 2011, 5:17:03 PM3/10/11
to jpos-...@googlegroups.com
Hello Mark,
I'll look into it when I get back to the office (it's quite late here and I need my beauty sleep ;), but I'm not sure how to use one schema for requests and responses.
 
Thanks

Mark Salter

unread,
Mar 10, 2011, 5:32:08 PM3/10/11
to jpos-...@googlegroups.com
On 10/03/2011 22:17, 'Dipo Odumosu wrote:
> I'll look into it when I get back to the office (it's quite late here and I
> need my beauty sleep ;)
I understand, there is no timeout on my advise.

> , but I'm not sure how to use one schema for requests
> and responses.

Check back up this thread, Andy kindly shared an example when I checked
my understanding here:-

http://groups.google.com/group/jpos-users/msg/b03eb9b17d35a29c

You will need to adjust to fit, but the *key* part of the schema to
observe and consider is:-

key='true'

then observe the file names and see if the penny (hopefully) drops.

I will be listening out from it, I suspect the impact might be 'heard'
on the Richter scale 8).

--
Mark

'Dipo Odumosu

unread,
Mar 11, 2011, 7:29:01 AM3/11/11
to jpos-...@googlegroups.com
Hello,

As Mark pointed out, my key field (schema selection field) is being set
to null, yet from what the simulator is sending, there is no null. The
simulator dumps the hex of what it sends out, and I can't see a hex 00
(null character) in it. Here's the hex output for the diagnostics test:
41 41 41 41 4E 44 30 30 37 42 34 34 41 43 31 44 44 45 45 32 41 39 34 42
30 30 30 37 2D 45 30 30 30

What I've noticed is that if I have only either hsm-resp-base.xml or
hsm-resp-null.xml in my cfg folder, I get RuntimeException thrown on
unpack. If I have both, the 'right thing' magically happens, as shown in
this output:

<fsdmsg schema='file:cfg/hsm-base'>
command: 'NC'
</fsdmsg>
<fsdmsg schema='file:cfg/hsm-resp-base'>
response: 'ND'
error: '00'
lmk-check-value: '7B44AC1DDEE2A94B'
firmware-number: '0007-E000'
</fsdmsg>

For what it's worth, this is the definition of createMsg() on my channel
class:

@Override
public ISOMsg createMsg() {
return new FSDISOMsg(new FSDMsg("file:cfg/hsm-resp-"));
}

So I guess the hierarchy is properly being implemented.

Mark was right - I was over-analyzing (and over-complicating) things and
should just have trusted the schema hierarchy to properly work. I guess
setting the base schema (using the FSDMsg(String, String) constructor in
createMsg()) was one of the bad ideas I had, since when I tried
unpacking the bytes received on the channel manually with an FSDMsg
instance created using the FSDMsg(String, String) constructor, I got
nonsense data (or an exception, I can't really remember, it was a couple
of hours ago, and quite a bit of water has passed under the bridge since).

Also, a particular operation I was carrying out failed with an error,
and I noticed that case (or in any instance where the unpacking was
producing nonsense data - due to a misconfiguration), a 'magic field' of
sorts with the name EOF was set to true (or 'true', I suppose).

So the prevailing issue right now is eliminating the need for a
-null.xml file. Thanks for all the help, especially Mark, Andy and Dave.

Regards

antiguy

unread,
Mar 11, 2011, 8:48:25 AM3/11/11
to jPOS Users
>
> Perhaps instead of using random strings in 11 and 41, concoct a way of
> indicating the schema/packager to be use?
>
The random strings were just placeholders to ensure there was 'valid'
data, so the mux routing would work. I was testing with just one
request.
> BUT, I still don't understand though *why* just one schema - with key
> fields to trigger schema qualification can't just work for you?
>
> Perhaps you missed it, but I asked why a single set of schema files
> won't work for you.  My fear is you are making this *much* harder than
> you need to.
I didn't properly understand the hierarchy would 'come to life', as it
were, during the unpacking. My (wrong) assumption was that setting the
base schema would allow the 'correct thing' to happen. Unless I
mistake the results I got with testing, it doesn't. One should merely
set the base path, and the proper schema will be loaded during
unpacking, unless some magic happens.
>
>
>
>
> Is basePath always being set ahead of being used?
>
Yessir, it was properly being set. The issue, as I understand it, is
that baseSchema was also being set.

> >           }
> >      }
>
> >    * ThalesAdaptor.java:
>
> >      public class ThalesAdaptor {
> >           public FSDMsg diagnostics() {
> >               FSDMsg msg = new FSDMsg("file:cfg/thales-diagnostics-");
>
> Why do you need a schema for requests...
>
> >               return msg;
> >           }
>
> >           public FSDMsg diagnostics2() {
> >               FSDMsg msg = new FSDMsg("file:cfg/thales-diagnostics-resp-");
>
> and another one for responses???
>
> This is *key* and the answer is important and will make you life much
> easier, so please answer this time  8)
My interpretation of Andy's example for the ThalesAdapter was that the
response code from the HSM (simulator in this case) would be used to
set a key field and select the subschema to use. So if you sent an NC
command, you would automatically select the hsm-NC.xml schema, and the
response, ND would be used to select the hsm-resp-ND.xml schema.
>
> [snip]
>
> >      Sending: AAAAND007B44AC1DDEE2A94B0007-E000
>
> Can you correctly/manually parse this response - ever?
>
At the moment, no. Here's the code I'm using:

byte [] buffer = ISOUtil.hex2byte("ND007B44AC1DDEE2A94B0007-
E000");

FSDMsg msg = new FSDMsg("file:cfg/hsm-resp-");

FSDISOMsg rep = new FSDISOMsg(msg);

rep.unpack(buffer);

rep.dump(System.out, "\t");

I'm ignoring the header (AAAA) since there's no provision for it in
the hsm-resp-base.xml schema. I'm not including the FSD separator in
the string, since it's probably not present in the stream read off the
channel.

Contents of hsm-resp-base.xml:

<?xml version='1.0' encoding='UTF-8'?>
<schema>
<!--
<field id='nboheader' type='A' length='2'/>
<field id='isoheader' type='A' length='4'/>
-->
<field id='response' type='A' length='2' key='true'/>
<field id='error' type='A' length='2'/>
</schema>

Mark Salter

unread,
Mar 11, 2011, 9:03:09 AM3/11/11
to jpos-...@googlegroups.com
I've just noticed we have merged threads here, two different
hreads/subjects, so please be careful.

On 11/03/2011 13:48, antiguy wrote:
>>
>> Perhaps instead of using random strings in 11 and 41, concoct a way of
>> indicating the schema/packager to be use?
>>
> The random strings were just placeholders to ensure there was 'valid'
> data, so the mux routing would work. I was testing with just one
> request.

Fine, but this would be a place to make 'better' use.

>> BUT, I still don't understand though *why* just one schema - with key
>> fields to trigger schema qualification can't just work for you?
>>
>> Perhaps you missed it, but I asked why a single set of schema files
>> won't work for you. My fear is you are making this *much* harder than
>> you need to.
> I didn't properly understand the hierarchy would 'come to life', as it
> were, during the unpacking. My (wrong) assumption was that setting the
> base schema would allow the 'correct thing' to happen. Unless I
> mistake the results I got with testing, it doesn't. One should merely
> set the base path, and the proper schema will be loaded during
> unpacking, unless some magic happens.

I think you were missing the attributes to tell the FSDMsg to treat some
fields as keys and this use their content to construct the file name
selection?

>> This is *key* and the answer is important and will make you life much
>> easier, so please answer this time 8)
> My interpretation of Andy's example for the ThalesAdapter was that the
> response code from the HSM (simulator in this case) would be used to
> set a key field and select the subschema to use. So if you sent an NC
> command, you would automatically select the hsm-NC.xml schema, and the
> response, ND would be used to select the hsm-resp-ND.xml schema.

As long as the key=true' is present in base, I think it should work, I
think you had the 'key' attribute missing.

I think 'response message id' is clearer

>>> Sending: AAAAND007B44AC1DDEE2A94B0007-E000
>>
>> Can you correctly/manually parse this response - ever?
>>
> At the moment, no. Here's the code I'm using:

No?

What happens, you don't seem to say anywhere?

>
> byte [] buffer = ISOUtil.hex2byte("ND007B44AC1DDEE2A94B0007-
> E000");
>
> FSDMsg msg = new FSDMsg("file:cfg/hsm-resp-");
>
> FSDISOMsg rep = new FSDISOMsg(msg);
>
> rep.unpack(buffer);
>
> rep.dump(System.out, "\t");
>
> I'm ignoring the header (AAAA) since there's no provision for it in
> the hsm-resp-base.xml schema. I'm not including the FSD separator in
> the string, since it's probably not present in the stream read off the
> channel.

The header is dealt with by the Channel and is dealt with before
unpacking/packing.

>
> Contents of hsm-resp-base.xml:
>
> <?xml version='1.0' encoding='UTF-8'?>
> <schema>
> <!--
> <field id='nboheader' type='A' length='2'/>
> <field id='isoheader' type='A' length='4'/>
> -->
> <field id='response' type='A' length='2' key='true'/>
> <field id='error' type='A' length='2'/>
> </schema>
>

So...

what does your hsm-resp-ND.xml file look like?


--
Mark

Mark Salter

unread,
Mar 11, 2011, 9:05:41 AM3/11/11
to jpos-...@googlegroups.com
On 11/03/2011 12:29, 'Dipo Odumosu wrote:
> So the prevailing issue right now is eliminating the need for a
> -null.xml file.

Can you post your current schema files please?

Perhaps there is something 'obvious'...

8)


--
Mark

'Dipo Odumosu

unread,
Mar 11, 2011, 9:26:52 AM3/11/11
to jpos-...@googlegroups.com
I wasn't aware that replying & changing the subject would merge both
threads. My apologies to all.

> Fine, but this would be a place to make 'better' use.
Sure. In a more realistic test, I would actually set up a sequencer for
the tag field(s).

>
> I think you were missing the attributes to tell the FSDMsg to treat some
> fields as keys and this use their content to construct the file name
> selection?
>

No, I don't think so. For example, I haven't modified my hsm-base.xml
file in over a month. The modifications to hsm-resp-base.xml were to
deal with the NBO message length and 'tag' headers when using
channel.getBytes().


> I think 'response message id' is clearer
>

Well said, sir.


>>>> Sending: AAAAND007B44AC1DDEE2A94B0007-E000
>>> Can you correctly/manually parse this response - ever?
>>>
>> At the moment, no. Here's the code I'm using:
> No?
>
> What happens, you don't seem to say anywhere?
>

Output:
<fsdmsg schema='file:cfg/hsm-resp-base'>
response: '��'
error: '� <--- some other character here, but I can't paste the
entire thing. EOF, maybe?
EOF: 'true'
</fsdmsg>


>> byte [] buffer = ISOUtil.hex2byte("ND007B44AC1DDEE2A94B0007-
>> E000");
>>
>> FSDMsg msg = new FSDMsg("file:cfg/hsm-resp-");
>>
>> FSDISOMsg rep = new FSDISOMsg(msg);
>>
>> rep.unpack(buffer);
>>
>> rep.dump(System.out, "\t");
>>
>> I'm ignoring the header (AAAA) since there's no provision for it in
>> the hsm-resp-base.xml schema. I'm not including the FSD separator in
>> the string, since it's probably not present in the stream read off the
>> channel.
> The header is dealt with by the Channel and is dealt with before
> unpacking/packing.
>

Hence my not including it in the test code above


>> Contents of hsm-resp-base.xml:
>>
>> <?xml version='1.0' encoding='UTF-8'?>
>> <schema>
>> <!--
>> <field id='nboheader' type='A' length='2'/>
>> <field id='isoheader' type='A' length='4'/>
>> -->
>> <field id='response' type='A' length='2' key='true'/>
>> <field id='error' type='A' length='2'/>
>> </schema>
>>
> So...
>
> what does your hsm-resp-ND.xml file look like?
>

Complete with warts ;)

----- hsm-base.xml -----
<?xml version='1.0' encoding='utf-8'?>
<schema>

<field id='command' type='A' length='2' key='true'/>
</schema>

----- hsm-resp-base.xml (hsm-resp-null.xml is an exact duplicate of this
file) -----


<?xml version='1.0' encoding='UTF-8'?>
<schema>
<!--
<field id='nboheader' type='A' length='2'/>
<field id='isoheader' type='A' length='4'/>
-->
<field id='response' type='A' length='2' key='true'/>
<field id='error' type='A' length='2'/>
</schema>

----- hsm-NC.xml -----


<?xml version='1.0' encoding='UTF-8'?>

<schema id='NC'>
</schema>

----- hsm-resp-ND.xml -----


<?xml version="1.0" encoding="UTF-8"?>

<schema id='ND'>
<!--this is the 2-byte message length header-->
<!--<field id="messagelength" type="A" length="2" />-->
<!--this is the 4-byte header you need to prepend the actual data with,
byte[] header = new byte [] { 0, 0, 0, 0 }; // or whatever
header you decide, useful for tagging requests
//...
wrapper.setHeader(header);
-->
<!--<field id="header" type="A" length="4" />-->
<!--this is the response id. in this case, it'll always be ND-->
<!--<field id="command" type="A" length="2" />-->
<!--this is the response code-->
<!--<field id="response" type="N" length="2" />-->


<!--this is the LMK check value-->

<field id="lmk-check-value" type="A" length="16" />

<!--this is the firmware number-->

<field id="firmware-number" type="A" length="9" />

</schema>

I've limited it to these 4 since the others are basically extensions of
the same.

Thanks for all the help.

Mark Salter

unread,
Mar 11, 2011, 9:50:20 AM3/11/11
to jpos-...@googlegroups.com
On 11/03/2011 14:26, 'Dipo Odumosu wrote:
>> I think you were missing the attributes to tell the FSDMsg to treat some
>> fields as keys and this use their content to construct the file name
>> selection?
>>
> No, I don't think so. For example, I haven't modified my hsm-base.xml
> file in over a month. The modifications to hsm-resp-base.xml were to
> deal with the NBO message length and 'tag' headers when using
> channel.getBytes().
Ok, perhaps the thread mix is confusing things (me).

>> I think 'response message id' is clearer
>>
> Well said, sir.

8)

>>>>> Sending: AAAAND007B44AC1DDEE2A94B0007-E000
>>>> Can you correctly/manually parse this response - ever?
>>>>
>>> At the moment, no. Here's the code I'm using:
>> No?
>>
>> What happens, you don't seem to say anywhere?
>>
> Output:
> <fsdmsg schema='file:cfg/hsm-resp-base'>
> response: '��'
> error: '� <--- some other character here, but I can't paste the
> entire thing. EOF, maybe?
> EOF: 'true'
> </fsdmsg>

???


>>> byte [] buffer = ISOUtil.hex2byte("ND007B44AC1DDEE2A94B0007-
>>> E000");

Oh I see it, this String is *not* hex data, you do not want to use hex2byte!

Try:-

byte [] buffer = "ND007B44AC1DDEE2A94B0007-E000".getBytes("UTF-8");


>>>
>>> FSDMsg msg = new FSDMsg("file:cfg/hsm-resp-");
>>>
>>> FSDISOMsg rep = new FSDISOMsg(msg);
>>>
>>> rep.unpack(buffer);
>>>
>>> rep.dump(System.out, "\t");
>>>
>>> I'm ignoring the header (AAAA) since there's no provision for it in
>>> the hsm-resp-base.xml schema. I'm not including the FSD separator in
>>> the string, since it's probably not present in the stream read off the
>>> channel.
>> The header is dealt with by the Channel and is dealt with before
>> unpacking/packing.
>>
> Hence my not including it in the test code above

I know, I was checking you understood 8).

>>> Contents of hsm-resp-base.xml:
>>>
>>> <?xml version='1.0' encoding='UTF-8'?>
>>> <schema>
>>> <!--
>>> <field id='nboheader' type='A' length='2'/>
>>> <field id='isoheader' type='A' length='4'/>
>>> -->
>>> <field id='response' type='A' length='2' key='true'/>

Good I see the key now.

>>> <field id='error' type='A' length='2'/>
>>> </schema>
>>>
>> So...
>>
>> what does your hsm-resp-ND.xml file look like?
>>
> Complete with warts ;)

Ok...

>
> ----- hsm-base.xml -----
> <?xml version='1.0' encoding='utf-8'?>
> <schema>
> <field id='command' type='A' length='2' key='true'/>
> </schema>
>

The above file is not currently used.

> ----- hsm-resp-base.xml (hsm-resp-null.xml is an exact duplicate of this
> file) -----
> <?xml version='1.0' encoding='UTF-8'?>
> <schema>
> <!--
> <field id='nboheader' type='A' length='2'/>
> <field id='isoheader' type='A' length='4'/>
> -->
> <field id='response' type='A' length='2' key='true'/>
> <field id='error' type='A' length='2'/>
> </schema>
>
> ----- hsm-NC.xml -----
> <?xml version='1.0' encoding='UTF-8'?>
> <schema id='NC'>
> </schema>
>
> ----- hsm-resp-ND.xml -----

I'm going to try and insert your sample message into :-

ND007B44AC1DDEE2A94B0007-E000

> <?xml version="1.0" encoding="UTF-8"?>
> <schema id='ND'>
ND
> <!--this is the 2-byte message length header-->
> <!--<field id="messagelength" type="A" length="2" />-->
> <!--this is the 4-byte header you need to prepend the actual data with,
> byte[] header = new byte [] { 0, 0, 0, 0 }; // or whatever
> header you decide, useful for tagging requests
> //...
> wrapper.setHeader(header);
> -->
> <!--<field id="header" type="A" length="4" />-->
> <!--this is the response id. in this case, it'll always be ND-->
> <!--<field id="command" type="A" length="2" />-->
> <!--this is the response code-->
> <!--<field id="response" type="N" length="2" />-->
> <!--this is the LMK check value-->
> <field id="lmk-check-value" type="A" length="16" />

007B44AC1DDEE2A9


> <!--this is the firmware number-->
> <field id="firmware-number" type="A" length="9" />

4B0007-E0
> </schema>
>
So, "00" is left and if you had converted to hex, it would be a 'null'.

Is the length of that last field wrong, or do you have one missing?

--
Mark

Mark Salter

unread,
Mar 11, 2011, 10:17:56 AM3/11/11
to jpos-...@googlegroups.com
PS

On 11/03/2011 14:50, Mark Salter wrote:
> So, "00" is left and if you had converted to hex, it would be a 'null'.

Although other fields would not have aligned.

I think the null was due to the hex2byte...

.. ND just doesn't convert, so not sure what would be produced. As the
very next byte would have been x'00', perhaps this was the null?

--
Mark

'Dipo Odumosu

unread,
Mar 11, 2011, 11:04:07 AM3/11/11
to jpos-...@googlegroups.com
You're right about changing the ISOUtil.hex2byte() call to
getBytes("UTF-8"). Maybe I'm too used to working with ISOUtil.hex2byte()?

I think you'd have to start from the hsm-resp-base.xml (I might be wrong):

<schema>
<field id='response' type='A' length='2' key='true>ND</field>
<field id='error' type='A' length='2'>00</field>
<field id='lmk-check-value' type='A' length='2'>7B44AC1DDEE2A94B</field>
<field id='firmware-number' type='A' length='2'>0007-E000</field>
</schema>

IDEA seems to agree with me:

<fsdmsg schema='file:cfg/hsm-resp-base'>


response: 'ND'
error: '00'
lmk-check-value: '7B44AC1DDEE2A94B'
firmware-number: '0007-E000'
</fsdmsg>

Process finished with exit code 0

Mark Salter

unread,
Mar 11, 2011, 11:27:13 AM3/11/11
to jpos-...@googlegroups.com
On 11/03/2011 16:04, 'Dipo Odumosu wrote:
> You're right about changing the ISOUtil.hex2byte() call to
> getBytes("UTF-8"). Maybe I'm too used to working with ISOUtil.hex2byte()?
>
> I think you'd have to start from the hsm-resp-base.xml (I might be wrong):
No, you are correct...

>
> <schema>
> <field id='response' type='A' length='2' key='true>ND</field>
> <field id='error' type='A' length='2'>00</field>

...I missed this field didn't I!

> <field id='lmk-check-value' type='A' length='2'>7B44AC1DDEE2A94B</field>
> <field id='firmware-number' type='A' length='2'>0007-E000</field>
> </schema>
>
> IDEA seems to agree with me:
>
> <fsdmsg schema='file:cfg/hsm-resp-base'>
> response: 'ND'
> error: '00'
> lmk-check-value: '7B44AC1DDEE2A94B'
> firmware-number: '0007-E000'
> </fsdmsg>
>
> Process finished with exit code 0

Good, so now back to the actual exchange...

... is it all working without 'packager gymnastics'?

--
Mark

'Dipo Odumosu

unread,
Mar 11, 2011, 11:45:40 AM3/11/11
to jpos-...@googlegroups.com

Good, so now back to the actual exchange...

... is it all working without 'packager gymnastics'?

It's working, for controlled definitions of 'working'. Without both hsm-resp-base.xml and hsm-resp-null.xml, I get a RuntimeException.

If hsm-resp-base.xml is not present:


 <fsdmsg schema='file:cfg/hsm-base'>
   command: 'NC'
 </fsdmsg>
Exception in thread "main" org.jpos.iso.ISOException: java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-base.xml not found (java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-base.xml not found)

    at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
    at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
    at com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:90)
    at com.projxchange.sandbox.Program.main(Program.java:24)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Nested:java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-base.xml not found
    at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:602)
    at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:572)

    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:193)
    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:210)
    at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
    at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
    at com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:90)
    at com.projxchange.sandbox.Program.main(Program.java:24)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)

If hsm-base-null.xml is not present:


<fsdmsg schema='file:cfg/hsm-base'>
   command: 'NC'
 </fsdmsg>
Exception in thread "main" org.jpos.iso.ISOException: java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not found (java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not found)
    at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
    at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
    at com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:90)
    at com.projxchange.sandbox.Program.main(Program.java:24)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
Nested:java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-null.xml not found
    at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:602)
    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:433)
    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:193)
    at org.jpos.util.FSDMsg.unpack(FSDMsg.java:210)
    at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
    at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
    at com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:90)
    at com.projxchange.sandbox.Program.main(Program.java:24)

    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)

If both files are present:


 <fsdmsg schema='file:cfg/hsm-base'>
   command: 'NC'
 </fsdmsg>
 <fsdmsg schema='file:cfg/hsm-resp-base'>
   response: 'ND'
   error: '00'
   lmk-check-value: '7B44AC1DDEE2A94B'
   firmware-number: '0007-E000'
 </fsdmsg>

Here's the code in main():


        ThalesAdaptor adaptor = new ThalesAdaptor("127.0.0.1", 9998);
        adaptor.connect();

        FSDMsg req = adaptor.diagnostics();

        req.dump(System.out, " ");

        FSDMsg resp = adaptor.command(req);

        resp.dump(System.out, " ");

No packager gymnastics (whew!). ThalesAdaptor uses an FSDChannel subclass described as I posted earlier.


Mark Salter

unread,
Mar 11, 2011, 12:22:50 PM3/11/11
to jpos-...@googlegroups.com
On 11/03/2011 16:45, 'Dipo Odumosu wrote:
>
>> Good, so now back to the actual exchange...
>>
>> ... is it all working without 'packager gymnastics'?
>>
> It's working, for controlled definitions of 'working'. Without /*both*/

> hsm-resp-base.xml and hsm-resp-null.xml, I get a RuntimeException.
>
We don't need the special hsm-resp-* anymore, I thought/hoped all
messages would use the same single schema?

The 'response code' field would need to move to the lower level file (ND
etc) file though - I assume the response code is not on a request message.

> If hsm-resp-base.xml is not present:

How are you triggering the use of the *resp* schema, I'd hoped to remove
all trace of it - where is the use of this schema coming from?

Does an NC request message have a response code field?

Can you combine request and response into a single schema - I guess (I
didn't go looking) you have some code to set the response schema - that
we could also remove, if the schema can be combined?

I still don't see where the 'null' is coming from, a network trace might
be usefully done, just so you can check it is not there somewhere,
debugging to see would be better.


--
Mark

Mark Salter

unread,
Mar 12, 2011, 6:00:13 AM3/12/11
to jpos-...@googlegroups.com
On 11/03/2011 12:29, 'Dipo Odumosu wrote:
> Also, a particular operation I was carrying out failed with an error,
> and I noticed that case (or in any instance where the unpacking was
> producing nonsense data - due to a misconfiguration), a 'magic field' of
> sorts with the name EOF was set to true (or 'true', I suppose).
I just wanted to point out that the problem here was the code was using

hex2byte([hsm_command])

which had the effect of breaking the data, but also dividing the length
of the String in half, thus the unpack was hitting the end of the byte[]
before reaching the end of the schema; resulting in an EOF.

--
Mark

Dipo Odumosu

unread,
Mar 12, 2011, 7:40:12 AM3/12/11
to jpos-...@googlegroups.com
We don't need the special hsm-resp-* anymore, I thought/hoped all
messages would use the same single schema?
They can.  
 
The 'response code' field would need to move to the lower level file (ND
etc) file though - I assume the response code is not on a request message.

> If hsm-resp-base.xml is not present:
How are you triggering the use of the *resp* schema, I'd hoped to remove
all trace of it - where is the use of this schema coming from?
The -resp schema is used in the overridden createMsg() of the channel. I found createMsg() is used for creating response messages, which is why I implemented it that way.

Does an NC request message have a response code field?
No. Requests don't have a response code field. 

Can you combine request and response into a single schema - I guess (I
didn't go looking) you have some code to set the response schema - that
 we could also remove, if the schema can be combined?
Sure, they can be.

Dipo Odumosu

unread,
Mar 12, 2011, 7:41:46 AM3/12/11
to jpos-...@googlegroups.com
If there's an error, the simulator will not return any data for the fields after the response code, resulting in EOF as well.

'Dipo Odumosu

unread,
Mar 15, 2011, 12:19:08 PM3/15/11
to jpos-...@googlegroups.com
Hello,
I ran a couple of tests today, and even when I used the same base file
for sending and receiving responses, I still needed a -null.xml file (a
copy of my base schema) present to run exception-free. I was inclined to
assume the issue lay with the simulator, so I ran a quick test to dump
the data received on the channel using ISOUtil.hexdump(). Here's my output:


0000 00 21 41 41 41 41 4E 44 30 30 37 42 34 34 41 43 .!AAAAND007B44AC
0010 31 44 44 45 45 32 41 39 34 42 30 30 30 37 2D 45 1DDEE2A94B0007-E
0020 30 30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 000.............

If we assume the first two bytes are the NBO message length, then I
can't tell why I need a -null.xml file present for the unpacking to work
properly. Still, until I figure it out, I suppose I'll take it as a
workaround.

Regards.

Mark Salter

unread,
Mar 15, 2011, 1:58:20 PM3/15/11
to jpos-...@googlegroups.com
On 15/03/2011 16:19, 'Dipo Odumosu wrote:
> 0000 00 21 41 41 41 41 4E 44 30 30 37 42 34 34 41 43 .!AAAAND007B44AC
> 0010 31 44 44 45 45 32 41 39 34 42 30 30 30 37 2D 45 1DDEE2A94B0007-E
> 0020 30 30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 000.............
>
> If we assume the first two bytes are the NBO message length, then I
> can't tell why I need a -null.xml file present for the unpacking to work
> properly. Still, until I figure it out, I suppose I'll take it as a
> workaround.

Why do you need to assume ?

Check the manual for the length format and structure.

*If* the length is x'0021', then what are all those extra x'00''s, they
may well fall into the 'next' message causing the null issue.

Perhaps the length *doesn't* include the header or other (wrong) strange
behaviour.

Please may I also check that *non* of this work will end up in
production using live keys and data?

Nothing you have discussed so far should *ever* reach a production
environment, unless you really are (trying) to talk to a real HSM of
course. Some of your most recent comments make me worried and I wish I
had thought to check your intentions before.

--
Mark

'Dipo Odumosu

unread,
Mar 15, 2011, 2:04:49 PM3/15/11
to jpos-...@googlegroups.com
Hello,

> Why do you need to assume ? Check the manual for the length format and structure.
It is, in fact. The extra 0x00's are there because the code is like this:
byte [] buffer = new buffer[1024];

channel.getBytes(buffer);

System.out.println(ISOUtil.hexdump(buffer));


> Please may I also check that *non* of this work will end up in
> production using live keys and data?
>
> Nothing you have discussed so far should *ever* reach a production
> environment, unless you really are (trying) to talk to a real HSM of
> course. Some of your most recent comments make me worried and I wish I
> had thought to check your intentions before.
>

Specifically, what has you worried? As I said, I'm talking to a Thales
simulator, in the hope that sometime before we go live, I will actually
get my grubby paws on a real HSM and won't have to make too many code
changes, and it seems the higher-ups are sold on Thales equipment. In
this case, I'll assume the statement "if you're confident, you're
definitely ignorant" applies here.

Thanks, and regards.

Mark Salter

unread,
Mar 15, 2011, 2:19:05 PM3/15/11
to jpos-...@googlegroups.com
On 15/03/2011 18:04, 'Dipo Odumosu wrote:
>> Why do you need to assume ? Check the manual for the length format and
>> structure.
> It is, in fact. The extra 0x00's are there because the code is like this:
> byte [] buffer = new buffer[1024];
>
> channel.getBytes(buffer);
>
> System.out.println(ISOUtil.hexdump(buffer));
Ok.

So the zeros are only there as fill, the length is *always* used going
forward?

What does the length describe please? The message bytes that follow
(the length), or the bytes of the whole message including the 2 byte length?


>> Please may I also check that *non* of this work will end up in
>> production using live keys and data?
>>
>> Nothing you have discussed so far should *ever* reach a production
>> environment, unless you really are (trying) to talk to a real HSM of
>> course. Some of your most recent comments make me worried and I wish I
>> had thought to check your intentions before.
>>
> Specifically, what has you worried?

When you said - in the other thread - :-

"
In the test I ran with a third party, I was given a clear ZMK (when I
was slightly younger and significantly more ignorant), which I worked
with using the JCE to decrypt the PWKs I requested for when connecting
to the remote party.
"

"A clear ZMK" - alarm bells rang in my head (only).

I should have noticed the 'test' at the start of your sentence, but didn't.

I have expressed my relief already, but again here for completeness :-

Phew, test keys now and a real HSM later.

> As I said, I'm talking to a Thales
> simulator, in the hope that sometime before we go live, I will actually
> get my grubby paws on a real HSM and won't have to make too many code
> changes, and it seems the higher-ups are sold on Thales equipment. In
> this case, I'll assume the statement "if you're confident, you're
> definitely ignorant" applies here.

Others currently and will not have such good intentions...

... 8)
I have no intention of helping *anyone* produce a 'working' software HSM
for production use. I feel that would make me as irresponsible as them,
and I have no intention of that happening.


--
Mark

Mark Salter

unread,
Mar 15, 2011, 2:28:28 PM3/15/11
to jpos-...@googlegroups.com
On 15/03/2011 18:19, Mark Salter wrote:
> So the zeros are only there as fill, the length is *always* used going
> forward?
Don;t forget the length (and header) would already be stripped before
unpack touches anything.

Perhaps it is time to check some code again?

--
Mark

'Dipo Odumosu

unread,
Mar 15, 2011, 3:15:02 PM3/15/11
to jpos-...@googlegroups.com
Yes, they would be stripped off. That means the first 6 bytes won't be
in the message. The nulls at the end of the message are because I
created a 1k buffer to hold the bytes received on the channel. normally,
the stream would end at the last 0 (character, not byte value)

'Dipo Odumosu

unread,
Mar 15, 2011, 3:17:32 PM3/15/11
to jpos-...@googlegroups.com

> So the zeros are only there as fill, the length is *always* used going
> forward?
>
> What does the length describe please? The message bytes that follow
> (the length), or the bytes of the whole message including the 2 byte length?
>
>
the message length in nbo order (divisor in first byte, remainder in second)
Touche. I've no intention of using a soft SM either.

Mark Salter

unread,
Mar 15, 2011, 6:29:39 PM3/15/11
to jpos-...@googlegroups.com
On 15/03/2011 19:15, 'Dipo Odumosu wrote:
> Yes, they would be stripped off. That means the first 6 bytes won't be
> in the message. The nulls at the end of the message are because I
> created a 1k buffer to hold the bytes received on the channel. normally,
> the stream would end at the last 0 (character, not byte value)
>
So, may I just check how long the byte[] that goes into the unpack
method is please?

--
Mark

Mark Salter

unread,
Mar 15, 2011, 7:53:59 PM3/15/11
to jpos-...@googlegroups.com
On 15/03/2011 19:17, 'Dipo Odumosu wrote:

>> What does the length describe please? The message bytes that follow
>> (the length), or the bytes of the whole message including the 2 byte
>> length?
>>
>>
> the message length in nbo order (divisor in first byte, remainder in
> second)

And is the message length itself included or excluded in this count?

This may be a dead end check, but I am just wondering 8).


>> I have no intention of helping *anyone* produce a 'working' software HSM
>> for production use. I feel that would make me as irresponsible as them,
>> and I have no intention of that happening.
>>
>>
> Touche. I've no intention of using a soft SM either.
>

Great, I wish everyone had the same intent...

... I fear *others* have different intentions.

8(

--
Mark

'Dipo Odumosu

unread,
Mar 16, 2011, 1:54:22 AM3/16/11
to jpos-...@googlegroups.com
hello,

> So, may I just check how long the byte[] that goes into the unpack
> method is please?
>
The length is the length of the actual message, as seen in this code I
tried to use to communicate with the simulator shows:

Socket socket = new Socket("127.0.0.1", 9998);
InputStream in = socket.getInputStream();
OutputStream out = socket.getOutputStream();
BufferedOutputStream buffered = new BufferedOutputStream(out,
1024);

// perform a diagnostics check
// send header
byte [] command = "0000NC".getBytes("UTF-8"); // <-- doesn't
work without the 0000
byte [] inBuffer = new byte[2];
inBuffer[0] = (byte) (command.length / 256);
inBuffer[1] = (byte) (command.length % 256);
//System.out.println((int) inBuffer[1]);
buffered.write(inBuffer);
buffered.write(command);
buffered.flush();

// read response
BufferedReader reader = new BufferedReader(new
InputStreamReader(in));

System.out.println("Response is: " + reader.readLine());

The simulator returns a response (which I'm probably not dealing with
properly, as my application doesn't display anything) and the output can
be seen in the simulator's interface.

Regards

sali.lala

unread,
Apr 7, 2011, 8:57:40 AM4/7/11
to jPOS Users
please can you tell me how did you solve the problem of hsm-resp-
null.xml error without adding the xml file?

On 7 mar, 10:57, 'Dipo Odumosu <dipo.odum...@projxchange.com> wrote:
> Hello again,
>
> I've been working on the Thales simulator with FSDMsg (although I've not
> yet switched my code to using the MUX as I want to be certain I
> understand the points raised by Andy, David and Mark previously), and
> I've run into issues with getting RuntimeException being thrown on
> message unpacking.
>
> However, if I add ahsm-resp-null.xmlfile with the same contents as my
> hsm-resp-base.xmlfile, I don't run into issues.
>
> I want to apologize for dumping the entire listing below. I thought it
> might be easier to see the code in one page than upload an archive with
> the sources. If the latter is more acceptable, please let me know.
> thanks for your help in advance.
>
> Here are my source files:
>
> ----- Program.java -----
>
> public class Program {
>      public static void main(String [] args) throws Exception {
>          ThalesAdaptor adaptor = new ThalesAdaptor("127.0.0.1", 9998);
>          adaptor.connect();
>
>          FSDMsg req = adaptor.diagnostics();
>
>          req.dump(System.out, " ");
>
>          FSDMsg resp = adaptor.command(req);
>
>          resp.dump(System.out, " ");
>
>          req = adaptor.generateDoubleLengthKey();
>
>          req.dump(System.out, " ");
>
>          resp = adaptor.command(req);
>
>          resp.dump(System.out, " ");
>
>          // this returns a ZMK parity error (code 10)
>          req =
> adaptor.importDoubleLengthKey("C15745088502B0D3A2EF68435EE6D075",
> "5C2F0A5F099655FB9ED9905B5C6743CE");
>
>          req.dump(System.out, " ");
>
>          resp = adaptor.command(req);
>
>          resp.dump(System.out, " ");
>      }
>
> }
>
> ----- ThalesAdaptor.java -----
> public class ThalesAdaptor {
>      private ThalesChannel channel;
>
>      public ThalesAdaptor(String host, int port) {
>          channel = new ThalesChannel();
>          channel.setHost(host, port);
>          channel.setPackager(new DummyPackager());
>      }
>
>      public void connect() throws Exception {
>          channel.connect();
>      }
>
>      private FSDMsg createRequest(String command) {
>          FSDMsg req = new FSDMsg("file:cfg/hsm-");
>
>          if (command != null)
>              req.set("command", command);
>
>          return req;
>      }
>
>      private FSDMsg createResponse(String response) {
>          FSDMsg resp = new FSDMsg("file:cfg/hsm-resp-");
>
>          if (response != null)
>              resp.set("response", response);
>
>          return resp;
>      }
>
>      public FSDMsg diagnostics() {
>          return createRequest("NC");
>      }
>
>      public FSDMsg generateDoubleLengthKey() {
>          FSDMsg req = createRequest("A0");
>          req.set("mode", "0");
>          req.set("key-type", "001");
>          req.set("key-scheme-lmk", "U");
>
>          return req;
>      }
>
>      public FSDMsg importDoubleLengthKey(String zmk, String key) {
>          FSDMsg req = createRequest("A6");
>          req.set("key-type", "001");
>          req.set("zmk", "X" + zmk);
>          req.set("key-under-zmk", "X" + key);
>          req.set("key-scheme", "X");
>
>          return req;
>      }
>
>      public FSDMsg command(FSDMsg request) throws Exception {
>          StringBuffer sbuffer = new StringBuffer(request.get("command"));
>          sbuffer.setCharAt(1, (char) (sbuffer.charAt(1) + 1));
>          FSDMsg resp = createResponse(sbuffer.toString());
>
>          FSDISOMsg msg = new FSDISOMsg(request);
>          msg.setHeader(new byte[] { (byte) 1, (byte) 2, (byte) 3, (byte)
> 4});
>
>          channel.setBasePath(resp.getBasePath());
>          channel.setSchema(resp.getBaseSchema());
>          channel.send(msg);
>
>          FSDISOMsg response = (FSDISOMsg) channel.receive();
>
>          resp.merge(response.getFSDMsg());
>
>          return resp;
>      }
>
> }
>
> ----- ThalesChannel.java -----
>
> public class ThalesChannel extends FSDChannel {
>      String basePath;
>      String schema;
>
>      public String getBasePath() {
>          return basePath;
>      }
>
>      public void setBasePath(String basePath) {
>          this.basePath = basePath;
>      }
>
>      public String getSchema() {
>          return schema;
>      }
>
>      public void setSchema(String schema) {
>          this.schema = schema;
>      }
>
>      @Override
>      public ISOMsg createMsg() {
>          if (basePath != null && schema != null)
>              return new FSDISOMsg(new FSDMsg(basePath, schema));
>
>          if (basePath != null)
>              return new FSDISOMsg(new FSDMsg(basePath));
>
>          return new FSDISOMsg();        // no-op. will generate
> exceptions if used.
>      }
>
> }
>
> ----- hsm-base.xml-----
>
> <?xmlversion='1.0' encoding='utf-8'?>
> <schema>
> <field id='command' type='A' length='2' key='true'/>
> </schema>
>
> ----- hsm-resp-base.xml-----
>
> <?xmlversion='1.0' encoding='UTF-8'?>
> <schema>
> <field id='response' type='A' length='2' key='true'/>
> <field id='error' type='A' length='2'/>
> </schema>
>
> -----hsm-resp-null.xmlhas the same contents as the above -----
>
> ----- hsm-NC.xml-----
> <?xmlversion='1.0' encoding='UTF-8'?>
> <schema id='NC'>
> </schema>
>
> ----- hsm-resp-ND-xml-----
> <?xmlversion="1.0" encoding="UTF-8"?>
> <schema id='ND'>
> <!--this is the LMK check value-->
> <field id="lmk-check-value" type="A" length="16" />
> <!--this is the firmware number-->
> <field id="firmware-number" type="A" length="9" />
> </schema>
>
> ----- hsm-A0.xml-----
> <?xmlversion='1.0' encoding='utf-8'?>
> <schema id='A0'>
> <field id='mode' type='N' length='1'/>
> <field id='key-type' type='A' length='3'/>
> <field id='key-scheme-lmk' type='A' length='1'/>
> </schema>
>
> ----- hsm-resp-A1.xml-----
> <?xmlversion='1.0' encoding='utf-8'?>
> <schema id='A1'>
> <field id='key-lmk' type='A' length='33'/>
> <field id='check-value' type='A' length='6'/>
> </schema>
>
> ----- hsm-A6.xml-----
> <?xmlversion='1.0' encoding='UTF-8'?>
> <schema id='A6'>
> <field id='key-type' type='A' length='3'/>
> <field id='zmk' type='A' length='33'/>
> <field id='key-under-zmk' type='A' length='33'/>
> <field id='key-scheme' type='A' length='1'/>
> </schema>
>
> ----- hsm-A7.xml(i realized this file was incorrectly named, but
> renaming it to hsm-resp-A7.xmldidn't affect anything) -----
> <?xmlversion='1.0' encoding='UTF-8'?>
> <schema id='A7'>
> <field id='key-under-lmk' type='AFS' length='33'/>
> <field id='check-value' type='AFS' length='6'/>
> </schema>
>
> ----- Output withhsm-resp-null.xmlpresent: -----
> <fsdmsg schema='file:cfg/hsm-base'>
>     command: 'NC'
> </fsdmsg>
> <fsdmsg schema='file:cfg/hsm-resp-base'>
>     response: 'ND'
>     error: '00'
>     lmk-check-value: '7B44AC1DDEE2A94B'
>     firmware-number: '0007-E000'
> </fsdmsg>
> <fsdmsg schema='file:cfg/hsm-base'>
>     command: 'A0'
>     mode: '0'
>     key-type: '001'
>     key-scheme-lmk: 'U'
> </fsdmsg>
> <fsdmsg schema='file:cfg/hsm-resp-base'>
>     response: 'A1'
>     error: '00'
>     key-lmk: 'U94D098BBE54537CC78BE9ED1DFC46F0A'
>     check-value: '02686B'
> </fsdmsg>
> <fsdmsg schema='file:cfg/hsm-base'>
>     command: 'A6'
>     key-type: '001'
>     zmk: 'XC15745088502B0D3A2EF68435EE6D075'
>     key-under-zmk: 'X5C2F0A5F099655FB9ED9905B5C6743CE'
>     key-scheme: 'X'
> </fsdmsg>
> <fsdmsg schema='file:cfg/hsm-resp-base'>
>     response: 'A7'
>     error: '10'
>     EOF: 'true'
> </fsdmsg>
>
> ----- Output withhsm-resp-null.xmlabsent ----
> <fsdmsg schema='file:cfg/hsm-base'>
>     command: 'NC'
> </fsdmsg>
> Exception in thread "main" org.jpos.iso.ISOException:
> java.lang.RuntimeException: C:\sandbox\HSMDemo\cfg\hsm-resp-null.xmlnot
> found (java.lang.RuntimeException:
> C:\sandbox\HSMDemo\cfg\hsm-resp-null.xmlnot found)
>      at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:52)
>      at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
>      at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
>      at
> com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:96)
>      at com.projxchange.sandbox.Program.main(Program.java:22)
>      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>      at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>      at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>      at java.lang.reflect.Method.invoke(Method.java:597)
>      at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115)
> Nested:java.lang.RuntimeException:
> C:\sandbox\HSMDemo\cfg\hsm-resp-null.xmlnot found
>      at org.jpos.util.FSDMsg.getSchema(FSDMsg.java:602)
>      at org.jpos.util.FSDMsg.unpack(FSDMsg.java:433)
>      at org.jpos.util.FSDMsg.unpack(FSDMsg.java:193)
>      at org.jpos.util.FSDMsg.unpack(FSDMsg.java:210)
>      at org.jpos.iso.FSDISOMsg.unpack(FSDISOMsg.java:49)
>      at org.jpos.iso.BaseChannel.unpack(BaseChannel.java:902)
>      at org.jpos.iso.BaseChannel.receive(BaseChannel.java:670)
>      at
> com.projxchange.sandbox.security.thalesadaptor.ThalesAdaptor.command(ThalesAdaptor.java:96)
>      at com.projxchange.sandbox.Program.main(Program.java:22)

'Dipo Odumosu

unread,
Apr 7, 2011, 9:43:32 AM4/7/11
to jpos-...@googlegroups.com

> please can you tell me how did you solve the problem of hsm-resp-
> null.xml error without adding the xml file?
>
I didn't. I had to duplicate hsm-resp-base.xml to hsm-resp-null.xml

Regards.

Mark Salter

unread,
Apr 7, 2011, 9:53:28 AM4/7/11
to jpos-...@googlegroups.com
It did sort of stop being fixed and investigate, I do still think it is
data related, but would need more information to dig into.

If you can share more Dido, that would be good.

Why do you ask Sali, are you having a similar problem?

--
Mark

sali lala

unread,
Apr 7, 2011, 9:51:01 AM4/7/11
to jpos-...@googlegroups.com
it  isn't a buge in the jPos source?

2011/4/7 'Dipo Odumosu <dipo.o...@projxchange.com>


--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org

You received this message because you are subscribed to the  "jPOS Users" group.
Please see http://jpos.org/wiki/JPOS_Mailing_List_Readme_first
To post to this group, send email to jpos-...@googlegroups.com
To unsubscribe, send email to jpos-users+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/jpos-users

Mark Salter

unread,
Apr 7, 2011, 9:57:38 AM4/7/11
to jpos-...@googlegroups.com
On 07/04/2011 14:51, sali lala wrote:
> it isn't a buge in the jPos source?
I don't think so, not at this time?

Are you seeing the same thing?

--
Mark

sali lala

unread,
Apr 7, 2011, 9:55:49 AM4/7/11
to jpos-...@googlegroups.com
yes i have the same probleme

2011/4/7 Mark Salter <marks...@talktalk.net>

Mark Salter

unread,
Apr 7, 2011, 10:00:42 AM4/7/11
to jpos-...@googlegroups.com
On 07/04/2011 14:55, sali lala wrote:
> yes i have the same probleme

Can you present your config and detail of the problem?

Did you copy the setup and config that has been presented here - that
might be the problem, assuming you have a different HSM and network and
the like?

I would start a new thread, this one was a mix.

--
Mark

Reply all
Reply to author
Forward
0 new messages