FSDMsg and variable length fields

[Geoff Chiang]

Hi all,

I'm trying to port our Thales adapter to use FSDMsg, more as a
learning exercise than anything else. I'm probably missing something
fairly obvious, but I'm running into a problem with variable length
fields. The "GW" Thales command (generate a MAC) has a fixed length
"data length" field followed by a variable length "data to MAC"
field. How do you reflect this in the FSDMsg schema? Obviously, the
schema field definitions have a length, so what sort of value would
one use for a variable length field?

I note the small amount of discussion on the subject in this post:
http://groups.google.com/group/jpos-users/browse_thread/thread/6fbb995354d9848a/c010c6958e1f897e.
Unfortunately, I'm still in the dark as to how to work with variable
length fields.

Thanks in advance for any pointers.

Geoff

[Mark Salter]

Geoff Chiang wrote:

> I'm trying to port our Thales adapter to use FSDMsg, more as a
> learning exercise than anything else.

> The "GW" Thales command (generate a MAC) has a fixed length "data
> length" field followed by a variable length "data to MAC" field.

This is interesting as the data is variable length (multiples of 8
bytes), but the terminator that follows this field is only present if
your are specifying a 'message trailer'.

So we need a variable length field with no terminator byte.

> How do you reflect this in the FSDMsg schema? Obviously, the schema
> field definitions have a length, so what sort of value would one use
> for a variable length field?

If you were specifying a message trailer then you would need a field
that terminated itself with a x'19' byte, which will need a new separator.

At the moment I don't think it is possible to achieve what is needed.

May I suggest adding a dummy separator type that will prevent padding,
that packs to nothing?

Like:-

<schema id='fsdmsg-test'>
<field id="length" type="N" length="4" />
<field id="vardata" type="ADS" length="32" />
</schema>

Where the DS type is a dummy separator and would satisfy the test case:-

FSDMsg m = new FSDMsg ("file:..[above]");
macData = "12345678123456781234567812345678";
m.set ("length", "32");
m.set ("vardata", macData);
assertEquals("Dummy separator long
data","003212345678123456781234567812345678",m.pack());

I have the code ready to commit, but would like to check no one sees a
way to support this asis.

--
Mark

[Alejandro Revilla]

Looks good to me.

[Geoff Chiang]

Hi Mark,

Thanks for the reply. I'm probably not adding anything, but I just
wanted to confirm what you have already said. We have never needed to
use the optional terminator and message trailer after the variable
length field before. Our currently deployed implementation (based on
ISOMsg) uses the LLLL_whatever field type for the variable length
field, so the length is figured out dynamically and the terminator has
never been required.

Geoff

[Mark Salter]

Mark Salter wrote:
> I have the code ready to commit, but would like to check no one sees a
> way to support this asis.
>

Committed r2646

--
Mark

[Geoff Chiang]

Mark, thanks for the very quick code update. I've come into the
office this morning and given it a go, and I've gotten a result which
may or may not have been anticipated. As you noted, the variable
length field can have a length of any multiple of 8. For the field
length in the message schema, I put an arbitrary large number. When
the message is packed, the two byte message length header shows the
maximum length of the message (i.e. the sum of all the lengths in the
schema file). The HSM then complains when it receives the message, as
it is given a message length much longer than the length of the actual
message.

Is there some way around this?

One other question that is tangentially related: if you're talking to
a Thales HSM through a MUX, what MUX key fields do you use? We
currently place a counter in the four byte header and override one of
the MUX implementations to read the header data to match requests to
responses, but I'm wondering if there is a better way?

Geoff

[Mark Salter]

Geoff Chiang wrote:
> For the field length in the message schema, I put an arbitrary large
> number. When the message is packed, the two byte message length
> header shows the maximum length of the message (i.e. the sum of all

> the lengths in the schema file).I initially thought the channel might be the place to start looking, but
I think the m.pack() method must be returning too many bytes.

> The HSM then complains when it
> receives the message, as it is given a message length much longer
> than the length of the actual message.

And rightly so.

>
> Is there some way around this?

I think may be some code to fix that does not currently know about the
dummy separator and that the whole length is not used, I will take a
look later and come back.

May I ask to see your schema - or at least the part including the "data
to be MAC'd" field?

>
> One other question that is tangentially related: if you're talking to
> a Thales HSM through a MUX, what MUX key fields do you use? We
> currently place a counter in the four byte header and override one of
> the MUX implementations to read the header data to match requests to
> responses, but I'm wondering if there is a better way?

I'm sorry to say I don't connect to any HSMs using jPOS or even java.

The header sounds ok, but remember the 'message trailer' that was
mentioned earlier?
I think I would seek to use that field - as long as the HSM echos it
back! There may be 'issues' as it's position probably varies across
message types? In the MUX, are the fields referenced by number
(position) or by name?

--
Mark

[Mark Salter]

Mark Salter wrote:
> May I ask to see your schema - or at least the part including the
> "data to be MAC'd" field?
>

Just so this stands out, may I also clarify what I would like to see?

I am interested in the definition for the field that holds the data
requiring MACing.

Did you chnage you field definition to use the new type="ADS" on your
data field like :-

<schema id='fsdmsg-test'>
.
.
<field id="dataForMACing" type="ADS" length="32768" />
.
.
</schema>

I am hopeful there is a typo or something stopping this working, it
works for me you see.

--
Mark

[Geoff Chiang]

I won't be in the office until Monday, so I don't have the exact
schema, but the relevant lines are as follows:

<schema id='GW'>
.
<field id="data-length" type="A" length="4" />
<field id="data" type="BDS" length="2048" />
.
</schema>

When the message is packed, this bit works fine. I've fired up
Wireshark to confirm that this part of the message is identical to our
working implementation (aside from a large number of trailing zero
bytes, but I don't believe they should be relevant). It's the two
byte length header on the entire message which seems to ignore the
actual length of the last field.

Geoff

[Geoff Chiang]

After writing my last post, it occurred to me what's going on here.
Notice that my field is defined as a binary field. If we have 17
bytes of data, we round it up to 24 (multiple of 8) and right pad with
zero bytes. But if my field is defined as "BDS" with length of say,
32, then placing a the 24 bytes in there will left pad with 8 zero
bytes! If this is sent to the HSM, then the length field says to read
the first 24 bytes of the 32 byte field, which would include the 8
zero bytes and the first 16 bytes of the data, which is obviously
incorrect.

So in the test case that I was basing my comments on, I was creating a
32 byte array of zero bytes and copying my data into the first 24
bytes, effectively right padding (I checked this in Wireshark). But
the consequence which should have been obvious to me was that I was
affecting the length of not only the variable length field, but also
of the message in its entirety. I assume, Mark, that your change
would have been putting the dummy field separator at the end of the 32
bytes in this case.

OK, moving on then, is there a way to make this work with binary
fields?

Geoff

[Mark Salter]

Geoff Chiang wrote:
> OK, moving on then, is there a way to make this work with binary
> fields?

Yes, committed r2647.

My test cases were not using Binary type fields, instead I had assumed
alphanumeric data (Character hex in fact). Checking the spec the data
into the transaction is binary.

This is why it did not work, and should work for you now.

--
Mark

[Geoff Chiang]

Thanks Mark. I'll give it a go on Monday.

Geoff

[Geoff Chiang]

Mark, I've confirmed that your last commit works perfectly with the
binary variable length field. Thanks again for all your help.

Geoff

[z!†™]

guys..

how come i can't see the separator in my FSDMsg?

i used a sniffer to see what's being passed to the server..
i don't see the separator..

[Mark Salter]

z!†™ wrote:
> how come i can't see the separator in my FSDMsg?

What do your configuration files look like?

Without this detail we are blind to your issue?

--
Mark

[z!†™]

i simpLy want to know if
in FSDMsg the separator that you set is actuaLLy inserted after you
"fsd.set" the fieLds?

my messages are sent and responded just fine..

[Anthony Schexnaildre]

Being that jPOS is open source why not just take a look at the method?

-Anthony

[David Bergert]

Just to be extra helpful:

jpos6\modules\jpos\src\org\jpos\util\FSDMsg.java


here is FSD.set method()
....
public void set (String name, String value) {
if (value != null)
fields.put (name, value);
else
fields.remove (name);
}
....
hmm, fields just looks like a Map



and here is a small snippet of the FSD.pack method()

.....
if (isSeparated(type)) {
char c = getSeparator(type);
if (c > 0)
sb.append(c);
}
.....
hmm, looks like the separators are being appending to a string buffer ?


Now you try :)


--Dave
http://www.paymentsystemsblog.com

[Mark Salter]

z!†™ wrote:
>
> i simpLy want to know if
> in FSDMsg the separator that you set is actuaLLy inserted after you
> "fsd.set" the fieLds?

The separators will not be placed in the message until the pack method
is called by the Channel.

This assumes that you are using field definitions that include an
indicator of the separator required. You say 'separator that you set'
rather than the separator that is defined, so...

...You could be rolling your own separators (by using distinct fields)
in which case the behaviour is different, and unless the field is set it
will have no content on pack.

If you had supplied even a subset of the schema you are using as I
requested, we might now have a better idea or been able to advise more
accurately.

FSDMsgs and their derivatives are working for many people. If they
didn't work I think those people would be asking questions here. This
therefore suggests to me that either your view of the flow is incorrect
or you do not have the correct schema.

Our answers could have been better perhaps, but your question and it's
content set the stage.

Take a(nother) read of the very useful and often referenced :-

http://www.catb.org/~esr/faqs/smart-questions.html

It may help us all in the future.

>
> my messages are sent and responded just fine..

So if your message exchanges are working, what caused you to look more
closely and worry that separators - I assume the target host is
expecting and parsing - are somehow absent?

--
Mark