Connection refused followed by multiple timeout until server restart

[Anirudh Singh]

Hi,
I am facing a peer disconnection / connection refused followed by multiple timeout. Until i restart the server, it does not work.
The packager at client/server is absolute duplicate.

Everything works fine until i received this :
From client to server :

<log realm="channel/XXXXXXXXX" at="Mon Oct 17 22:23:10 XXX 2016.493" lifespan="201400ms">
  <receive>
    <isomsg direction="incoming">
      <!-- org.jpos.iso.packager.GenericPackager[cfg/packager/iso93ascii.xml] -->
      <field id="0" value="1820"/>
      <field id="7" value="1017222310"/>
      <field id="11" value="003240"/>
      <field id="12" value="161017222310"/>
      <field id="24" value="831"/>
      <field id="42" value="111111111111111"/>
    </isomsg>
  </receive>
</log>


At the client side after the above :

<log realm="channel/XXXXXXXXXXX" at="Mon Oct 17 22:23:33 XXX 2016.706" lifespan="23151ms">
  <receive>
    <peer-disconnect/>
  </receive>
</log>
<log realm="org.jpos.q2.iso.ChannelAdaptor" at="Mon Oct 17 22:23:33 XXX2016.722">
  <warn>
    channel-receiver-XXX_OUT
    <exception name="null">
    java.io.EOFException
    at java.io.DataInputStream.readFully(DataInputStream.java:197)
    at org.jpos.iso.channel.ASCIIChannel.getMessageLength(ASCIIChannel.java:93)
    at org.jpos.iso.BaseChannel.receive(BaseChannel.java:699)
    at org.jpos.q2.iso.ChannelAdaptor$Receiver.run(ChannelAdaptor.java:317)
    at java.lang.Thread.run(Thread.java:745)
    </exception>
  </warn>
</log>
<log realm="channel/XXXXXXXXX" at="Mon Oct 17 22:23:35 XXX2016.937" lifespan="1014ms">
  <connect>
    XXXXXXXXXXX:XXXX
      Connection refused: connect
    Unable to connect
  </connect>


The below is never received at client side :

<log realm="channel/XXXXXXXXX" at="Mon Oct 17 22:23:10 XXX 2016.493">
  <send>
    <isomsg direction="outgoing">
      <!-- org.jpos.iso.packager.GenericPackager[cfg/packager/iso93ascii.xml] -->
      <field id="0" value="1830"/>
      <field id="7" value="1017192310"/>
      <field id="11" value="003240"/>
      <field id="12" value="161017222310"/>
      <field id="39" value="800"/>
      <field id="42" value="111111111111111"/>
    </isomsg>
  </send>
</log>

please let me know for any suggestions. I use 1 server with 3 client/fep

[chhil]

What version of jpos are you on?

Thats the only way I can validate the line numbers in the exceptions against the code in github.

-chhil

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
 
Join us in IRC at http://webchat.freenode.net/?channels=jpos
 
You received this message because you are subscribed to the "jPOS Users" group.
Please see http://jpos.org/wiki/JPOS_Mailing_List_Readme_first
To post to this group, send email to jpos-...@googlegroups.com
To unsubscribe, send email to jpos-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/025059b8-b562-45c3-a521-cc26517c9288%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Anirudh Singh]

Thanks for the response.

jPos : 1.9.8
jPosEE : 2.0.4

To unsubscribe, send email to jpos-users+...@googlegroups.com

For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.

[chhil]

The client exception is thrown at 

https://github.com/jpos/jPOS/blob/v1_9_8/jpos/src/main/java/org/jpos/iso/channel/ASCIIChannel.java#L93

The client gets disconnected and the server disconnected it (peer disconnect).

What does your server deploy file look like, how many threads have you allocated?

Do you see any warnings about pool exhaustion?

If you run a netstat , is the server running/listening on the port?

Did you try redeploying the server deploy file?

-chhil

To unsubscribe, send email to jpos-users+unsubscribe@googlegroups.com

For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/d883764a-1df1-48ce-9588-12872e7d9dd7%40googlegroups.com.

[Anirudh Singh]

What does your server deploy file look like, how many threads have you allocated?

It looks like this
client 1 config : switch,mux,saf
client 2 config : switch,mux,saf
client 3 config : switch,mux,saf
client 4 config : switch,mux,saf
Logger
server
mux
trxn mgr
db check

Do you see any warnings about pool exhaustion? No

If you run a netstat , is the server running/listening on the port? Yes

Did you try redeploying the server deploy file? I think it does that at every q2 restart ?

[Anirudh Singh]

how many threads have you allocated?:

If by thread you mean the sessions in trxn manager :
<trxm class="org.jpos.transaction.TransactionManager" logger="Q2">
   
    <property name="queue" value="TRXMGR_localFEP_SERVER.txn" />
    <property name="sessions" value="100" />
    <property name="persistent-space" value="tspace:default" />
    <property name="debug" value="true"/>
    <property name="recover" value="true"/>
    <property name="retry-interval" value="250"/>
    <property name="pause-timeout" value="2000"/>

[chhil]

​​

What does your server deploy file look like, how many threads have you allocated?

​
It looks like this
client 1 config : switch,mux,saf
client 2 config : switch,mux,saf
client 3 config : switch,mux,saf
client 4 config : switch,mux,saf
Logger
server
mux
trxn mgr
db check​
​

​​

I was looking for the specific xml file contents of the deploy file, not sure what the above is supposed to mean
​ maybe you are indicating the flow of the​

​.

e.g. of a server deploy

​

<?xml version="1.0" ?>
<server name="fsapi-network-server-1" class="org.jpos.q2.iso.QServer" logger="Q2">
    <attr name="port" type="java.lang.Integer">1234</attr>
    <channel class="com.ols.xxx.TwoByteLengthHeaderChannel" logger="Q2" realm="api-channel-1" packager="org.jpos.iso.packager.GenericPackager">

        <property name="packager-config" value="cfg/blah.xml" />

    </channel>

    <request-listener class="com.ols.jpos.Dispatcher" logger="Q2" realm="incoming-api-request-listener-1">

    </request-listener>

    <in>
​api1_NETWORK_IN</in>
    <out>api1_NETWORK_OUT</out>

</server>

​

To unsubscribe, send email to jpos-users+unsubscribe@googlegroups.com

For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/a44511fb-f539-4fbe-8e04-5dbd1732987e%40googlegroups.com.

[Anirudh Singh]

Hi,

It looks like this :

<?xml version="1.0" encoding="UTF-8"?>
<server class="org.jpos.q2.iso.QServer" logger="Q2" name="ANIRUDH_SERVER_2150_CMS">
   
 <attr name="port" type="java.lang.Integer">7767</attr>
 
 <!-- Channel -->
 <channel class="org.jpos.iso.channel.ASCIIChannel" logger="Q2" packager="org.jpos.iso.packager.GenericPackager">
         <property name="packager-config" value="cfg/packager/iso93ascii.xml" />
        <property name="packager-realm" value="SWITCH_TO_CMS" />
 </channel>
 
 <!-- MUX -->
 <in>SWITCH_CMS_MUX_IN_2150_CMS</in>
 <out>SWITCH_CMS_MUX_OUT_2150_CMS</out>
 
</server>

Mux looks like this :

<mux class="org.jpos.q2.iso.QMUX" logger="Q2" name="SWITCH_CMS_MUX_2150_CMS">
   
 <in>SWITCH_CMS_MUX_OUT_2150_CMS</in>
 <out>SWITCH_CMS_MUX_IN_2150_CMS</out>
 
 <!-- Key -->
 <key>11,12</key>
 
<!-- Request Listener -->
<request-listener class="anirudh.CMSServerDispatcher" logger="Q2" realm="cms-incoming-request-listener">
</request-listener>
 
</mux>

[chhil]

Thanks for the deploys. They look fine.

Here is what I think is happening. 

Your server has multiple clients connected to it.

It receives a 1820 (at="Mon Oct 17 22:23:10 XXX 2016.493")  

And sends a response 1830 at at="Mon Oct 17 22:23:10 XXX 2016.493"

What you need to check is the port number in the channel/XXXXXX that you have, if its sending the response to the same client it received the request from.

The client gets disconnected by the server in 23 secs at at="Mon Oct 17 22:23:33 XXX 2016.706, do you have 23 configured somewhere?

Are there any devices (firewalls) between the server and client that could prevent the message from reaching the client?

If its reproducible, you may want to sniff the raw traffic (wireshark or some other app).

-chhil

To unsubscribe, send email to jpos-users+unsubscribe@googlegroups.com

For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/09c76136-edf0-42ae-922a-132dbda43778%40googlegroups.com.

[Anirudh Singh]

Thanks for the response .
I looked into what you said further and found out that a Random IP sent a request to the client.

Would this cause the client to be disconnected from the server ?

However i don't understand that why dint re-connection happen ?
Also is it possible to restrict the IP and port from which the client might get accessed ?

[chhil]

I looked into what you said further and found out that a Random IP sent a request to the client.

​

Now that's scary.

Would this cause the client to be disconnected from the server ?

​

The client did not disconnect, the server disconnected the client, is what I understood from the log. The why needs to be looked into.

However i don’t understand that why dint re-connection happen ?

​

Its up to the client to attempt a reconnect. If its jpos based it will retry based on the timeouts configured for the channel.

Also is it possible to restrict the IP and port from which the client might get accessed ?

​

There is configuration to allow and deny IP addresses via configuration, take a look at 

https://github.com/jpos/jPOS/blob/master/jpos/src/main/java/org/jpos/iso/ISOServer.java#L150-L197

You can search the group, its been discussed.

-chhil

To unsubscribe, send email to jpos-users+unsubscribe@googlegroups.com

For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/0b79f052-eeac-4aa8-8a11-2f1760cdcf66%40googlegroups.com.