High memory usage

151 views
Skip to first unread message

Mohd rully k

unread,
Apr 4, 2017, 3:55:43 AM4/4/17
to jPOS Users
Dear all 

First I want to say many thanks to Creator, Developer, Support ,Maintaner and etc for JPOS framework .

So , for the record and to be honest I'm a newbie . I develope a gateway using Jpos version 1.9.8 , library in use Spring+Hibernate for log and query , Apache HTTP Client and Netty for convert ISO 8583 to object and send http request and get response in socket mode , and convert to ISO 8583 , also using HikariCP and MySQL for connection pooling also database. 
Normal usage memory not even above 20 % . But after several days ( > 2 days ) memory become growing more and more until  88% 
I'm linux using ubuntu 14.04.5 , Xeon 6 core , 8 GB RAM . 
this is my configuration :
<server class="org.jpos.q2.iso.QServer" logger="Q2" name="Server-A">
<attr name="port" type="java.lang.Integer">*****</attr>
<attr name="maxSessions" type="java.lang.Integer">100</attr>
<channel name="Server-A-Channel" class="org.jpos.iso.channel.ASCIIChannel"
logger="Q3" packager="org.jpos.iso.packager.GenericPackager">
<property name="packager-config" value="cfg/packager/iso87ascii.xml" />
</channel>
<request-listener class="com.******.listener.ServerListener"
logger="Q2">
<property name="space" value="transient:default" />
<property name="queue" value="TransactionQueue" />
</request-listener>
</server>

<txnmgr name="txnmgr" 
class="org.jpos.transaction.TransactionManager">
<property name="space" value="transient:default" />
<property name="queue" value="TransactionQueue" />
<property name="max-sessions" value="100" />
<property name="debug" value="false" />
<participant class="com.******.transaction.TransMan"
logger="Q3">
.......

<channel-adaptor name='client-adaptor' class="org.jpos.q2.iso.ChannelAdaptor"
logger="Q2">
<channel class="org.jpos.iso.channel.ASCIIChannel" packager="org.jpos.iso.packager.GenericPackager">
<property name="host" value="127.0.0.1" />
<property name="port" value="*****" />
<property name="packager-config" value="cfg/packager/iso87ascii.xml" />
<property name="keep-alive" value="true" />
</channel>
</channel-adaptor>
<log realm="org.jpos.q2.qbean.SystemMonitor" at="Tue Apr 04 14:30:43 WIB 2017.603">
  <info>
               OS: Linux
             host: DBDEV/127.0.1.1
          version: 1.9.8 (f46a5dc)
         instance: b95804b9-1219-4b8d-a5d0-b8936b4c19e5
           uptime: 1d 04:00:07.967
       processors: 6
           drift : 0
    memory(t/u/f): 332/267/64
          threads: 63
            Thread[Reference Handler,10,system]
            Thread[Finalizer,8,system]
            Thread[Signal Dispatcher,9,system]
            Thread[main,5,main]
            Thread[Q2-b95804b9-1219-4b8d-a5d0-b8936b4c19e5,5,main]
            Thread[pool-1-thread-1,5,main]
            Thread[Timer-0,5,main]
            Thread[nioEventLoopGroup-2-1,10,main]
            Thread[nioEventLoopGroup-2-2,10,main]
            Thread[threadDeathWatcher-3-1,1,main]
            Thread[Abandoned connection cleanup thread,5,main]
            Thread[Hikari Housekeeping Timer,5,main]
            Thread[Hikari Housekeeping Timer,5,main]
            Thread[Thread-6,5,main]
            Thread[Timer-1,5,main]
            Thread[txnmgr-0:idle,5,main]
            Thread[channel-sender-null,5,main]
            Thread[channel-sender-hsm-send,5,main]
            Thread[channel-receiver-hsm-receive,5,main]
            Thread[SystemMonitor,5,main]
            Thread[nioEventLoopGroup-2-3,10,main]
            Thread[nioEventLoopGroup-2-4,10,main]
            Thread[txnmgr-1:idle,5,main]
            Thread[nioEventLoopGroup-2-5,10,main]
            Thread[nioEventLoopGroup-2-6,10,main]
            Thread[nioEventLoopGroup-2-7,10,main]
            Thread[nioEventLoopGroup-2-8,10,main]
            Thread[nioEventLoopGroup-2-9,10,main]
            Thread[nioEventLoopGroup-2-10,10,main]
            Thread[nioEventLoopGroup-2-11,10,main]
            Thread[nioEventLoopGroup-2-12,10,main]
            Thread[MySQL Statement Cancellation Timer,5,main]
            Thread[MySQL Statement Cancellation Timer,5,main]
            Thread[PooledThread-0-idle,5,ThreadPool-0-1]
            Thread[PooledThread-1-running,5,ThreadPool-0-1]
            Thread[PooledThread-2-idle,5,ThreadPool-0-1]
            Thread[PooledThread-3-idle,5,ThreadPool-0-1]
            Thread[PooledThread-4-idle,5,ThreadPool-0-1]
            Thread[PooledThread-5-idle,5,ThreadPool-0-1]
            Thread[PooledThread-6-idle,5,ThreadPool-0-1]
            Thread[PooledThread-7-idle,5,ThreadPool-0-1]
            Thread[PooledThread-8-idle,5,ThreadPool-0-1]
            Thread[PooledThread-9-idle,5,ThreadPool-0-1]
            Thread[PooledThread-10-idle,5,ThreadPool-0-1]
            Thread[PooledThread-11-idle,5,ThreadPool-0-1]
            Thread[PooledThread-12-idle,5,ThreadPool-0-1]
            Thread[PooledThread-13-idle,5,ThreadPool-0-1]
            Thread[PooledThread-14-idle,5,ThreadPool-0-1]
            Thread[PooledThread-15-idle,5,ThreadPool-0-1]
            Thread[PooledThread-16-idle,5,ThreadPool-0-1]
            Thread[PooledThread-17-idle,5,ThreadPool-0-1]
            Thread[PooledThread-18-idle,5,ThreadPool-0-1]
            Thread[PooledThread-19-idle,5,ThreadPool-0-1]
            Thread[PooledThread-20-idle,5,ThreadPool-0-1]
            Thread[PooledThread-21-idle,5,ThreadPool-0-1]
            Thread[PooledThread-22-idle,5,ThreadPool-0-1]
            Thread[PooledThread-23-idle,5,ThreadPool-0-1]
            Thread[PooledThread-24-idle,5,ThreadPool-0-1]
            Thread[PooledThread-25-idle,5,ThreadPool-0-1]
            Thread[PooledThread-26-idle,5,ThreadPool-0-1]
            Thread[PooledThread-27-idle,5,ThreadPool-0-1]
            Thread[PooledThread-28-idle,5,ThreadPool-0-1]
            Thread[PooledThread-29-idle,5,ThreadPool-0-1]
            Thread[PooledThread-30-running,5,ThreadPool-0-1]
            Thread[PooledThread-31-idle,5,ThreadPool-0-1]
            Thread[PooledThread-32-idle,5,ThreadPool-0-1]
         encoding: US-ASCII
    name-registrar:
      logger.Q2: org.jpos.util.Logger
      server.Server-A: org.jpos.iso.ISOServer
        connected=2, rx=3430, tx=3432, last=1491291029223, idle=14407ms
          172.18.24.35:45942: rx=2232, tx=2233, last=1491291029223
          172.18.24.67:45583: rx=1198, tx=1199, last=1491291029223
      mux.hsm-mux: org.jpos.q2.iso.QMUX
        tx=27, rx=26, tx_expired=0, tx_pending=0, rx_expired=1, rx_pending=0, rx_unhandled=0, rx_forwarded=0, connected=true, last=1491279686446, idle=11357184ms
      tspace:org.jpos.transaction.TransactionManager@1041cf19: org.jpos.space.TSpace


and my problem is when memory becomes grow larger the application suddenly shutdown(hook) , Sorry for my bad english . Hope anyone can give me pointer ,

Thank in advance 

Rully 




htop_gateway.JPG

Victor Salaman

unread,
Apr 4, 2017, 4:22:39 AM4/4/17
to jpos-...@googlegroups.com
Hi:

You might want to try using a Profiler such as XRebel or YourKit to find who is the culprit. I would start by analyzing all the non jPOS stuff you added.

/V

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/914b0455-b730-4599-b1b9-8513cd86ffaa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mohd rully k

unread,
Apr 4, 2017, 9:36:48 PM4/4/17
to jPOS Users
Hi Mr.Victor 

Ok , I will try as you suggested . 

Thank you 

Regard

Rully
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.

A Haris Kurniawan

unread,
Apr 5, 2017, 1:53:37 AM4/5/17
to jpos-...@googlegroups.com
Wow if those stuff considered as a newbie than i'm probably suits as retarted ;)

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.

To post to this group, send email to jpos-...@googlegroups.com.

Mohd rully k

unread,
Apr 5, 2017, 9:33:39 AM4/5/17
to jPOS Users
Hi All , finally I ended up using debian 6 box 64bit with old kernel 2.6.x with MySQL 5.1, JDK still same Oracle 1.7.80. Until now memory use still safe and did not grow above expectation (only 700MB max) . No change in code and configuration still same , just hope it will be fine because still confuse how using xrebel or profiler .

@Harris : My point of view not for comparation , maybe 'still learning' is the best words , Aren't we ? 

Andy Orrock

unread,
Apr 5, 2017, 9:47:31 AM4/5/17
to jpos-...@googlegroups.com
Sounds like a case of "good switch, bad switch."


But here's the thing:  You can use jPOS to build a good switch...or a bad one.  There's nothing preventing you from taking jPOS and building a craptastic switch...nothing to prevent that at all.  


--------
Andy

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.
To post to this group, send email to jpos-...@googlegroups.com.

A Haris Kurniawan

unread,
Apr 5, 2017, 8:22:38 PM4/5/17
to jpos-...@googlegroups.com
I notice that you're running jar multiple times. Is that jar wraps jpos?

Now i'm wondering, each of those instances of jar probably has jpos-space configuration...
Will they (the space) interfere with each other (instance)

I assume that you're using Q2, Why not create a single instance and split the logic for different client using bit3 and bit32. The logic can be placed on different participants using the transaction manager. Different participant can connect to different database, soap.endpoint, rest, resources, etc. So it could be a big-arse switcher, that does its job... switching. 
Just my thoughts anyway...

Mohd rully k

unread,
Apr 5, 2017, 9:28:07 PM4/5/17
to jPOS Users
Hi Mr Andy 

I'm so happy legend of jpos comment my post btw . Yes , this a bad build , but I can reject it , because It's my job , and it's my first jpos project anyway . The client using strange communication to talk with they core banking . Even strange where the response is when have special character written in hexa . I have already using standart socket listener with thread ( blocking IO ) and performance was bad . In this build performance was satisfied even not reached hundred like you do ( only around 30 TPS ) and they not even complaining what I'm trying to use as long it run . Until now memory utilization still fine in devian box , even I know there is something wrong with my code , but maybe someday I'll figure it out .  

@haris : I dont understand what you talking about multiple jar, when you see htop/top in linux process are like that , It only 1 jar , can't split it out because a lot channel going through this build like HSM and etc . 


--------
Andy

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.

To post to this group, send email to jpos-...@googlegroups.com.

A Haris Kurniawan

unread,
Apr 5, 2017, 11:28:34 PM4/5/17
to jpos-...@googlegroups.com
not multiple jar, multiple instance of the same jar (Gateway-1A.jar)
CMIIW they have different PID, so each one of them is running as a different instance

To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+unsubscribe@googlegroups.com.

To post to this group, send email to jpos-...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
Best regards,
A. Haris Kurniawan

Mohd rully k

unread,
Apr 6, 2017, 10:28:34 PM4/6/17
to jPOS Users
Do you prefer this view @haris ? Btw I just figure it out the problem , many thank . 
htop_gw_238.PNG
Reply all
Reply to author
Forward
0 new messages