20 Nov 2016 13:45:25 WARN NonBlockingSocketWriter - [machine01]:5702 [dev] [3.6] hz._hzInstance_1_dev.IO.thread-out-2 Closing socket to endpoint Address[127.0.0.1]:49417, Cause:java.nio.channels.CancelledKeyException
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at com.hazelcast.nio.tcp.nonblocking.AbstractHandler.unregisterOp(AbstractHandler.java:73)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketWriter.unschedule(NonBlockingSocketWriter.java:295)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketWriter.handle(NonBlockingSocketWriter.java:344)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketWriter.run(NonBlockingSocketWriter.java:423)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.executeTask(NonBlockingIOThread.java:236)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.processTaskQueue(NonBlockingIOThread.java:229)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.runSelectLoop(NonBlockingIOThread.java:201)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:163)
20 Nov 2016 13:45:24 INFO TransactionManagerService - [machine01
]:5702 [dev] [3.6] Committing/rolling-back alive transactions of client, UUID: b6f268ed-b1fb-4c57-9693-322106f7372e
20 Nov 2016 13:44:56 INFO TransactionManagerService - [machine01
]:5702 [dev] [3.6] Committing/rolling-back alive transactions of client, UUID: 32233ce9-146d-48ff-8b22-32f93056e553
20 Nov 2016 13:44:53 INFO TransactionManagerService - [machine01
]:5702 [dev] [3.6] Committing/rolling-back alive transactions of client, UUID: 061a1282-d4b6-48f2-922a-46ef55a8e4b3
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/55ac6cd1-b11e-4214-9fad-b62dde8768d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
Just to clarify here - your JVM heap is set to 30Gb? If so, how long are your major GC events? That's a huge heap size for JVMs. Most folks run heaps of 4Gb due to the costs of GC. Ive seen heaps of 12-16Gb, but that required an awful lot of dedicated and frequent application analysis to fine tune the various generations within the heap space.
On Mon, Nov 28, 2016 at 4:36 AM, Benjamin E.Ndugga <bjnd...@gmail.com> wrote:
I have 62GB of RAM and most of the times I find myself using 90% of the heap size, I have set the the xms and xmx to 30GB yet most of my information is less then 1GB, Where does the rest of the Memory get used up?. I get to see this error and have to restart the instance
20 Nov 2016 13:45:25 WARN NonBlockingSocketWriter - [machine01]:5702 [dev] [3.6] hz._hzInstance_1_dev.IO.thread-out-2 Closing socket to endpoint Address[127.0.0.1]:49417, Cause:java.nio.channels.CancelledKeyException
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at com.hazelcast.nio.tcp.nonblocking.AbstractHandler.unregisterOp(AbstractHandler.java:73)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketWriter.unschedule(NonBlockingSocketWriter.java:295)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketWriter.handle(NonBlockingSocketWriter.java:344)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingSocketWriter.run(NonBlockingSocketWriter.java:423)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.executeTask(NonBlockingIOThread.java:236)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.processTaskQueue(NonBlockingIOThread.java:229)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.runSelectLoop(NonBlockingIOThread.java:201)
at com.hazelcast.nio.tcp.nonblocking.NonBlockingIOThread.run(NonBlockingIOThread.java:163)
20 Nov 2016 13:45:24 INFO TransactionManagerService - [machine01
]:5702 [dev] [3.6] Committing/rolling-back alive transactions of client, UUID: b6f268ed-b1fb-4c57-9693-322106f7372e
20 Nov 2016 13:44:56 INFO TransactionManagerService - [machine01
]:5702 [dev] [3.6] Committing/rolling-back alive transactions of client, UUID: 32233ce9-146d-48ff-8b22-32f93056e553
20 Nov 2016 13:44:53 INFO TransactionManagerService - [machine01
]:5702 [dev] [3.6] Committing/rolling-back alive transactions of client, UUID: 061a1282-d4b6-48f2-922a-46ef55a8e4b3
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/71b78306-bce6-41f5-a83c-ba567f822e1f%40googlegroups.com.
G1HeapRegionSize can be made bigger since I have set the xmx to 20GB
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=300"
JAVA_OPTS="$JAVA_OPTS -XX:G1HeapRegionSize=32m"
JAVA_OPTS="$JAVA_OPTS -XX:+UseGCLogFileRotation"
JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=18"
JAVA_OPTS="$JAVA_OPTS -XX:NumberOfGCLogFiles=5"
JAVA_OPTS="$JAVA_OPTS -XX:GCLogFileSize=10M"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDateStamps"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/opt/hazlecast_logs/garbageCollector.log"
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=/opt/hazlecast_logs/"
Cheers,
Martijn
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/2a67ba1f-e8f0-400e-8c82-a3eb617ff1ce%40googlegroups.com.
Take a Look at
https://www.voltdb.com/blog/linux-transparent-huge-pages-and-voltdb
Hi Mike,
Thank you so much the article is really informative, I guess with G1 GC its all about setting to –Xmx1G –Xms1G –XX:+UseG1GC –XX:+PrintGCDetails –XX:+PrintGCTimeStamps. This works well however I am still trying to optimise and see to it that the HeapUsage Stats go above 80% and now I have expanded the -Xmx to 60GB.
I have notice from the management centre that Memory Distribution the "Other memory" grows faster than the "free memory". I am now trying to investigate that, If you have any ideas why that happens kindly share.
Thank you So much
Benji
On Thursday, 1 December 2016 16:28:15 UTC+3, Mike Pilone wrote:Benjamin,The G1 collector uses a different layout for memory using many, fixed sized regions and marking those regions as eden, survivor, or humongous/old generation. I'm not an expert on G1 at all but my understanding is that there may be minor GCs at anytime that promote objects from one region to another. The benefit of G1 is that is can work on individual regions rather than one large block of eden or survivor space. It can also move only the surviving objects and then free an entire region quickly.With a 32MB region and 20GB heap, you'll end up with 20 * 1024 / 32 = 640 regions. The documentation states that the JVM aims for 2048 regions so I'm assuming there is some reason the designer's chose that number but as with all things GC, you'll need to see how you're using it and tune appropriately.I recommend reading https://www.infoq.com/articles/G1-One-Garbage-Collector-To-Rule-Them-All for an introduction to G1. Then grab your GC logs and run them through an analyzer to look at GC timings and any warnings or hints the GC might be giving you for tuning. But of course look for obvious leaks or bad code first. The best GC in the world won't fix poor code that is leaking memory or allocating quicker than the memory can be reclaimed.-mikeOn Dec 1, 2016, at 4:10 AM, Benjamin E.Ndugga <bjnd...@gmail.com> wrote:Hi Mike,
To get this clear you are saying that G1 GC if I set the JVM option G1HeapRegionSize=32m this would mean that all the partitions will have a maximum of 32 MegaBytes? and each time they get filled there are Minor GCs?
Please do advise when the application can do Major GCs as well.
Thank You in Advance
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/5f819b01-010e-43ab-9752-5c2cfdd851d1%40googlegroups.com.
On Dec 4, 2016, at 8:53 AM, Benjamin E.Ndugga <bjnd...@gmail.com> wrote:
Hi Mike,
Thank you so much the article is really informative, I guess with G1 GC its all about setting to –Xmx1G –Xms1G –XX:+UseG1GC –XX:+PrintGCDetails –XX:+PrintGCTimeStamps. This works well however I am still trying to optimise and see to it that the HeapUsage Stats go above 80% and now I have expanded the -Xmx to 60GB.
I have notice from the management centre that Memory Distribution the "Other memory" grows faster than the "free memory". I am now trying to investigate that, If you have any ideas why that happens kindly share.
Thank you So much
Benji
On Thursday, 1 December 2016 16:28:15 UTC+3, Mike Pilone wrote:Benjamin,The G1 collector uses a different layout for memory using many, fixed sized regions and marking those regions as eden, survivor, or humongous/old generation. I'm not an expert on G1 at all but my understanding is that there may be minor GCs at anytime that promote objects from one region to another. The benefit of G1 is that is can work on individual regions rather than one large block of eden or survivor space. It can also move only the surviving objects and then free an entire region quickly.With a 32MB region and 20GB heap, you'll end up with 20 * 1024 / 32 = 640 regions. The documentation states that the JVM aims for 2048 regions so I'm assuming there is some reason the designer's chose that number but as with all things GC, you'll need to see how you're using it and tune appropriately.I recommend reading https://www.infoq.com/articles/G1-One-Garbage-Collector-To-Rule-Them-All for an introduction to G1. Then grab your GC logs and run them through an analyzer to look at GC timings and any warnings or hints the GC might be giving you for tuning. But of course look for obvious leaks or bad code first. The best GC in the world won't fix poor code that is leaking memory or allocating quicker than the memory can be reclaimed.-mikeHi Mike,
To get this clear you are saying that G1 GC if I set the JVM option G1HeapRegionSize=32m this would mean that all the partitions will have a maximum of 32 MegaBytes? and each time they get filled there are Minor GCs?
Please do advise when the application can do Major GCs as well.
Thank You in Advance
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/5f819b01-010e-43ab-9752-5c2cfdd851d1%40googlegroups.com.
Hi Benjamin,Are you able to send me your GC log? Happy to take a look.
Cheers,
Martijn
On 4 December 2016 at 13:53, Benjamin E.Ndugga <bjnd...@gmail.com> wrote:
Hi Mike,
Thank you so much the article is really informative, I guess with G1 GC its all about setting to –Xmx1G –Xms1G –XX:+UseG1GC –XX:+PrintGCDetails –XX:+PrintGCTimeStamps. This works well however I am still trying to optimise and see to it that the HeapUsage Stats go above 80% and now I have expanded the -Xmx to 60GB.
I have notice from the management centre that Memory Distribution the "Other memory" grows faster than the "free memory". I am now trying to investigate that, If you have any ideas why that happens kindly share.
Thank you So much
Benji
On Thursday, 1 December 2016 16:28:15 UTC+3, Mike Pilone wrote:Benjamin,The G1 collector uses a different layout for memory using many, fixed sized regions and marking those regions as eden, survivor, or humongous/old generation. I'm not an expert on G1 at all but my understanding is that there may be minor GCs at anytime that promote objects from one region to another. The benefit of G1 is that is can work on individual regions rather than one large block of eden or survivor space. It can also move only the surviving objects and then free an entire region quickly.With a 32MB region and 20GB heap, you'll end up with 20 * 1024 / 32 = 640 regions. The documentation states that the JVM aims for 2048 regions so I'm assuming there is some reason the designer's chose that number but as with all things GC, you'll need to see how you're using it and tune appropriately.I recommend reading https://www.infoq.com/articles/G1-One-Garbage-Collector-To-Rule-Them-All for an introduction to G1. Then grab your GC logs and run them through an analyzer to look at GC timings and any warnings or hints the GC might be giving you for tuning. But of course look for obvious leaks or bad code first. The best GC in the world won't fix poor code that is leaking memory or allocating quicker than the memory can be reclaimed.-mikeOn Dec 1, 2016, at 4:10 AM, Benjamin E.Ndugga <bjnd...@gmail.com> wrote:Hi Mike,
To get this clear you are saying that G1 GC if I set the JVM option G1HeapRegionSize=32m this would mean that all the partitions will have a maximum of 32 MegaBytes? and each time they get filled there are Minor GCs?
Please do advise when the application can do Major GCs as well.
Thank You in Advance
--
You received this message because you are subscribed to the Google Groups "Hazelcast" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+...@googlegroups.com.
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/141c5ff6-6087-4406-a2d0-80ae1181b312%40googlegroups.com.
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
JAVA_OPTS="$JAVA_OPTS -XX:MaxGCPauseMillis=300"
JAVA_OPTS="$JAVA_OPTS -XX:+UseGCLogFileRotation"
JAVA_OPTS="$JAVA_OPTS -XX:ParallelGCThreads=20"
JAVA_OPTS="$JAVA_OPTS -XX:NumberOfGCLogFiles=5"
JAVA_OPTS="$JAVA_OPTS -XX:GCLogFileSize=10M"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDateStamps"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/u02/hazelcast-logs/garbageCollector.log"
JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:HeapDumpPath=/u02/hazelcast-logs/"
Cheers,
Martijn
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/1761c407-0a8d-4f7a-98f8-0fdddf1ad69f%40googlegroups.com.
Cheers,
Martijn
To unsubscribe from this group and stop receiving emails from it, send an email to hazelcast+unsubscribe@googlegroups.com.
To post to this group, send email to haze...@googlegroups.com.
Visit this group at https://groups.google.com/group/hazelcast.
To view this discussion on the web visit https://groups.google.com/d/msgid/hazelcast/0dacdcd8-bf04-4f22-bbc1-d3cf1b50e685%40googlegroups.com.