java.lang.IllegalStateException in VanillaChronicleMap

[Randy Prager]

We are seeing this exception

java.lang.IllegalStateException
	at net.openhft.chronicle.map.ShortShortMultiMap.nextPos(ShortShortMultiMap.java:227)
	at net.openhft.chronicle.map.VanillaChronicleMap$Segment.acquireWithoutLock(VanillaChronicleMap.java:1746)
	at net.openhft.chronicle.map.VanillaChronicleMap.lookupUsing(VanillaChronicleMap.java:810)
	at net.openhft.chronicle.map.VanillaChronicleMap.acquireUsingLocked(VanillaChronicleMap.java:772)
2015-04-10 14:51:26,729 WARN  [A-selector] net.openhft.chronicle.hash.ChronicleHashErrorListeners$LoggingErrorListener - Grabbing lock held by processId: 134217728, threadId: 0
writersWaiting has underflowed
writersWaiting has underflowed
writersWaiting has underflowed
writersWaiting has underflowed
2015-04-10 14:51:26,730 WARN  [B-selector] net.openhft.chronicle.hash.ChronicleHashErrorListeners$LoggingErrorListener - Grabbing lock held by threadId: 0
2015-04-10 14:51:26,731 WARN  [C-selector] net.openhft.chronicle.hash.ChronicleHashErrorListeners$LoggingErrorListener - Grabbing lock held by threadId: 0
2015-04-10 14:51:26,731 WARN  [D-selector] net.openhft.chronicle.hash.ChronicleHashErrorListeners$LoggingErrorListener - Grabbing lock held by threadId: 0
2015-04-10 14:51:26,730 WARN  [E-selector] net.openhft.chronicle.hash.ChronicleHashErrorListeners$LoggingErrorListener - Grabbing lock held by threadId: 0

Here is the code

 try (WriteContext<SmallCharSequence, LongValue> context = cache.acquireUsingLocked(small, longValue)) {

        longValue.setValue(helper.id);

 } 

Have you seen this before?  

Using 

chronicle-map 2.1.4, 

lang 6.6.2, 

chronicle 3.4.2

collections 3.3.2

compiler 2.2.0

[Rob Austin]

thanks we wil investigate this.

--
You received this message because you are subscribed to the Google Groups "Chronicle" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-chronicl...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Randy Prager]

The cause appears to be when the map is sized too small.  

[Greg Temchenko]

Getting this error too. The stacktrace is different though:

java.lang.IllegalStateException: writersWaiting has underflowed

        at net.openhft.lang.io.AbstractBytes.tryRWWriteLock0(AbstractBytes.java:2770)

        at net.openhft.lang.io.AbstractBytes.tryRWWriteLock(AbstractBytes.java:2735)

        at net.openhft.chronicle.map.VanillaChronicleMap$Segment.readLock(VanillaChronicleMap.java:1572)

        at net.openhft.chronicle.map.VanillaChronicleMap$Segment.containsKey(VanillaChronicleMap.java:2226)

        at net.openhft.chronicle.map.VanillaChronicleMap$Segment.access$000(VanillaChronicleMap.java:1442)

        at net.openhft.chronicle.map.VanillaChronicleMap.containsKey(VanillaChronicleMap.java:908)

        at net.openhft.chronicle.map.VanillaChronicleMap.containsKey(VanillaChronicleMap.java:887)

It happens only under heavy load.

Any explanations on what happens?

[Peter Lawrey]

When a thread waits a long time (more than 20 seconds) it grabs the lock on the assumption the process holding the lock has died. If the thread has not actually died it detects that it lost the write lock and complains.

We could do a better job of adjusting the time out for systems under extreme loads but I suggest you should avoid overloading the server to this degree.

[Randy Prager]

I believe we saw a similar exception that we fixed by ensuring all map access occurred inside the ReadContext and WriteContext and only operations that modified the map were done in the WriteContext.

[Peter Lawrey]

Doing this will improve concurrency. Only use the write context if you really need to.

On 16 May 2015 11:57 am, "Randy Prager" <ra...@fixflyer.com> wrote:

I believe we saw a similar exception that we fixed by ensuring all map access occurred inside the ReadContext and WriteContext and only operations that modified the map were done in the WriteContext.

[GorillaCoder]

RE: I suggest you should avoid overloading the server to this degree.

We are seeing this during a full gc. We can defer, but there is no way to stop full gcs. What do you recommend we do?

[Peter Lawrey]

It is possible to have systems without full GC,  but most systems will have them.
I will look at changing the strategy to allow for GC pauses.

[Peter Lawrey]

We are working on a number of options to improve this.  The first one is being more tolerant of jitter in the system.

https://higherfrequencytrading.atlassian.net/browse/HCOLL-332

[GorillaCoder]

Yeah, it is. I know how to do it, but am working on a system I inherited. There are bigger fish to fry first, but reducing/eliminating full GCs is on the agenda.

In a later reply you mention being more tolerant of jitter. You are probably already aware of this: Gene Tene (of Azul) has a great tool for measuring jitter. Pretty much the right approach, using essentially a log scale, to capture a histogram of jitter durations in a compact memory space. You don't really need the histogram over all time, so a smaller bounded buffer and a hi prio thread to measure the recent jitter should do.

BTW: was half way through my own implementation of a large MemoryBuffer when I found yours (Needed it both for large data sets and to eliminate java's MemoryBuffer's requirement to call System.gc to clean up resources). It's what brought Chronicle to our attention.

It was hard to find both your MemoryBuffer replacement and Chronicle in google searches. You might want to figure out how to bump them up in the rankings. 

[Peter Lawrey]

Hello GorillaCoder,

   The approach we are taking for systems with long pauses is to count down the timeout.  This will treat jump of 2 or more milliseconds as just one milli-second.  This is useful for not just GC pauses but debugging code, line by line without timeouts getting triggered.

Regards,

   Peter.

--