IllegalStateException on Matcher

[Kamyar Nadjafloo]

Hi guys

when we're doing concurrent testing on the matcher, we're getting an IllegalStateException com.googlecode.streamflyer.regex.OnStreamStandardMatcher#start(int)

This Exception is bubbling up when we're doing a regular expression replacement.

I'm doing

Modifier modifier = new RegexModifier(tokensMatcher, tokenProcessor, 1, maxBufferSize);
AbstractBody body = ((AbstractBody)exc.getResponse().getBody());
InputStream originalByteStream = body.getInputStream();

...

originalReader = new InputStreamReader(filteringInputStream, charset);
Reader reader = new ModifyingReader(originalReader, modifier);

InputStream modifyingByteStream = new ReaderInputStream(reader, charset, BUFFER_SIZE);

the problem is that while we're writing out from the inputStream, that  IllegalStateException occurs because there's no match and it blows up the entire write. buffer size is 2048 and we're reading a small text/html page with a simple replacement that's always there. 

I don't know if I'm doing something wrong in the way we use the library? Why's that exception being thrown. Thanks

[rw...@gmx.de]

Hi Kamyar,

You mention concurrent testing. Does that mean that many threads share the same matcher instance?

Do you describe a scenario where different threads read concurrently data from the same input stream? This would be surprising scenario. Could you describe the background a bit more?

This said, streamflyer has no internal synchronisation, i.e. it is not thread-safe because usually you don't need it.

And please post as much of the stacktrace of the exception as possible. There is no code in OnStreamStandardMatcher that throws an IllegalStateException. So it must be thrown by code  the class delegates to, most probably java.util.regex.Matcher.

Regards

Rod

[Kamyar Nadjafloo]

There's different stack traces. We're not running multiple threads on the same matcher. We're running our transformation class on multiple threads so each transformation _should_ be ever run on a single thread, that's what's puzzling

2015-01-19 16:10:45,666 TRACE  core.transport.http.HttpServerHandler:174 - Exception of type: class java.lang.IndexOutOfBoundsException occurred - start

java.lang.IndexOutOfBoundsException: start

at java.util.regex.Matcher.region(Matcher.java:1036)

at com.googlecode.streamflyer.regex.OnStreamStandardMatcher.findUnlessHitEnd(OnStreamStandardMatcher.java:91)

at com.googlecode.streamflyer.regex.addons.util.DelegatingMatcher.findUnlessHitEnd(DelegatingMatcher.java:84)

at com.googlecode.streamflyer.regex.RegexModifier.modify(RegexModifier.java:364)

at com.googlecode.streamflyer.core.ModifyingReader.readCharacter(ModifyingReader.java:275)

at com.googlecode.streamflyer.core.ModifyingReader.read(ModifyingReader.java:247)

at org.apache.commons.io.input.ReaderInputStream.fillBuffer(ReaderInputStream.java:198)

at org.apache.commons.io.input.ReaderInputStream.read(ReaderInputStream.java:242)

at org.apache.commons.io.input.ReaderInputStream.read(ReaderInputStream.java:261)

015-01-19 16:10:45,669 TRACE .core.transport.http.HttpServerHandler:174 - Exception of type: class java.lang.StringIndexOutOfBoundsException occurred - String index out of range: 136

java.lang.StringIndexOutOfBoundsException: String index out of range: 136

at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:210)

at java.lang.StringBuilder.charAt(StringBuilder.java:76)

at java.util.regex.Pattern$Slice.match(Pattern.java:3969)

at java.util.regex.Pattern$GroupHead.match(Pattern.java:4658)

at java.util.regex.Matcher.match(Matcher.java:1270)

at java.util.regex.Matcher.lookingAt(Matcher.java:682)

at com.googlecode.streamflyer.regex.OnStreamStandardMatcher.findUnlessHitEnd(OnStreamStandardMatcher.java:93)

at com.googlecode.streamflyer.regex.addons.util.DelegatingMatcher.findUnlessHitEnd(DelegatingMatcher.java:84)

at com.googlecode.streamflyer.regex.RegexModifier.modify(RegexModifier.java:364)

at com.googlecode.streamflyer.core.ModifyingReader.readCharacter(ModifyingReader.java:275)

at com.googlecode.streamflyer.core.ModifyingReader.read(ModifyingReader.java:247)

at org.apache.commons.io.input.ReaderInputStream.fillBuffer(ReaderInputStream.java:198)

at org.apache.commons.io.input.ReaderInputStream.read(ReaderInputStream.java:242)

at org.apache.commons.io.input.ReaderInputStream.read(ReaderInputStream.java:261)

---

---

[Kamyar Nadjafloo]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

[Kamyar Nadjafloo]

2015-01-20 10:53:39,853 TRACE .core.transport.http.HttpServerHandler:174 - Exception of type: class java.lang.IllegalStateException occurred - No match found

java.lang.IllegalStateException: No match found

at java.util.regex.Matcher.group(Matcher.java:536)

at com.googlecode.streamflyer.regex.OnStreamStandardMatcher.group(OnStreamStandardMatcher.java:177)

at com.googlecode.streamflyer.regex.addons.util.DelegatingMatcher.group(DelegatingMatcher.java:69)

at com.googlecode.streamflyer.regex.addons.tokens.TokenProcessor.process(TokenProcessor.java:59)

at com.googlecode.streamflyer.regex.RegexModifier.modify(RegexModifier.java:396)

at com.googlecode.streamflyer.core.ModifyingReader.readCharacter(ModifyingReader.java:275)

at com.googlecode.streamflyer.core.ModifyingReader.read(ModifyingReader.java:247)

at org.apache.commons.io.input.ReaderInputStream.fillBuffer(ReaderInputStream.java:198)

at org.apache.commons.io.input.ReaderInputStream.read(ReaderInputStream.java:242)

at org.apache.commons.io.input.ReaderInputStream.read(ReaderInputStream.java:261)

[Kamyar Nadjafloo]

I also found this. not sure if it's related.

http://stackoverflow.com/questions/23123871/java-regex-cannot-match-one-line-comment

[Kamyar Nadjafloo]

I'm getting the exception throw new RuntimeException("never to happen if used with " + TokensMatcher.class); from TokenProcessor.java

[Kamyar Nadjafloo]

sorry for spam posts but I checked out the source code and modified com.googlecode.streamflyer.regex.OnStreamStandardMatcher and made every method of that class synchronized. it seems like 99% of synchronization issues are gone. Not sure still the real cause. Any ideas? thanks again

[Kamyar Nadjafloo]

ok and finally answering my own question, you cannot share instances of TokenProcessor among different threads....

[rw...@gmx.de]

finally answering my own question, you cannot share instances of TokenProcessor among different threads....

Yes, exactly. As I suggested in my first post, both OnStreamStandardMatcher is not thread-safe. It uses java.util.regex.Matcher which is not thread-safe itself.

The right solution for you is the following: Use a different Modifier (here: RegexModifier) for each thread, i.e. use thread-specific instances. Then your problem will disappear.

I have just updated the Wiki documentation and the code documentation to make clear that thread-specific instances are required.

Regarding your suggestion: Making OnStreamStandardMatcher synchronized does not solve the problem properly because a RegexModifier has a state which must not be shared.

Best Regards

Rod

[Kamyar Nadjafloo]

great. thanks for the explanation. At first for performance reasons I was sharing instances of TokenProcesses (used in RegexModifier) which was causing the issue. I was creating a new instance of RegexModifier but it was sharing the TokenProcessor and TokensMatcher.

Cheers

[rw...@gmx.de]

I am glad that I could help.

Regards

Rod