Deadlocked threads using PipedInput/OutputStream with AsyncCompletionHandler

[Tim Jones]

Hi,

We are using ahc with much success; it's a great piece of software.  However, we have an intermittent issue where threads stall out while reading data.  

Our ahc client calls a custom HTTP service, and there is a 3rd-party API which consumes an InputStream when parsing the result.  We use a PipedInputStream for the parser; the PipedOutputStream is passed to our AsyncCompletionHandler impl.  Every once in awhile, we'll see some stuck threads (this is running in weblogic) with the following trace:

    -- Waiting for notification on: java.io.PipedInputStream@ffffffffe3af4840[fat lock]

    java.io.PipedInputStream.read(PipedInputStream.java:238)

    java.io.PipedInputStream.read(PipedInputStream.java:295)

    java.io.BufferedInputStream.read1(BufferedInputStream.java:247)

    java.io.BufferedInputStream.read(BufferedInputStream.java:304)

    ^-- Holding lock: java.io.BufferedInputStream@ffffffffe3ae5f52[thin lock]

    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:397)

    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:436)

    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:150)

    java.io.InputStreamReader.read(InputStreamReader.java:167)

    ^-- Holding lock: java.io.InputStreamReader@ffffffffe3ae6de7[thin lock]

    java.io.Reader.read(Reader.java:122)

    com.....ResultParser.parse(ResultParser.java:44)

    ...

At it's worse, these errors can exhaust file descriptors for the process, but we've only seen that happen once.  The problem is very intermittent, and I've been unable to reproduce.

A trimmed-down version of the code is at https://gist.github.com/t-jones/5640237.  The impl makes the following assumptions which I think are right, but validation would be nice:

- The PipedOutputStream will be written to by an ahc thread handling the response, including being closed in the onCompleted() or onThrowable() methods.

- The thread handling the request will block in the ResultParser on the input stream (i.e. the PipedInputStream) until EOF.

Does the code look right?  Are there any cases where the OutputStream would not be closed?

What I'll try next: upgrade to latest AHC (currently using 1.6.5 which is a little old); check out BodyDeferringAsyncHandler, although it looks pretty similar to what we have.

Thanks for any help,

Tim

[Mikhail Mazursky]

Hi, Tim,

we are planing to use similar aproach with piped streams but we haven't tested it under load. To me the code looks ok. What java version are you using? Can you provide the trace of the second thread that is blocked with this one in a deadlock? I mean are you sure it's a deadlock - maybe it's just waiting that never ends for some other reason?

Regards,

Mikhail.

2013/5/24 Tim Jones <timoth...@hp.com>

--
You received this message because you are subscribed to the Google Groups "asynchttpclient" group.
To unsubscribe from this group and stop receiving emails from it, send an email to asynchttpclie...@googlegroups.com.
To post to this group, send email to asyncht...@googlegroups.com.
Visit this group at http://groups.google.com/group/asynchttpclient?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Tim Jones]

Hi Mikhail,

You are right, it is not deadlock.  There is no trace from another thread.  This thread is just hung, waiting for bytes which never come.  This is running on jrockit 1.5.0_15 (for various enterprise-y reasons).

With the exception of this issue, this code handles 800K requests per day; 250-300 TPS at peak load.

Regards,

Tim

[Stéphane Landelle]

Hi,

From PipedInputStream javadoc: "Attempting to use both objects (PipedInputStream and PipedOutputStream) from a single thread is not recommended, as it may deadlock the thread."

I think that's exactly what you're doing.

2013/5/24 Tim Jones <timoth...@hp.com>

[Tim Jones]

On Friday, May 24, 2013 12:50:13 AM UTC-7, Stéphane Landelle wrote:

From PipedInputStream javadoc: "Attempting to use both objects (PipedInputStream and PipedOutputStream) from a single thread is not recommended, as it may deadlock the thread."

I think that's exactly what you're doing.

Thanks for the feedback.  Doesn't the AsyncCompletionHandler run in a different thread?  This code is essentially the same as the examples given in the javadocs for BodyDeferringAsyncHandler (piped output passed into handler; piped input read from in the current thread).

[Stéphane Landelle]

My bad, I didn't understand it this way...

How does your Handler handle exceptions?

2013/5/24 Tim Jones <timoth...@hp.com>

[Tim Jones]

Nothing special; just overrides onThrowable to log the error:

                        @Override

                        public void onThrowable(Throwable se) {

                            try {

                                out.close();

                            }

                            catch (IOException e) {

                                log.error("Exception closing output.", e);

                            }

                            log.error("There was an error ...");

                            log.error("Caught Exception: {}", se.getMessage());

                        }

It will throw out of onCompleted on application error from the http service we are calling (https://gist.github.com/t-jones/5640237#file-ahcclientimpl-java-L40).

[Stéphane Landelle]

Did you notice any error log?

Have you tried upgrading AHC and Netty (latest 3.X)?

2013/5/24 Tim Jones <timoth...@hp.com>

[Tim Jones]

Nothing in the error logs.  Only thing I've found is some sort of network hiccup around the time of some of the errors.  Will look into this more.

We are going to upgrade to AHC 1.7.16 (Netty 3.6.3.Final).  It will be a couple weeks before it makes it into production.

[Stéphane Landelle]

Beware that's 1.7.17 with Netty 3.6.6 is planned for this week.

2013/5/28 Tim Jones <timoth...@hp.com>