JrppBufferedOutputStream locked
andr...@moon.com.au
servers. Periodically random requests get locked when flushing the
output stream (see stack trace below for an example). Typically one of
these requests will run for about 100 seconds, then finally execute.
Requests to the same page while such a request is locked will be
processed normally.
Every few days, a collection of these requests will lock all at once,
causing hundreds of requests to be queued, which can bring the
ColdFusion service down.
After Googling extensively, I am no nearer to a solution. Any ideas
anyone?
Andrew
--------
stack trace
--------
jrpp-1506" prio=5 tid=0x0392c790 nid=0x1620 runnable
[3eaaf000..3eaafd94]
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:
92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at jrun.servlet.jrpp.JrppOutputStream$SpillStream.write
(JrppOutputStream.java:182)
at jrun.servlet.io.MetricsOutputStream.write(MetricsOutputStream.java:
75)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:
66)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
- locked <0x1edf42d8> (a jrun.servlet.jrpp.JrppBufferedOutputStream)
at jrun.servlet.jrpp.JrppOutputStream.flush(JrppOutputStream.java:45)
at com.intergral.fusionreactor.A.A.flush(Unknown Source)
at com.intergral.fusionreactor.A.A.C(Unknown Source)
at com.intergral.fusionreactor.A.A.write(Unknown Source)
at com.intergral.fusionreactor.filter.A.C.write(Unknown Source)
at sun.nio.cs.StreamEncoder$CharsetSE.writeBytes(StreamEncoder.java:
336)
at sun.nio.cs.StreamEncoder$CharsetSE.implWrite(StreamEncoder.java:
395)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:136)
- locked <0x21d3b890> (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:191)
at java.io.PrintWriter.write(PrintWriter.java:200)
- locked <0x21d3b890> (a java.io.OutputStreamWriter)
at java.io.CharArrayWriter.writeTo(CharArrayWriter.java:120)
- locked <0x21d3a250> (a coldfusion.runtime.CharBuffer)
charlie arehart
week). Among its improvements is mentioned a specific one related to this
issue, labeled "socketWrite0 protection". I only heard a little about it
during the beta cycle, so hopefully one of the Intergral folks will step in
to explain it better. I couldn't find more on it in the release notes or on
the FR site. In the meantime, let us know if you are ok trying the upgrade
to report back if it makes them go away.
If it doesn't, then I would propose you do some analysis to determine if
there's any pattern: time of day? Kind of request? Specific URL? It could be
about the kind of data being sent to the user, or it could be about the kind
of user agent (which you can view either in the request details while the
request is running or in the request history, or in the web server log,
though sadly it's not tracked in the FR request log.) But hopefully instead
it's the issue fixed by 3.01.
/charlie
Darren Pywell
This problem is common to many J2EE servers but seems to affect some
ColdFusion servers on a more regular basis.
socketWrite0() is the native C method that is called to actually send
the content from the web page down the socket to the client. The
problem is that sometimes the client stops consuming data but keeps
the socket open effectively keeping a thread locked. When multiple
requests do this then eventually you run out of threads and the server
hangs. There are a number of web browsers out there with bugs that
cause this (including some older versions of Internet Explorer).
We've been working on a new but still experimental crash protection
method called socketWrite0 protection that tries to prevent this
problem in FusionReactor version 3.0.1. This crash protection method
is NOT active by default (because it is still experimental as I
mentioned) and cannot be enabled via the UI. If you want to give this
a try then you need to upgrade to FusionReactor 3.0.1 and then:
1. Stop your ColdFusion/J2EE server
2. Make a backup of the reactor.conf file for the instance that you
want to try this with
3. Add the following line to the end of reactor.conf file for the
instance that you want to try this with
crashprotection.buffering.active=true
4. Restart the server
If you have any issues then stop the server again, restore the backup
of reactor.conf and restart the server.
Please remember that although we have tested this feature it is still
regarded as experimental so you are activating this at your own
risk :-). If your server is hanging or crashing on a regular basis
though you may consider trying this to see if it helps you.
Hope that helps,
Darren
On Dec 5, 4:46 am, "andre...@moon.com.au" <andre...@moon.com.au>
wrote:
charlie arehart
have by enabling this? What are the risks, in other words? :-) Or more
important, how apparent are any such risks? Are there any which will be
obvious, either to site visitors or per the FR interface, logs, or perhaps
CF logs?
Though it's experimental, I hope you'll consider creating a technote,
whether with what you have or perhaps adding any answers to the above.
Thanks.
/charlie
-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of Darren Pywell
Sent: Friday, December 05, 2008 4:17 PM
To: FusionReactor
Darren Pywell
size of the pages that you generate). If you generate pages which are
Mega Bytes big then it could be that you notice this, but otherwise I
don't expect there to be too much of an impact.
Also, if you perform a flush / cfflush the content may not be flushed
exactly at that moment if we are unable to send the content down the
socket at that time.
Other than this we don't know of any other side effects but we are
fielding this as experimental because the socketWrite0 issue can occur
in so many complex ways for so many reasons. We certainly can't
guarantee that we have covered them all (we may never because the
problem is often caused by bugs and interplay issues between the
server and the client).
We have tested the feature as thoroughly as we could but we are
leading the way with this.
We will of course be producing a Technote but with Andrew highlighting
the problem here on the groups I didn't want to wait until that's
ready to try to help out :-)
Cheers,
Darren
On Dec 5, 11:18 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:
charlie arehart
/charlie
-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of Darren Pywell
Sent: Friday, December 05, 2008 5:51 PM
To: FusionReactor
Subject: FusionReactor Group: Re: JrppBufferedOutputStream locked
andr...@moon.com.au
socketWrite0 protection on, firstly on our dev box just to make sure
it doesn't do anything weird for us. I really appreciate your help on
this - our clients are starting to get very upset with us.
Cheers,
Andrew
charlie arehart
discussing some gems in the 3.0.1 update:
http://carehart.org/blog/client/index.cfm/2008/12/7/fusionreactor301_release
d_some_gems
In particular, I elaborate a bit on the improvements:
- Reports if you have any URLs set to be ignored
- Can re-run installer to add FR into additional instances/servers
- New URL params for redirected CP URLs
And a couple of others. I didn't cover them all, and might not have covered
some that deserve to be pointed out, so feel free to comment on the blog to
share any other thoughts (or reply here, but if it would benefit other
interested FR 3.0.1 updating users to read your comment, it would probably
be best to put them on the blog entry).
I mentioned there that I'd do another entry with more on the socketWrite0
protection experimental/optional feature, which Darren discussed some here
in recent days. I also share a slight caution about reading the descriptions
of the improvements. Sometimes you may read them differently if you realize
that they're written as descriptions of entries in a soure code control
system, so some describe what was wrong that's been improved.
/charlie
Darren Pywell
That is the right approach. We need feedback from users like you to
let us know if socketWrite0 protection does the job. If it improves
things (or not) please let us know.
Thanks!
Darren
On Dec 7, 11:08 pm, "andre...@moon.com.au" <andre...@moon.com.au>
wrote:
Darren Pywell
Good point on the descriptions on improvements etc. Tickets are often
submitted under titles from users, support emails and diverse other
sources. The ticket names don't always perfectly reflect the actual
improvement, fix, bug, new feature etc. We do improve them sometimes
and we'll keep an eye on this in the future.
Cheers,
Darren
On Dec 7, 11:09 pm, "charlie arehart" <charlie_li...@carehart.org>
wrote:
> discussing some gems in the 3.0.1 update:
>
charlie arehart
the note below), and thought some here may benefit seeing the additional
info:
----
Here's a bit of a hidden gem (I don't see it listed in the release notes,
but I may not be finding the right wording.)
For those using FR Enterprise, on the Enterprise Settings page, what used to
be a single entry for "Network Timeout" in a section of the page called
"Remote Instances" is now 3 entries in a section called "Heartbeat Settings
for Monitored Instances". That network timeout is now called Heartbeat
Timeout, and there is also a "Heartbeat Metrics Fetch Interval" and a
"Heartbeat Failure Threshold".
There is considerable text on the screen (and the same is in the online
help) explaining each of these new features. The bottom line is that they
help improve the process of one monitoring FR instance detecting (for its
Enterprise Dashboard and notifications) when another server/instance that
it's monitoring has become unavailable. The new settings should help prevent
(or help you tune how to prevent) false positives.
----
/charlie
-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
charlie arehart
were going to enable that experimental socketwrite0 fix that was introduced
as an option in FR 3.0.1. Did you end up enabling it? Did it help? Did it
hurt anything? Inquiring minds (someone who read this thread and asked me)
want to know. :-)
Perhaps the FR guys might have some new insights to share on how its gone
for those who have tried this.
/charlie
-----Original Message-----
From: fusion...@googlegroups.com [mailto:fusion...@googlegroups.com]
On Behalf Of andr...@moon.com.au
Sent: Sunday, December 07, 2008 5:08 PM
To: FusionReactor
Subject: FusionReactor Group: Re: JrppBufferedOutputStream locked