Editing CF8.0.1 Sandbox Settings Slows Down IIS7
elarin
I'm running CF8.0.1 on Windows Server 2008 Standard (all 32-bit) and our web
site content is located on a UNC share. When I log into ColdFusion
Administrator and perform any task under Sandbox Security (i.e. adding a
sandbox, editing properties of existing sandboxes), our web site comes to a
crawl and becomes very unresponsive. Sometimes the behavior does not correct
itself automatically forcing me to restart the WWW and CF services on the
server.
Does anyone else experience such a behavior? I don't know if it's isolated to
Windows 2008 or CF8.0.1. Seems like this is somewhat of a performance bug. Yes,
I know I can manipulate sandboxes via adminapi. In fact, we do that most of the
time. However, there are circumstances when I need to edit them within CF admin.
Overall, our performance has declined a little since we moved from ColdFusion
7.0.2 running on Windows 2003. This is somewhat disappointing.
Erick
BKBK
carehart
of the details at
http://russ.michaels.me.uk/index.cfm/2009/3/19/ColdFusion-8-performance-Issues-w
hen-using-Java-6. Bottom line, it was related to the oft-mentioned problem of a
class loading bug in Java, fixed by upgrading the JVM to 1.6.10+ or back to
Java 1.5. The blog entry mentioned (and many others) talk about how to do that.
My theory is that making changes in the sandbox may force Java (the sandbox
security is built atop the underlying java security manager) to unload and
reload the classes for CF pages in any sandboxes (I say "any", because he was
seeing that changes to ANY sandbox affected performance of ALL templates, and
in his case he had a sandbox defined for all CF apps.)
Let us know if that's the solution for you, Elarin.
elarin
Thank you for providing me the link to that page as it does seem I'm not the
only one having this issue. I am currently running JVM 1.6.0_12 on the server.
I did not keep the default 1.6.0_4 that ships with CF 8.0.1 due to the class
bug.
In that page, there's mention of Cumulative Hot Fix 2 that states something
about memory leaks. Perhaps I should download and install that.
Erick
carehart
while I addressed this. All he did was do the JVM update, and things were much
better. He must have applied the hotfix the next day. We wondered if it was
possible that the improvement we were seeing immediately may have had anything
to do with the server being more responsive as much because it had just been
restarted. We didn't dig in to confirm (via jvm debugging output showing class
loading details) whether and how the jvm update may have helped. It could
certainly be that the other fix was as or more important, so sure, please do
apply it and let us know. (I would wonder if it's really the combination of the
two that's key.)
/charlie
elarin
(.cfm, .htm, .asp) would either time out or take minutes to complete. What I
saw was the OS thread count for the application pool process (w3wp.exe)
assigned to my site keep increasing and the jrun process sat at 0% CPU
utilization. I understand CF runs under the jrun.exe process, but an
application pool (w3wp.exe process) does seem to come into play for CF
requests. You can see this by viewing current requests under "Worker
Processes". I know this means IIS is simply getting the request first and then
handing off to CF. So what I'm experiencing is a contrast to most people saying
CF and IIS application pools are not related. They seem to be with IIS 7 and CF
8. I had to ditch the server and revert back to my old server with Windows 2003
and CF 7.
In conclusion, I think there's a serious compatibility issue between Windows
2008 and CF 8, maybe isolated to its 32-bit counterparts. I say this because a
quick test of both 64-bit versions appears to perform as expected. However,
64-bit is not an option for us.
Erick
BKBK
with the sandbox, hence with Coldfusion's built-in objects and classes. If
Coldfusion objects cause performance problems, then the obvious place to verify
this is the monitor. Hence my suggestion earlier for you to [i]go to the Server
Monitor section of the Coldfusion Administrator, and examine the Request
Statistics and Memory Usage.[/i]
If the monitor tells you performance is good, then the likely cause is IIS. In
IIS, the likely suspect is the application pool.
elarin
Security. My web server runs great when it's disabled. The moment I enable it,
performance degrades quickly. The bad news is I don't know how to fix this. I
need it enabled to I can restrict tags/functions at a global level. I'm
thinking my main sandbox needs more files/folders added to its list. I
currently have the following:
<root of my web content>
<root of my web content>\-
C:\Windows\fonts\*
D:\ColdFusion8\runtime\servers\coldfusion\SERVER-INF\temp\wwwroot-tmp\-
Just having those 4 worked with CF7. What else do I need to add for CF8?
Erick
elarin
and no matter what I do, I'm getting a performance hit with sandbox security
enabled. It seems sandbox security with CF8 is slower than with CF7 no matter
what. I wonder if this is a known issue and/or there's a hotfix available?!
Erick
carehart
for 8.0.1, as discussed early on in this thread?
The thread has clarified from the beginning that the issue is due to Sandbox
security. No new conclusion there. :-) But since you say it worked fine on 7
and not on 8, the 2 proposed solutions have worked for others.
Just for completeness sake, can you look in your CF Admin and confirm in the
System Settings page both that you see that 1.6.0_12 jvm level (you said you're
running) indicated in the "java version" field, and (if you did apply the
hotfix) that you see the "update level" pointing to lib/updates/chf8010002.jar?
I'm not doubting your integrity or intelligence. :-) I'm just being diligent,
so we don't go chasing some other problem if in fact either of these wasn't
applied. If those are both applied, then we do have a curious additional cause
in your case. I'd love to help see this resolved.
I have another thought I'll share separately, as this one is long enough
already.
carehart
and the CHF2, the next thought I have is that you could do some confirmation of
what's slowing the pages down, specifically, by watching them run using a tool
like FusionReactor or SeeFusion. (You said you're running on Standard, so you
can't use the CF 8 Server Monitor, since it's Enterprise/Developer only.)
Both FR and SF have free trials. It may be worth your checking them out.
Besides showing what requests are running at any point in time, you can also
ask each tool to show you a stack trace for a single request, which can show
you exactly what line of CFML the request is running. If you repeat that and
the request is stuck on the same line, now you have a smoking gun to
investigate.
In working with Russ, though, we never saw it stuck on any one line, which is
what led us ultimately to think outside the box and I proposed the JVM update.
If that's not your issue, I wonder if you may see different information
revealed in stack tracing the requests (or at least confirming for yourself the
"slow" pages appearing in the request monitor tools).
They can be a great diagnostic, as sometimes you DON'T see slow requests in
there running, which then tells you that the problem is somewhere else, like
the web server or the web server connector, etc. (I do realize you wouldn't
expect that to be the case with your problem, tied so clearly as it is to the
Sandbox.)
Let us know what you think of this idea.
elarin
CHF2 has been applied. CHF2 does not seem to have helped. When I first posted
this, I thought my performance issue was restricted to editing sandboxes in
CFAdmin. But now I've confirmed the fact of having sandbox security enabled
itself is harmful.
Erick
carehart
tell you why requests are slow?
And I was mistaken in something I said. I had looked at your first note where
you said you were running "Windows Server 2008 Standard", not CF 8 Standard. In
fact, had I been thinking, the fact that you are using Sandbox Security should
have clued me that you're running Enterprise, since it's in Enterprise only.
Have you tried the active requests page, to see what requests are running (if
you enable "start monitoring")? And if you drill into a request while "Start
Profiling" is enabled, that will show you the stack trace for that request at
that moment.
For more on using the monitor, I'll point out that I did a 4-part series of
articles on the CF 8 server monitor, starting at
http://www.carehart.org/articles/#2007_2.
For those not on CF 8 Enterprise, again, the other tools can be valuable. Some
problems are just not easily resolved without them. Again, the tools are free
to try and generally very easy to install and use. I should have offered URLs
for them: http://www.seefusion.com and http://www.fusion-reactor.com.
elarin
I have used the Server Monitor tool in CF8 and it's not helping out. Memory
usage is low and nothing is sticking out in any of the requests' stack trace.
In fact, sometimes I can't view Active Requests and get kicked out of Server
Monitor because CF slows down or gets backed up too much.
Using ProcessMonitor from sysinternals helped me track down the issue to the
sandbox security. When sandbox security is off, I see the jrun.exe process go
immediately to the required file(s) to serve out the resulting page. Remember,
my content is on a UNC share. So for example, it goes to
"\\myFiler\shareroot\web\application\index.cfm".
With sandbox security on, ProcessMonitor shows jrun.exe "traversing" the
entire folder structure for each required file. I see it going to
"\\myFiler\shareroot" then to \\myFiler\shareroot\web" and so on. So imagine a
single CF template with many cfincludes to other UNC paths. And CF also
traverses to files in its "ColdFusion8" program folder. I understand that's how
file retrievals are implemented with sandbox security. I just feel there's an
underlying I/O issue with CF8 that's maybe compounded with UNC content.
Erick
carehart
memory. But if you say the stack traces don't show requests to be stuck at the
same point for extended periods, ok.
Your observation about the folder traversal is interesting. Most would expect
to see traversal "up" the structure when looking for the application.cfm, but
this reverse order I would guess might instead reflect the fact that you either
have multiple sandboxes (for the levels you see it traversing). Is that the
case?
As for it going to the CF8 directory, is that the wwwroot directory, or
further down into its WEB-INF.Just seems worth clarifying. That's a really good
diagnostic you've done. I've not done it before myself so don't know what's
normal. The UNC path issue could well be significant as well.
One last thing, though: you say "When sandbox security is off, I see the
jrun.exe process go immediately to the required file(s) to serve out the
resulting page." That's a little curious, in that I wouldn't expect it to "go
to" each page at least to "run" the page, in that it should find the page in
the template cache. But it would also look at the directory where a file lives
when it did the check before each execution to see if the file source had
changed. This is what the "trusted cache" option is for: if you know that the
source is not changing. I would wonder if things would change dramatically for
you if that was turned on, even if just temporarily for you to test things.
elarin
files located in folders like "C:\ColdFusion8\lib" and
"C:\ColdFusion8\wwwroot\WEB-INF".
I'm very familiar with the Trusted Cache option as we enable it for our
intranet sites. Unfortunately that's not an option with this particular server
as our web developer audience is so spread out.
If you're interested Charlie, I can email you trace files from ProcessMonitor
which better show the folder traversing behavior I'm seeing.
Erick
carehart
temporarily for you to test things."
Would you be willing to try it just briefly to see if it has an impact on your
problem? Could help identify a solution or workaround (or bug).
BTW, as far as the fear many have over using the trusted cache, and the
concern that developers need to be able to easily implement updates to code,
there's a nice solution to that in CF8. Ray Camden blogged about it and how in
combination with the filewatcher gateway, CF can automatically pick up changes
to code and clear the cache for those specifically changed files. More here:
http://www.coldfusionjedi.com/index.cfm/2008/6/19/Clearing-individual-filesfolde
rs-from-ColdFusion-templates-cache
http://www.coldfusionjedi.com/index.cfm/2007/6/7/ColdFusion-8-Admin-API-and-Trus
ted-Cache
elarin
Securtiy enabled as well. It did seem to help my performance some. Using
ProcessMonitor, the reason why it's a little faster is jrun.exe process is not
traversing down through folders looking for application.cfc or application.cfm
templates. It still traverses down to the specified ColdFusion template and a
few ColdFusion .jar files.
This option is enabled for our intranet sites and our CF admin for that area
already implements a clear cache automated routine via adminapi. However, I
know enabling this option on my production CF server will not fly with our web
developers and managers.
So where I'm at right now is my production server is running with Sandbox
Security and Trusted Cache off.
Erick
carehart
longer looks for each template (not just application.cfc/cfm) before it
executes it, if it's already in the cache.
And BTW, you could still have it looking for files, if they're not in the
cache, which could happen if the cache is not sized appropriately for the
number of templates that get put in there. Just being clear. In fact, I've seen
people do some strange coding practices that REALLY hammered the template
cache, so that even with trusted cache they were still causing that IO to the
files as the template cache was clearing out files often to make room for newly
requested ones. At least in CF 8 Enterprise, the Server Monitor (and Admin API)
report the template cache hit ratio, so you can know if this is happening. We
used to have that in CF5 and before, too, in the CFSTAT, but it reports 0 since
CF6.
Anyway, just one last thing about your observation of trying to run it in
production. You conclude that it just won't fly, but are you saw you are aware
of an automated approach using the Admin API. Do you know if it goes the extra
step of using the CF Directory Watcher event gateway? If so, there's really no
reason for the devs to balk. It's an amazing clean solution. More here:
http://www.adobe.com/devnet/coldfusion/articles/cacheclear.html
Hope that's helpful.
Back to your original issue, it still seems that having the sandbox on is
adding a burden on your machine. We need to get to the bottom of that. It may
be that there's something unique about your setup (or volume) compared to
others, or perhaps there's a known issue that I'm just not aware of myself.