ACF10 and Service Temporary Unavailable

[hofar...@houseoffusion.com]

Since upgrading from ACF8 to ACF10, every once in a while (about once every
2-3 days?), under normal load, all .cfm reqs produce:

Service Temporary Unavailable!

The server is temporarily unable to service your request due to maintenance
downtime or capacity problems. Please try again later.

Jakarta/ISAPI/isapi_redirector/1.2.32 ()

Copyright � 1999-2011 Apache Software Foundation
All Rights Reserved

...I'm using IIS. Restarting the CF Server service makes the problem go
away.

Ideas?

--
John Bliss - http://about.me/jbliss

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion
Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355534

[hofar...@houseoffusion.com]

[crickets] :-)

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355540

[hofar...@houseoffusion.com]

On Tue, Apr 23, 2013 at 10:00 AM, John M Bliss <bliss...@gmail.com> wrote:

> [crickets] :-)


Usually when the web connector starts failing, I rebuild it. Sometimes I
never know WHY it started failing. That would be my first step.

-Cameron

...

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355541

[hofar...@houseoffusion.com]

I'm going to guess that Tomcat isn't being allotted enough memory.

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355542

[hofar...@houseoffusion.com]

I'm not using Tomcat. I'm using IIS.

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355543

[hofar...@houseoffusion.com]

If you're using CF10, you're using Tomcat.

On Tue, Apr 23, 2013 at 10:18 AM, John M Bliss <bliss...@gmail.com> wrote:

>
> I'm not using Tomcat. I'm using IIS.
>
>
> On Tue, Apr 23, 2013 at 9:15 AM, Matt Quackenbush <quack...@gmail.com
> >wrote:
>
> >
> > I'm going to guess that Tomcat isn't being allotted enough memory.
> >
> >
> > On Tue, Apr 23, 2013 at 10:00 AM, John M Bliss <bliss...@gmail.com>
> > wrote:
> >
> > >
> > > [crickets] :-)
> > >
> > >
> > > On Mon, Apr 22, 2013 at 8:23 PM, John M Bliss <bliss...@gmail.com>
> > > wrote:
> > >
> > > > Since upgrading from ACF8 to ACF10, every once in a while (about once
> > > > every 2-3 days?), under normal load, all .cfm reqs produce:
> > > >
> > > > Service Temporary Unavailable!
> > > >
> > > > The server is temporarily unable to service your request due to
> > > > maintenance downtime or capacity problems. Please try again later.
> > > >
> > > > Jakarta/ISAPI/isapi_redirector/1.2.32 ()
> > > >

> > > > Copyright Š 1999-2011 Apache Software Foundation

> > > > All Rights Reserved
> > > >
> > > > ...I'm using IIS. Restarting the CF Server service makes the problem
> go
> > > > away.
> > > >
> > > > Ideas?
> > > >
> > > > --
> > > > John Bliss - http://about.me/jbliss
> > > >
> > >
> > >
> > >
> > > --
> > > John Bliss - http://about.me/jbliss
> > >
> > >
> >
> >
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355544

[hofar...@houseoffusion.com]

My bad. How do I allot more memory to Tomcat?

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355545

[hofar...@houseoffusion.com]

are you using the default jvm settings ? maybe they need increasing too

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355546

[hofar...@houseoffusion.com]

I've never looked at CF10 so I'm not familiar with whatever packaging Adobe
decided to go with, but in a normal Tomcat installation on Linux (have no
idea about Windows, either, sorry) it'd be in /etc/default/tomcat7.

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355547

[hofar...@houseoffusion.com]

Tomcat stuff appears to be in C:\ColdFusion10\cfusion\runtime\lib but there
are no config files in there. Anyone have an idea as to where to look (for
the file to edit to give Tomcat more memory)?

Russ, WRT JVM, I changed it to Minimum JVM Heap Size (in MB): 1024 and
Maximum JVM Heap Size (in MB): 1024.

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355548

[hofar...@houseoffusion.com]

> Tomcat stuff appears to be in C:\ColdFusion10\cfusion\runtime\lib but there
> are no config files in there. Anyone have an idea as to where to look (for
> the file to edit to give Tomcat more memory)?
>
> Russ, WRT JVM, I changed it to Minimum JVM Heap Size (in MB): 1024 and
> Maximum JVM Heap Size (in MB): 1024.

Tomcat is the J2EE application server used to run CF 10 by default. If
you've given CF more memory, you've given Tomcat more memory. You
don't have to poke around in a different area for that.

That said, I doubt that this is a memory problem, unless you've also
seen CF consume the entire heap when you've had this problem. Like
Cameron said, this sounds like a connector problem. Have you installed
the latest CF 10 update? When you install these updates, I think you
need to uninstall and reinstall your connectors to get to the latest
version of those. I believe that's described in the release notes.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355549

[hofar...@houseoffusion.com]

ACF10 uses the isapi_redirect.dll for connecting Tomcat to IIS which is
an open-source connector developed by the good folks in the Tomcat
project. The error message is coming from the connector, so what you can
do is enable logging on the connector and it should tell you what the
problem is.

You can enable logging (or set the logging level to a be more
informative) by editing the connector properties file, usually located here:

{CF-HOME}\config\wsconfig\{magic-number}\isapi_redirect.properties

Specifically, look for the following entries:

log_file = {CF-HOME}\config\wsconfig\{magic-number}\isapi_redirect.log
log_level = info

If the default log_level of "info" doesn't indicate anything in the
logs, try setting the level higher. Appropriate values are documented here:

http://tomcat.apache.org/connectors-doc/reference/iis.html

... but chances are good the problem is already being logged.

If I had to guess, I would say you're bumping up against the default
"connection_pool_size" which unfortunately defaults to 250 for IIS. I've
seen this limit hit frequently on even low-use windows machines.

The connection_pool_size attribute can be configured in the
workers.properties file, located here by default:

{CF-HOME}\config\wsconfig\{magic-number}\

The connection_pool_size attribute is documented here:
http://tomcat.apache.org/connectors-doc/reference/workers.html

And you can read about manually installing the connector for ACF10 here:
http://helpx.adobe.com/coldfusion/kb/coldfusion10-iis-manual-connector-configuration.html

Hope this helps John! =)

Warm Regards,
Jordan Michaels

On 04/22/2013 06:23 PM, John M Bliss wrote:
>
> Since upgrading from ACF8 to ACF10, every once in a while (about once every
> 2-3 days?), under normal load, all .cfm reqs produce:
>
> Service Temporary Unavailable!
>
> The server is temporarily unable to service your request due to maintenance
> downtime or capacity problems. Please try again later.
>
> Jakarta/ISAPI/isapi_redirector/1.2.32 ()
>
> Copyright � 1999-2011 Apache Software Foundation
> All Rights Reserved
>
> ...I'm using IIS. Restarting the CF Server service makes the problem go
> away.
>
> Ideas?

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355553

[hofar...@houseoffusion.com]

> When you install these updates, I think you need to uninstall and
reinstall your connectors to get to the latest version of those. I believe
that's described in the release notes.

In CF Admin > Server Update > Updates, for Update 9, it reads, "You might
have to reconfigure the connector in certain cases. For more details refer
to the tech note by clicking on "Read More"."

When I click "Read More," I go to
http://www.adobe.com/support/security/bulletins/apsb13-10.html which says
nothing about the connector. Do you still recommend uninstalling and
reinstalling?

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355589

[hofar...@houseoffusion.com]

> Do you still recommend uninstalling and reinstalling?

Yes, I recommend you uninstall and reinstall the connectors. That is
the only way to ensure you have the latest version of the connectors.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355590

[hofar...@houseoffusion.com]

OK. And I can do that via:

1. Web Server Configuration Tool
2. highlight current configured web server
3. click Remove
4. click Add...
5. click OK

...right? Do I need to stop CF Server service first?

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355591

[hofar...@houseoffusion.com]

Jordan (or anyone else :-), here is a snip of my isapi_redirect.log from
just before the last fail. From this, can you see what caused it? Dave's
"non-updated connector" issue? Your "connection_pool_size" issue?

http://www.brandiandjohn.com/codeshare/isapi_redirect.log

On Tue, Apr 23, 2013 at 10:59 AM, Jordan Michaels <jor...@viviotech.net>wrote:

> ACF10 uses the isapi_redirect.dll for connecting Tomcat to IIS which is an
> open-source connector developed by the good folks in the Tomcat project.
> The error message is coming from the connector, so what you can do is
> enable logging on the connector and it should tell you what the problem is.
>
> You can enable logging (or set the logging level to a be more informative)
> by editing the connector properties file, usually located here:
>

> {CF-HOME}\config\wsconfig\{**magic-number}\isapi_redirect.**properties

>
> Specifically, look for the following entries:
>

> log_file = {CF-HOME}\config\wsconfig\{**magic-number}\isapi_redirect.**log

> log_level = info
>
> If the default log_level of "info" doesn't indicate anything in the logs,
> try setting the level higher. Appropriate values are documented here:
>

> http://tomcat.apache.org/**connectors-doc/reference/iis.**html<http://tomcat.apache.org/connectors-doc/reference/iis.html>

>
> ... but chances are good the problem is already being logged.
>
> If I had to guess, I would say you're bumping up against the default
> "connection_pool_size" which unfortunately defaults to 250 for IIS. I've
> seen this limit hit frequently on even low-use windows machines.
>
> The connection_pool_size attribute can be configured in the
> workers.properties file, located here by default:
>

> {CF-HOME}\config\wsconfig\{**magic-number}\

>
> The connection_pool_size attribute is documented here:

> http://tomcat.apache.org/**connectors-doc/reference/**workers.html<http://tomcat.apache.org/connectors-doc/reference/workers.html>

>
> And you can read about manually installing the connector for ACF10 here:

> http://helpx.adobe.com/**coldfusion/kb/coldfusion10-**
> iis-manual-connector-**configuration.html<http://helpx.adobe.com/coldfusion/kb/coldfusion10-iis-manual-connector-configuration.html>

>
> Hope this helps John! =)
>
> Warm Regards,
> Jordan Michaels
>
>
> On 04/22/2013 06:23 PM, John M Bliss wrote:
>
>>
>> Since upgrading from ACF8 to ACF10, every once in a while (about once
>> every
>> 2-3 days?), under normal load, all .cfm reqs produce:
>>
>> Service Temporary Unavailable!
>>
>> The server is temporarily unable to service your request due to
>> maintenance
>> downtime or capacity problems. Please try again later.
>>

>> Jakarta/ISAPI/isapi_**redirector/1.2.32 ()
>>
>> Copyright Š 1999-2011 Apache Software Foundation

>> All Rights Reserved
>>
>> ...I'm using IIS. Restarting the CF Server service makes the problem go
>> away.
>>
>> Ideas?
>>
>>

--
John Bliss - http://about.me/jbliss

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355592

[hofar...@houseoffusion.com]

> OK. And I can do that via:
>
> 1. Web Server Configuration Tool
> 2. highlight current configured web server
> 3. click Remove
> 4. click Add...
> 5. click OK
>
> ...right? Do I need to stop CF Server service first?

You don't need to stop CF. However, in between uninstalling and
reinstalling, I'd close the web server configuration tool and reopen
it, just to be sure.

Dave Watts, CTO, Fig Leaf Software
http://www.figleaf.com/
http://training.figleaf.com/

Fig Leaf Software is a Veteran-Owned Small Business (VOSB) on
GSA Schedule, and provides the highest caliber vendor-authorized
instruction at our training centers, online, or onsite.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355593

[hofar...@houseoffusion.com]

John,

Looks like you're hitting the connection_pool_size issue:

[Mon Apr 22 20:20:13.670 2013] [5012:5128] [info]
ajp_process_callback::jk_ajp_common.c (2058): current reuse count is 7
of max reuse connection 250 and total endpoint count 250

The 250 limit is the default limit for IIS, as mentioned here:
http://tomcat.apache.org/connectors-doc/reference/workers.html

Under the "connection_pool_size" attribute:
"For IIS the default value is 250 (before version 1.2.20: 10), for
Netscape/Sun the default value is 1."

When these are maxed out, you'll get the "cannot connect to backend" and
"failed to open socket" errors you're getting.

Unfortunately, updating the connectors won't help this situation. You'll
need to adjust your connector settings.

You could also potentially review your code. See if there's any place
you can reduce the number of calls you make to CF - like, if possible,
combine redundant or similar calls into a single call, and make sure IIS
is handling simple calls for static resources and such and not passing
them back to CF for some reason.

Hope this helps!

Warm Regards,
Jordan Michaels
Vivio Technologies

>>> Copyright � 1999-2011 Apache Software Foundation

>>> All Rights Reserved
>>>
>>> ...I'm using IIS. Restarting the CF Server service makes the problem go
>>> away.
>>>
>>> Ideas?
>>>
>>>
>
>

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355611

[hofar...@houseoffusion.com]

Thanks!

On Fri, Apr 26, 2013 at 12:51 PM, Jordan Michaels <jor...@viviotech.net>wrote:

> John,
>
> Looks like you're hitting the connection_pool_size issue:
>
> [Mon Apr 22 20:20:13.670 2013] [5012:5128] [info]

> ajp_process_callback::jk_ajp_**common.c (2058): current reuse count is 7

> of max reuse connection 250 and total endpoint count 250
>
> The 250 limit is the default limit for IIS, as mentioned here:

> http://tomcat.apache.org/**connectors-doc/reference/**workers.html<http://tomcat.apache.org/connectors-doc/reference/workers.html>

>
> Under the "connection_pool_size" attribute:
> "For IIS the default value is 250 (before version 1.2.20: 10), for
> Netscape/Sun the default value is 1."
>
> When these are maxed out, you'll get the "cannot connect to backend" and
> "failed to open socket" errors you're getting.
>
> Unfortunately, updating the connectors won't help this situation. You'll
> need to adjust your connector settings.
>
> You could also potentially review your code. See if there's any place you
> can reduce the number of calls you make to CF - like, if possible, combine
> redundant or similar calls into a single call, and make sure IIS is
> handling simple calls for static resources and such and not passing them
> back to CF for some reason.
>
> Hope this helps!
>
> Warm Regards,
> Jordan Michaels
> Vivio Technologies
>
>
>
> On 04/25/2013 08:52 AM, John M Bliss wrote:
>
>>
>> Jordan (or anyone else :-), here is a snip of my isapi_redirect.log from
>> just before the last fail. From this, can you see what caused it? Dave's
>> "non-updated connector" issue? Your "connection_pool_size" issue?
>>

>> http://www.brandiandjohn.com/**codeshare/isapi_redirect.log<http://www.brandiandjohn.com/codeshare/isapi_redirect.log>

>>
>>
>> On Tue, Apr 23, 2013 at 10:59 AM, Jordan Michaels <jor...@viviotech.net
>> >wrote:
>>
>> ACF10 uses the isapi_redirect.dll for connecting Tomcat to IIS which is
>>> an
>>> open-source connector developed by the good folks in the Tomcat project.
>>> The error message is coming from the connector, so what you can do is
>>> enable logging on the connector and it should tell you what the problem
>>> is.
>>>
>>> You can enable logging (or set the logging level to a be more
>>> informative)
>>> by editing the connector properties file, usually located here:
>>>

>>> {CF-HOME}\config\wsconfig\{****magic-number}\isapi_redirect.***

>>> *properties
>>>
>>>
>>> Specifically, look for the following entries:
>>>

>>> log_file = {CF-HOME}\config\wsconfig\{****magic-number}\isapi_redirect.*
>>> ***log

>>>
>>> log_level = info
>>>
>>> If the default log_level of "info" doesn't indicate anything in the logs,
>>> try setting the level higher. Appropriate values are documented here:
>>>

>>> http://tomcat.apache.org/****connectors-doc/reference/iis.****html<http://tomcat.apache.org/**connectors-doc/reference/iis.**html>
>>> <http://tomcat.apache.**org/connectors-doc/reference/**iis.html<http://tomcat.apache.org/connectors-doc/reference/iis.html>

>>> >
>>>
>>>
>>> ... but chances are good the problem is already being logged.
>>>
>>> If I had to guess, I would say you're bumping up against the default
>>> "connection_pool_size" which unfortunately defaults to 250 for IIS. I've
>>> seen this limit hit frequently on even low-use windows machines.
>>>
>>> The connection_pool_size attribute can be configured in the
>>> workers.properties file, located here by default:
>>>

>>> {CF-HOME}\config\wsconfig\{****magic-number}\

>>>
>>>
>>> The connection_pool_size attribute is documented here:

>>> http://tomcat.apache.org/****connectors-doc/reference/****workers.html<http://tomcat.apache.org/**connectors-doc/reference/**workers.html>
>>> <http://tomcat.**apache.org/connectors-doc/**reference/workers.html<http://tomcat.apache.org/connectors-doc/reference/workers.html>

>>> >
>>>
>>>
>>> And you can read about manually installing the connector for ACF10 here:

>>> http://helpx.adobe.com/****coldfusion/kb/coldfusion10-**<http://helpx.adobe.com/**coldfusion/kb/coldfusion10-**>
>>> iis-manual-connector-****configuration.html<http://**
>>> helpx.adobe.com/coldfusion/kb/**coldfusion10-iis-manual-**
>>> connector-configuration.html<http://helpx.adobe.com/coldfusion/kb/coldfusion10-iis-manual-connector-configuration.html>

>>> >
>>>
>>>
>>> Hope this helps John! =)
>>>
>>> Warm Regards,
>>> Jordan Michaels
>>>
>>>
>>> On 04/22/2013 06:23 PM, John M Bliss wrote:
>>>
>>>
>>>> Since upgrading from ACF8 to ACF10, every once in a while (about once
>>>> every
>>>> 2-3 days?), under normal load, all .cfm reqs produce:
>>>>
>>>> Service Temporary Unavailable!
>>>>
>>>> The server is temporarily unable to service your request due to
>>>> maintenance
>>>> downtime or capacity problems. Please try again later.
>>>>

>>>> Jakarta/ISAPI/isapi_****redirector/1.2.32 ()

>>>>
>>>>
>>>> Copyright � 1999-2011 Apache Software Foundation
>>>> All Rights Reserved
>>>>
>>>> ...I'm using IIS. Restarting the CF Server service makes the problem go
>>>> away.
>>>>
>>>> Ideas?
>>>>
>>>>
>>>>
>>
>>

--
John Bliss - http://about.me/jbliss

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~|
Order the Adobe Coldfusion Anthology now!
http://www.amazon.com/Adobe-Coldfusion-Anthology/dp/1430272155/?tag=houseoffusion

Archive: http://www.houseoffusion.com/groups/cf-talk/message.cfm/messageid:355612