Apache proxy to AJP much slower than hitting Tomcat directly

[Brad Wood]

This less of a Railo question and more of an Apache/Tomcat question but I've Googled a bit without results so I'm hoping someone here might know what's wrong.

Quite simply, I'm seeing a significant overhead of of proxying my requests through Apache and I'm hoping I just have something configured wrong.

I have an empty site with nothing but a blank index.cfm.  Railo 4.1.0.005 beta in installed on Tomcat7 using the common jars method.  If I just do a basic hit to the index file on port 8080, Tomcat processes the request directly.

http://site.com:8080

SeeFusion shows the request took 0ms and Chrome's developer tools shows a total of 1ms total request time.  No problem here.

I also have Apache 2.4.3 installed as part of XAMPP with mod_proxy.so and mod_proxy_ajp.so installed.  I have a minimalistic VirtualHost and this block of config above my VirtualHosts:

<Proxy *>

Allow from 127.0.0.1

</Proxy>

ProxyPreserveHost On

ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ ajp://localhost:8009/$1$2

When I hit the same empty page on port 80 as to redirect through Apache's proxy:

http://site.com:80

SeeFusion shows the request still took 0ms but Chrome's developer tools shows a total of 1000ms total request time.  That's a full second of proxy action.

I'll admit I don't know a great deal about how the proxying works-- I just copied the code from somewhere and away it went.  Any suggestions on how to pinpoint the slowness?  Any other configs I need to show on the Tomcat side to get to the bottom of this?  The Apache "access" and "error" logs don't show any errors at all.  Adobe ColdFusion is not installed on my PC at all so there are no ACF connectors anywhere.

In other news, hitting Tomcat directly with JMeter last night I was able to get about 5000 requests per second on my 4-core Intel 5.  That's pretty amazing.

Thanks!

~Brad

[Gerald Guido]

I have a feeling that it might be more of a XAMPP thing than an Apache thing. The standard instal of XAMPP comes with Tomcat. It is also meant to be a development server and thus Apache has everything is "turned on".

From the XAMPP site.. http://www.apachefriends.org/en/xampp-windows.html#1221

 As mentioned  at another place, XAMPP is not meant for production use but only for developers in a development environment. XAMPP is configured is to be as open as possible and to allow the web developer anything he/she wants. 

A total shot in the dark but the delay may be due to everything being open and/or turned on or it might be trying to connect to the built in Tomcat server.

I am running Uniform server and the difference between Tomcat port 8888 and Apache port 80 is approx 5-10 milliseconds, if that. This is a totally unscientific - me hitting refresh on the data source page in the server admin while looking at Chrome's developer tools.

G~

~Brad

--
Need help right now? Why not have one of the Railo Team help you directly: http://www.getrailo.com/index.cfm/consulting/
---
You received this message because you are subscribed to the Google Groups "Railo" group.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Gerald Guido
http://www.myinternetisbroken.com
 

[Brad Wood]

Thanks for the thoughts.  The Tomcat I'm hitting _is_ the built-in Tomcat server that came with XAMPP.  I did upgrade it from Tomcat 6 to 7 64-bit though.

I did notice a number of Apache configs being loaded.  I guess I could set up a separate Apache install outside of XAMPP with minimal configs/modules and see how the proxy behaves there.  

What's frustrating is not having any way to see what's going on under the hood with Apache.  I'm used to CF/Java where performance issues are usually just a stack trace away but Apache is kind of a black box to me.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

You received this message because you are subscribed to a topic in the Google Groups "Railo" group.

[Gerald Guido]

Another thing you could try is hitting the JSP apps that are installed with XAMPP (look in the XAMPP admin) and see you experience the same lag. That might narrow it down.

HTH

G!

[dajester2012]

In tomcat's server.xml file, look for protocol="AJP/1.3" and change it to protocol="org.apache.coyote.ajp.AjpNioProtocol". Disclaimer: apache regards this as an experimental feature, howeve r I haven't had any problems using it.

[Brad Wood]

Thanks for the suggestion.  I tried that but it didn't change the times at all.  When I get a chance, I'll try it with Apache stripped and see if that makes a difference.

[Tor-Einar Jarnbjo]

Am Mittwoch, 1. Mai 2013 22:17:23 UTC+2 schrieb Brad Wood:

This less of a Railo question and more of an Apache/Tomcat question but I've Googled a bit without results so I'm hoping someone here might know what's wrong.

Quite simply, I'm seeing a significant overhead of of proxying my requests through Apache and I'm hoping I just have something configured wrong.

It's just a guess, but could it be that your Apache is somehow configured to do a reverse DNS lookup to get the client's hostname? The reverse DNS lookup is prone to introduce strange delays, waiting for a response from the DNS server or running into timeouts when used in intranets with partially incomplete or incorrect DNS configurations.

There are unfortunately quite a few options in the Apache configuration, which may somehow enable reverse DNS lookups. First of all make sure that HostnameLookups is not set to anything but "Off" (default value), that you are not using the client host name in any log formats or that your have any rules using the client's host name (and not just the IP).

Tor

[Brad Wood]

Good ideas.  I didn't think about host name lookups.  I didn't have any HostnameLookups directives in my config, but I put one in and set it to off.  I also replaced the %h with %a in all the log formats.  Unfortunately, it doesn't appear to have had any effect.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

Tor

--
Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki
 
 
If you don't have time, add a request to the Railo Server wiki to-do page at https://github.com/getrailo/railo/wiki/Todo

[Denny]

On 5/2/13 10:21 AM, Brad Wood wrote:
> Good ideas. I didn't think about host name lookups. I didn't have any
> HostnameLookups directives in my config, but I put one in and set it to
> off. I also replaced the %h with %a in all the log formats.
> Unfortunately, it doesn't appear to have had any effect.

When I'm stumped I resort to things like tcpdump or wireshark or some
such (actually most often I use the TCP/IP proxy monitor that comes with
the Eclipse Testing Platform/Performance plugins).

There are also tools like strace, which are very nice for seeing what is
going on under the hood while the process is running.

Adding these tools to your arsenal will make you much stronger and
faster when confronted with things like this in the future. (Well,
that's the theory anyways, but it's more a rule of thumb than anything
definitive-- even I, a master debugger, get caught up in what amounts to
be a missing apostrophe or some such, for hours on end-- less often than
I used to though, I reckon. ;])

:Denny

--
Railo Technologies: getrailo.com Professional Open Source
Skype: valliantster (505)510.1336 de...@getrailo.com
GnuPG-FP: DDEB 16E1 EF43 DCFD 0AEE 5CD0 964B B7B0 1C22 CB62

[Brad Wood]

The problem I've had with TCP sniffers like WireShark in the past is they don't work unless the traffic is passing through my network card (or, quite possibly, I just don't have them configured correctly.)

In this instance, everything is local so I'm not sure WireShark would show any of the traffic.  If it did work, it would at least allow me to see how long Apache processed before handing off to AJP.  (Tomcat/Railo are known times since I hit them directly).  What I can't see is exactly what Apache is doing during the time it's processing.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

[Denny]

On 5/2/13 2:56 PM, Brad Wood wrote:
> The problem I've had with TCP sniffers like WireShark in the past is they
> don't work unless the traffic is passing through my network card (or, quite
> possibly, I just don't have them configured correctly.)

That's why the TCP/IP proxy is cool, as it uses a proxy.

> In this instance, everything is local so I'm not sure WireShark would show
> any of the traffic. If it did work, it would at least allow me to see how
> long Apache processed before handing off to AJP. (Tomcat/Railo are known
> times since I hit them directly). What I can't see is exactly what Apache
> is doing during the time it's processing.

Even local traffic is traffic, and it's easier to monitor local traffic
than remote traffic (unless you're on Windows or HP-UX/Solaris).

Using strace you should be able to pass the apache PID in and see
exactly what is going on under the covers.

[Brad Wood]

I'm on Windows :)  Does strace work there?

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

[Denny]

On 5/2/13 4:04 PM, Brad Wood wrote:
> I'm on Windows :) Does strace work there?

Here's one:

http://www.intellectualheaven.com/default.asp?BH=StraceNT

There are others, and there are others for *nix et al. as well-- I meant
to toss the class of tool out there, more than anything.

In general, you can do all the same stuff with every OS, but the toolset
will vary, and ease of use will vary (as will opinions on ease of use--
to each their own, etc. =]). If you've never used tools like these,
the curve may be a bit steep (or not) but the knowledge can be
invaluable. ('Tis worth noting that "value" isn't universal. =])

[Bruce Kirkpatrick]

I've also seen 5,000 requests per second to Railo Application.cfc file on my quad core core i7, but I was using ajp or http proxy as well and still got that performance.  There is definitely something interfering with normal operation if you have a second delay when using ajp.  There should be no significant delay for the proxy.  I've had it working fast on apache on linux and windows  and nginx on linux.

You might want to follow a guide of manually installing each piece in windows or linux.

I don't use the proxypassmatch, mod_cfml or mod_jk integration methods.  I used the mod_ajp_proxy with mod_rewrite when I used apache.  like RewriteRule ^/(.+\.cf[cm])(/.*)?$ ajp://127.0.0.1:8009/$1$2 [L,P]  (comment out everything else so this is the only proxy method) It was the most flexible one.   I don't know if there is a performance difference between them.  I would try setting it to 127.0.0.1, and changing localhost to 127.0.0.1 in railo/tomcat/conf/server.xml file as well.  That would eliminate the dns as an issue.

I force development dns either with the system hosts file or with a local dns server such as isc bind on windows or dnsmasq on linux so that it doesn't go out to the internet to find domain names that are hosted locally.   If you are just using localhost, maybe you have IIS and other stuff running on the same ports.  Ensure you have shut down all your other services that could be using port 80.   You could also check the tomcat/logs for any errors.   It is also possible to compile the APR native library for tomcat to get a performance boost for certain things.  this might be easier on linux then windows though.

[Paul Klinkenberg]

Though it's a full year later now, I happened to have the exact same problem on a new server setup.
As Bruce suggested, changing localhost to 127.0.0.1 completely fixed it. Awesome!
Old'n-bad: ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ ajp://localhost:8001/$1$2
New-'n-good: ProxyPassMatch ^/(.+\.cf[cm])(/.*)?$ ajp://127.0.0.1:8001/$1$2

Kind regards, Paul Klinkenberg

Op woensdag 1 mei 2013 22:17:23 UTC+2 schreef Brad Wood:

[Brad Wood]

Wow, you're right!  Somehow I had missed that last suggestion from Bruce on this thread.  I eventually stopped using Apache alltogether on my work machine because of this, but I just replaced "localhost" with "127.0.0.1" in my ProxyPassMatch line in the Apache config and new Apache requests are every bit as fast as hitting Tomcat directly.

Thanks!

~Brad

ColdBox Platform Evangelist

Ortus Solutions, Corp 

E-mail: br...@coldbox.org

ColdBox Platform: http://www.coldbox.org 

Blog: http://www.codersrevolution.com

--

Did you find this reply useful? Help the Railo community and add it to the Railo Server wiki at https://github.com/getrailo/railo/wiki

---
You received this message because you are subscribed to a topic in the Google Groups "Railo" group.

To view this discussion on the web visit https://groups.google.com/d/msgid/railo/941d2e56-68ba-4b5f-b980-b84597c67743%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Klinkenberg]

ROFL! I never thought my reply would be of help to the original op, but I'm glad to see it did :-D

Kind regards,

Paul Klinkenberg

You received this message because you are subscribed to the Google Groups "Railo" group.
To view this discussion on the web visit https://groups.google.com/d/msgid/railo/CALbQ1ok4whUqogH%2B52KUJER579Cmk93dAvugZ%2BdS44Fvx%2BQKnw%40mail.gmail.com.