Low request time

[Martin Schaible]

Hello

My question is not really directly connected with Lucee, but hopefully somebody has helpful ideas ;-)

We live in windows 2008R2 -> IIS7.5 --> Boncode --> Tomcat 8 --> Java 8 environment.

I always wondered, why i lose some speed somewhere compared to non-windows-Systems i know.
I'm sure, it's a matter of configuration and tuning.

I have the site http://www.hadiamedical.ch, which i can use to measure the loading time with Firebug. The site is quite small, i use RamCache for everything, The cache time is set to one day.

If i call the site, it needs a moment until something will be rendered: 1.2 sec. A reload delivers the page in under 200ms. I can click on every other main navigation item, every page will be served in under 200ms. This is okay, i think.

Now, i leave the site alone for one minute, afterwards i click the same page again and i have to wait again for 1.2 sec. After that, everything runs smoothly again. The server is bored to death anyway: No real CPU-load (8 Cores on 3GHz, 32GB RAM) . I measured the memory configuration with VisualVM, looks good to me.

I have this behaviour on every CFML-Server, independly what kind of CMS or other web apps the web sites are using.

The application pool (per site) of the websites are configured, that they never recycle.

My question is: Who is taking a nap?

Any ideas are very welcome.

Thanks a lot

Martin

[Igal @ Lucee.org]

@Martin,

what happens if you hit Tomcat directly, i.e. on port 8080 (or whatever port it's listening)...  do you get the same issue?

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/2a546c21-fe1e-40ac-8712-82a4f436fb8f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Schaible]

what happens if you hit Tomcat directly, i.e. on port 8080 (or whatever port it's listening)...  do you get the same issue?

It behaves like it should, the first GET, which is the important one, is on every site around 200 ms.

Now we could say, that IIS is the troublemaker, right? Not really, IIS is configured identically on all servers. All PHP and .NET sites respond very fast.

Do i have a communication problem between Tomcat and IIS? BonCode?

[Igal @ Lucee.org]

that's what I thought.

my bet is on BonCode.  Bilal did a great job there, but I think that there are some issues involved, and if I have to put my money on whether there's an issue with IIS or with BonCode then I go with BonCode since there is a much smaller user base with it.

on a somewhat personal note, I switched my servers from IIS to nginx a couple of years ago, and never looked back.  they've been running much more smoothly and efficiently.

Igal Sapir
Lucee Core Developer
Lucee.org

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/9598e7db-d835-4e3c-a367-8f6728152eee%40googlegroups.com.

[Martin Schaible]

my bet is on BonCode.

Unfortunately i think the same. In terms of IIS i applied all well known tuning and performance tweaks. Our Webmail and ticketing system are .NET Apps, the loading time is around 90ms. I randomly checked some PHP-sites using shops and CMS: Every site responds under 400 ms, average around 250 ms.

I really do not understand is the up and down of the response time. I'm measuring my actual project. The site responds in 1,2 secs. goes down to 300 ms, again up to 1 sec. and so on.
 

I hope that Bilal has someting to say or has even some hints how to go on with this problem. I wouldn't say anything, if we counting milliseconds, but we lose something more than 500 ms somewhere.

Are any other people here using a windows environment?

[Alex Skinner]

Hi,

We're using Windows and what you're experiencing I wouldn't say is "normal"

Here would be my approach

1. Create a new blank site with just an Application.cfc and index.cfm and Write out hello World

2. Hit the site with tomcat directly, measure the time in something like firefox or chrome dev tools on the network tab time to first byte

3. Turn on Debugging look at the time for the execution of the page should be near 0 ms compare the two,

4. Hit via iis and boncode, rinse and repeat compare the two.

There are two many factors to just point at Boncode including concurrency and what else is going on on the server.

Have you tweaked the Railo/Lucee settings since installing is it using the default JVM memory settings because if the system is running low on memory and doing Garbage collection then that could manifest itself as random pauses. But there are two many factors to guess so its really a case of systematically working your way front to back.

If you want some help commercially, Rasia are well geared up to point you in the right direction ?

Cheers

Alex

--

You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e34b54d6-bb62-4397-b14a-ca4a8c4dbfdb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Alex Skinner
Managing Director

Pixl8 Interactive, 3 Tun Yard, Peardon Street, London SW8 3HT, United Kingdom T: +44 [0] 845 260 0726• W: www.pixl8.co.uk• E: in...@pixl8.co.uk

Follow us on: Facebook Twitter LinkedIn

CONFIDENTIAL AND PRIVILEGED - This e-mail and any attachment is intended solely for the addressee, is strictly confidential and may also be subject to legal, professional or other privilege or may be protected by work product immunity or other legal rules. If you are not the addressee please do not read, print, re-transmit, store or act in reliance on it or any attachments. Instead, please email it back to the sender and then immediately permanently delete it. Pixl8 Interactive Ltd Registered in England. Registered number: 04336501. Registered office: 8 Spur Road, Cosham, Portsmouth, Hampshire, PO6 3EB

[Gert Franz]

Martin,

 

If you turn on debugging, what does it return? Always short times?

What happens with static pages?

If you have a simple index.cfm which is empty, what does it return?

I have a few sites running with Tomcat and Boncode and it normally is not an issue. It looks like it is a “keep connection alive for n seconds” kind of thing.

Perhaps Boncode is creating a new connection every time you leave the server alone for 1 minute…

 

Sincerely
Gert Franz

 

RASIA GmbH

Spittelgasse 7

5103 Moeriken-Wildegg

Email: ge...@rasia.ch
Skype: gert.franz

Phone Switzerland: +41 76 5680 231

--

[Martin Schaible]

Hello

Thanks for tips. i did already test with some really simple htm, jsp and cfm pages. I made the experience, that if call the page over the admin port, that the time is 1-3ms (Repeated 50 times by hitting refresh). Doing the same over IIS the time jumps around from 6 to 25 ms (Repeated 50 times by hitting refresh). Mostly the time is in between 6 to 8 ms.

I'm watching the memory behaviour of java constantly with VisualVM. From my point of view, the whole server is nearly in the state of getting bored.
Unfortunately VisualVM doesn't offer to get more infos about GC and the plugin GCViewer will work.
.
I also had the idea, that somebody drops connections. IIS has a keep alive function, which is active. I need to investigate, how Boncode deals with this.

[Igal @ Lucee.org]

can you try to connect IIS to Tomcat via the http APR connector (as opposed to AJP)?  you can do it with IIS' ARR.  see http://blogs.iis.net/eokim/archive/2010/03/09/arr-with-tomcat.aspx

if you can test IIS with Tomcat without BonCode then you'd be able to tell immediately whether BonCode plays a role in this issue or not.

I sincerely hope that I'm wrong about it, so please let us know your findings.

p.s. if you have the BonCode logging enabled -- turn it off and check if it still happens.

Igal Sapir
Lucee Core Developer
Lucee.org

[Martin Schaible]

@Igal:

I had also the idea to run tests with an alternative connector, which i will do.
Logging is switched off in production, on in developement, makes no difference.

[Bilal]

Martin,

what you are describing is most likely a timeout setting for the AJP connection that you have somewhere in the system.

There is overhead associated with establishing AJP connections to tomcat. This is true for all connection technologies that use AJP. 

It is in the neighborhood of 300-500ms per connection. The good news is that we use a connection pool and reuse the connections so that after the connections are established, the reuse overhead is minimal.

There are two things to further look at to see whether this is the issue:

On the Tomcat side please check your definition of the AJP channel in the server.xml file

Find a line like this:

<Connector port="8009" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout="60000" />

If there there is a connectionTimeout parameter, change it to something reasonable or remove it (blank will persist forever). Fair warning, some connection go stale and the timeout allows the recyling of those.

On the IIS side check the "Idle Timeout (minutes)" of your application pool. You can find it under the advanced settings for the application pool in the Process Model section. The default should be 20 minutes but see whether it got modified.

 

Hope this helps,

Bilal

[Martin Schaible]

Hi Bilal

Thanks for answering. Actually i have connectionTimeout="20000", which i will change afterward to the new value.

The idle timout of the application pools is set to zero. My application pools never sleep or getting recycled.

I added one site to the monitoring: The upper chart shows the site using iis, the lower chart over the admin port, eg.  www.mysite.ch:/<port>/news. The site will be measured every 5 minutes.

[Martin Schaible]

Changing the value to 60000 didn't help. Since i changed this value, i got twice this error:

Generic Connector Communication Error:

---

Please check and adjust your setup:
Ensure that Tomcat is running on given host and port.
If this is a timeout error consider adjusting IIS timeout by changing executionTimeout attribute in web.config (see manual).

Never ever had this before. Maybe a sign in which direction the problem points?

[Martin Schaible]

I'm sorry, i did read this wrong:

If there there is a connectionTimeout parameter, change it to something reasonable or remove it (blank will persist forever). Fair warning, some connection go stale and the timeout allows the recyling of those.

I removed the parameter now

[Jonathan Brookins]

Igal, the nginx sounds interesting.  Would you think it would be worth looking into replacing IIS with nginx on the Win 2003 machine that will eventually be the Lucee server?  Any caveats I should know before attempting it?

[Mark Drew]

What do you use IIS for apart from serving static content and passing requests off to Lucee?

Mark Drew


develop • deploy • deliver
http://charliemikedelta.com

[Igal @ Lucee.org]

from my personal experience -- running several e-commerce sites in production on Win 2008R2 -- nginx is superb!  my websites perform much better than with IIS, and I've measured everything when it comes to performance, both on the server and on the client sides.

once you understand the config file it's extremely simple to use as well, and it's easy to copy & paste config snippets that others share with you (as opposed to be given a list of 10 steps that you much follow to the letter in the GUI).

there's lots of documentation about it, and the nginx mailing list is very responsive.

an extra benefit is that it's cross platform, so if at some point you decide to move from Windows to *nix systems, there's one less thing to worry about.

Igal Sapir
Lucee Core Developer
Lucee.org

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/0A76A98E-2F63-4447-BF7A-7FC2577BD5AC%40gmail.com.

[Michael Hnat]

You're right on the one hand. Nginx is sure the best Webserver actually. But it also have some missing features like no .htaccess support, etc.

 

If you're running a server with several e-commerce sites where you are managing everthing this is fine. If you're offering a hosting service where customers are able to do things on their own (it's not only the webserver, it is also the managing of FTP, Mail and DNS Settings, etc.) with a control panel, then nginx is not the best solution for that.

 

And exactly this is the point where we don't want and can't move to nginx or linux. But the performance problem is still here. And it's just fascinating, because we can't see why only the .cfm stuff is slow. static files, asp and php - no problem.

 

Michi

 

---

Von: "Igal @ Lucee.org" <ig...@lucee.org>
Gesendet: Donnerstag, 12. Februar 2015 17:09
An: lu...@googlegroups.com
Betreff: Re: [Lucee] Low request time

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/54DCCFFF.8020601%40lucee.org.

[Igal @ Lucee.org]

but IIS doesn't support .htaccess either, does it?

and you're right, I do not offer hosting services, but nginx is a generic reverse proxy server that can also front mail servers, ftp servers, etc.  I've actually heard of many users who front their apache httpd with nginx, and claim that it improves performance...  go figure. 

Igal Sapir
Lucee Core Developer
Lucee.org

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/39cceea7630841fbb080e5bf1c535eb2%40bluegras.de.

[Martin Schaible]

but IIS doesn't support .htaccess either, does it?

With Helicon Ape 100% support of .htaccess. Loving it...

[Bilal]

That sounds like an interesting scenario indeed.

Normally the IIS overhead should be minimal after warm-up, really not noticeable at all even when compared to tomcat direct access.

Is there a way for you to run the connection monitoring on a specific empty .cfm page on both ports.

e.g.:

www.mysite.com/empty.cfm

I would also suggest you experiment with two different values for your polling.

a) inside the timeout, e.g. every 15 seconds

b) outside the timeout, e.g. every 5 minutes.

I would like to exclude anything that would prevent the channel from closing/hanging. 

When you move the timeout to 60 seconds and get the generic error it could be that you have used all connections inadvertently.

I would also check whether all available AJP connections have been used on the tomcat side (the host manager on tomcat can show you the status). This would also be a good indicator that something is preventing the channel from closing (that could be the reason you have such a low connection timeout and need to clear the channel), which in-turn causes frequent new tcp-ip connections to be formed etc. (I see wheels within wheels ;o) 

If you contact me via the project contact list we can see whether I can replicate your setup.

Best,

Bilal

[Martin Schaible]

Normally the IIS overhead should be minimal after warm-up, really not noticeable at all even when compared to tomcat direct access.

Yes, i like that very much ;-)

Is there a way for you to run the connection monitoring on a specific empty .cfm page on both ports.

e.g.:

www.mysite.com/empty.cfm

I would also suggest you experiment with two different values for your polling

i did that a minute ago. a new site without anything else, except a speed.cfm which spits out some text only. Unfortunately the minimum poll time is 30 seconds given by the PRTG monitoring software. I check if i can overrule somewhere.

 

I would also check whether all available AJP connections have been used on the tomcat side (the host manager on tomcat can show you the status)

Max threads: 1000 Current thread count: 10 
Actual situation:

| If you contact me via the project contact list we can see whether I can replicate your setup.

I started to document "my way" to setup everything @ Lucee Wiki. In terms of IIS setup and tuning everything is in "packages", mostly scripts, made for our Software Distribution Software. I can extract the appcmd-statements, which needs some time.

Thanks!

[Bilal]

Is there a way for you to run the connection monitoring on a specific empty .cfm page on both ports.

e.g.:

www.mysite.com/empty.cfm

I would also suggest you experiment with two different values for your polling

i did that a minute ago. a new site without anything else, except a speed.cfm which spits out some text only. Unfortunately the minimum poll time is 30 seconds given by the PRTG monitoring software. I check if i can overrule somewhere.
 

==> You can set the connectionTimeout to a higher value, e.g. 40000

 

Also I can create an EC2 instance on Amazon for us to play with if that is easier to recreate your server?

[Martin Schaible]

One situation is really strange: I can reload a page over and over, maybe 50 times, the measured time is constantly around a second. From one refresh to the other, the time drops down to 300 ms

The small speed.cfm behaves now very well. The times IIS versus Tomcat directly are equal at 1-2 ms. On the other hand, our very simple intranet page jumps up and down from 20 ms to 500 ms.

I sounds to me like a kind of a hickup. Timeouts somewhere and if yes, how to measure?

[Martin Schaible]

Also I can create an EC2 instance on Amazon for us to play with if that is easier to recreate your server?

Good idea, but i'm really a virgin in this, never used

What do you think about a remote session, that you can have a close view to my box?

[Bilal]

I will have get access to ports and deploy debug builds to analyse. That is normally easier on Amazon.

If you contact me via email I can  give you access information. 

You only need RDP client and internet access.

Bilal 

[Martin Schaible]

Hello

New day, new game :-)

I have build a clone of my development server with some difference:
- A copy of the website, using Mura with some pages is installed, which is the object to measure
- Tomcat acts as the HTTP-Server. I created a virtual host for the website
- Tomcat serves also IIS via a reverse proxy

In term ofs the first GET, which is the CFML-Template, it makes no real difference.
Both variants are delivering a nice response time: About 300 ms +-10%
I can hit the F5 for 100 times and more, the site behaves all the time very nice and no hickups at all.

Also the "white page" disappeared, which is typically if the website will be called for the first time.
Or in other words: I can enter the url, hit enter and the site is immediately loaded.

The same site on my regular development server runs like described in the previous posts.
If i call the site via Lucee's admin port, the response times are not different like times measured on the new testing server.

Now i have the proof, that only the site which is connected via Boncode to Tomcat suffers from high response time and non constant response times.

Additionally all four sites are monitored with PRTG. Each site will be polled every 30 seconds.
With this short poll rate, nothing can ever fall asleep. Except maybe the guy in front of the screen. 
The measurements do confirm, what i experienced by refreshing the pages manually using Firebug for measurements.
PRTG does measure a bit different. The charts are showing the complete load times, not only the first GET, which i watched.

Notes:
- Both servers are having the same virtual hardware specs, they are running on a esxi-server.
- All relevant software (OS, IIS) are equally configured.
 

Tomcat as HTTP-Server

IIS as Proxy for Tomcat

Original: Tomcat with Boncode and IIS

Thanks!

[Igal @ Lucee.org]

very nice Martin!  thank you for sharing this with us.

what version of Tomcat is that?

does this include only dynamic (.cfm) requests or also static resources like .js, .css, and images?  Tomcat by itself seems much faster than IIS, I wonder if that would still be true when static resources are fetched.  can you test that and let us know?

if that's the case, then fronting Tomcat with a web server is no longer needed for the sake of performance (older versions of Tomcat definitely required it).  though there are still other valid reasons to front it with a web server.

Igal Sapir
Lucee Core Developer
Lucee.org

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/e94e9475-daa2-4c70-b790-de0ce958e04c%40googlegroups.com.

[Martin Schaible]

what version of Tomcat is that?

Version 8.0.18

does this include only dynamic (.cfm) requests or also static resources like .js, .css, and images?

The times measured contains loading the complete page. Measuring with Firebug i only watched the first GET, which is the CFML-template

Tomcat by itself seems much faster than IIS

I don't think so.

 I wonder if that would still be true when static resources are fetched.  can you test that and let us know?

I'm not sure, what this has to do with the main issue. I prefer to concentrate the resources i have for the main issue, if you don't mind.

 though there are still other valid reasons to front it with a web server.

Indeed. A hosting environment on the windows platform world relies very often that an IIS is around :-) Simple example: The control panel allows to manage domains and websites based on IIS. Therefore cfml needs to passed through IIS to Tomcat . 

[Igal @ Lucee.org]

Tomcat by itself seems much faster than IIS

I don't think so.

then I guess misread the charts.

 I wonder if that would still be true when static resources are fetched.  can you test that and let us know?

I'm not sure, what this has to do with the main issue.

it doesn't have anything to do with your current issue.  it's something that was very true in the past, and may or may not be true anymore.  I thought that since you have the whole setup ready it might be easy to test.

--
You received this message because you are subscribed to the Google Groups "Lucee" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lucee+un...@googlegroups.com.
To post to this group, send email to lu...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lucee/548b43b8-30f6-4823-9021-93345d683dd8%40googlegroups.com.

[Martin Schaible]

I think, we found something:

Bilal ask me to switch off mod_cfml testwise.

- I added my test website as a virtual host to Tomcat.
- I removed mod_cfml from server.xml and finally told Boncode to not use mod_cfml.

After the needed restart of Tomcat,  my website is one the same speed as i documented last week, if i would use Tomcat as the HTTP server. No delay between Tomcat <-> BonCode <-> IIS.

Great news so far.

But what is wrong with mod_cfml? As we know: mod_cfml ist involved one time only per site at the time the website is warming up. It looks like, that mod_cfml doesn't like my envrionment ;-) Is mod_cfml active at every request? If yes, why only in my environment?

Any idea, how to continue to analyze this thing are welcome :-)

Cheers

Martin