Percentile Performance Degradation
55 views
Skip to first unread message
Jonathan Klein
Aug 3, 2010, 9:39:17 AM8/3/10
to Make the Web Faster
Hello,
I'd like to get a feel for an acceptable level of performance
degradation as you progress from your average to the 80th percentile
to the 95th percentile. We use Coradiant, and for a specific page
that we monitor these are the host time numbers (time it takes the
server to crunch the page):
Average: 1.9 seconds
80th Percentile: 2.6 seconds
95th Percentile: 3.9 seconds
I'm curious if anyone else has similar numbers, and if a 2x slowdown
between the average and the 95th percentile is normal or if that
indicates a scalability problem. Thanks in advance for your input.
-Jonathan
I'd like to get a feel for an acceptable level of performance
degradation as you progress from your average to the 80th percentile
to the 95th percentile. We use Coradiant, and for a specific page
that we monitor these are the host time numbers (time it takes the
server to crunch the page):
Average: 1.9 seconds
80th Percentile: 2.6 seconds
95th Percentile: 3.9 seconds
I'm curious if anyone else has similar numbers, and if a 2x slowdown
between the average and the 95th percentile is normal or if that
indicates a scalability problem. Thanks in advance for your input.
-Jonathan
Patrick Meenan
Aug 3, 2010, 1:22:16 PM8/3/10
to make-the-...@googlegroups.com
I poked around and looked at some highly scaled sites that I do have access
to the metrics for and see a similar 2x+ fall-off going from avg to 95th
percentile but the 75th percentiles are riught on top of the averages (and I
should mention that 2 seconds is exceptionally long for a server response in
most cases - I'm used to seeing numbers < 100ms). A lot of it is going to
depend on the app and hosting platform though.
to the metrics for and see a similar 2x+ fall-off going from avg to 95th
percentile but the 75th percentiles are riught on top of the averages (and I
should mention that 2 seconds is exceptionally long for a server response in
most cases - I'm used to seeing numbers < 100ms). A lot of it is going to
depend on the app and hosting platform though.
-Pat
Hello,
-Jonathan
--
You received this message because you are subscribed to the Google Groups
"Make the Web Faster" group.
To post to this group, send email to make-the-...@googlegroups.com.
To unsubscribe from this group, send email to
make-the-web-fa...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/make-the-web-faster?hl=en.
Alois Reitbauer
Aug 3, 2010, 1:36:50 PM8/3/10
to make-the-...@googlegroups.com
Jonathan,
Wait is this page doing. Normally response times for similar requests
should be rather constant. If you see a big discrepancy between
response times this looks like a data-driven problem or a possible
resource shortage on the server side. If the higher response times
can be limited to a specific timeframe it might also be reasons like
Garbage Collection or Overall System CPU usage. If you are using Java
or .NET you can also ask the Coradiant guys about App Visibility.
This can help to analyze this kind of backend issues
Alois
> --
> You received this message because you are subscribed to the Google
> Groups "Make the Web Faster" group.
> To post to this group, send email to make-the-web-
> fas...@googlegroups.com.
> To unsubscribe from this group, send email to make-the-web-faster
> +unsub...@googlegroups.com.
Jonathan Klein
Aug 3, 2010, 2:02:39 PM8/3/10
to make-the-...@googlegroups.com
Hey Pat,
Thanks for your response, I guess I should have just posted this on the webpagetest forums ;-). I actually realized after I sent the email that the numbers I gave were for E2E latency, and thus included network time as well. If I just look at host time the numbers look like this:
Average: 1.3 seconds
80th: 1.7 seconds
95th: 3 seconds
So they scale similarly, but our host times aren't quite as bad as they looked originally. I picked this page specifically because it is by far the slowest on our sites, and I agree that a 1.3 second host time is still pretty large. This is a complex page that has to interact heavily with a database, and as a result it takes quite a while for our servers to cook the page. We monitor the CPU/memory utilization on our servers very closely, and our DB slaves are consistently under 20% while the web servers hang out around 40%, so I don't think additional hardware would make much of a difference.
Alois - We are a SQL Server/IIS/Classic ASP shop, and while people are always quick to lambast Classic ASP we find that most of the load time is taking place on the DB side (I'm still pushing for a switch, but there is a fair amount of inertia). I believe this means that we can't use App Visibility.
A 30% increase in load time from the average to the 80th percentile sounds like it might be slightly concerning, I'll dig into this a bit and see what I come up with. If anyone else has numbers to share that would be great as well.
-Jonathan
Thanks for your response, I guess I should have just posted this on the webpagetest forums ;-). I actually realized after I sent the email that the numbers I gave were for E2E latency, and thus included network time as well. If I just look at host time the numbers look like this:
Average: 1.3 seconds
80th: 1.7 seconds
95th: 3 seconds
So they scale similarly, but our host times aren't quite as bad as they looked originally. I picked this page specifically because it is by far the slowest on our sites, and I agree that a 1.3 second host time is still pretty large. This is a complex page that has to interact heavily with a database, and as a result it takes quite a while for our servers to cook the page. We monitor the CPU/memory utilization on our servers very closely, and our DB slaves are consistently under 20% while the web servers hang out around 40%, so I don't think additional hardware would make much of a difference.
Alois - We are a SQL Server/IIS/Classic ASP shop, and while people are always quick to lambast Classic ASP we find that most of the load time is taking place on the DB side (I'm still pushing for a switch, but there is a fair amount of inertia). I believe this means that we can't use App Visibility.
A 30% increase in load time from the average to the 80th percentile sounds like it might be slightly concerning, I'll dig into this a bit and see what I come up with. If anyone else has numbers to share that would be great as well.
-Jonathan
Guy
Aug 3, 2010, 5:13:41 PM8/3/10
to make-the-...@googlegroups.com
my 2 cents.
The average metric is meaningless without standard deviation.
If your sandard deviation is high then your higher percentiles will be much higher than the average.
Look at the following table to see what I mean:
| 0 | 4 | |
| 1 | 5 | |
| 2 | 5 | |
| 3 | 5 | |
| 4 | 4 | |
| 5 | 5 | |
| 6 | 6 | |
| 7 | 5 | |
| 8 | 5 | |
| 9 | 5 | |
| 10 | 6 | |
| Average | 5 | 5 |
| 75th percentile | 7.75 | 5 |
| STD | 3.316625 | 0.632456 |
Jonathan Klein
Aug 5, 2010, 12:25:39 PM8/5/10
to make-the-...@googlegroups.com
Right I agree, but like you said there is a strong correlation between standard deviation and percentile distribution. So you can get the same information by looking at percentiles that you can by looking at standard deviation. I'm just curious about what an acceptable standard deviation is, and it looks like we are *kind of* in the acceptable range.
-Jonathan
-Jonathan
Drit
Aug 13, 2010, 5:41:41 PM8/13/10
to Make the Web Faster
Jonathan,
Standard deviation tells you the variation of data points from the
average. The higher it is, the more spread out the values are. The
lower it is, the more close to the average the values are. So what you
want is the lowest standard deviation possible, as that would mean you
stay true to the average and there is not high variability.
http://en.wikipedia.org/wiki/Standard_deviation
I looked at a couple of tests that we are running at Catchpoint and
here is some interesting stats from Amazon and Newegg, based on
testing from New York City (3 day time frame):
Amazon Homepage (Server Wait)
Average 310ms
95 Percentile 340ms
85 Percentile 327ms
Standard Deviation 18ms
Median 311ms
Geometrical Mean 309ms
Newegg Homepage (Server Wait)
Average 56ms
95 Percentile 107ms
85 Percentile 86ms
Standard Deviation 39ms
Median 77ms
Geometrical Mean 36ms
Amazon has a lower Standard Deviation, you can clearly see the
variation between Average, Median, 85%, and 95% is low; however, its
server perform worst than Newegg.
First, I would suggest you look at standard deviation, to see if you
have too much variance. Also you can look at 85th and 95th to see if
they are trending up, that would be a bad sign.
Secondly, make sure you compare apples to apples, and what I mean is
that you monitor the same page - or similar pages that have about the
same amount of processing/work load. A search results page, has
completely different performance from a product details page.
drit
http://twitter.com/dritans
On Aug 5, 12:25 pm, Jonathan Klein <jonathan.n.kl...@gmail.com> wrote:
> Right I agree, but like you said there is a strong correlation between
> standard deviation and percentile distribution. So you can get the same
> information by looking at percentiles that you can by looking at standard
> deviation. I'm just curious about what an acceptable standard deviation is,
> and it looks like we are *kind of* in the acceptable range.
>
> -Jonathan
>
>
>
> On Tue, Aug 3, 2010 at 5:13 PM, Guy <mug...@gmail.com> wrote:
> > my 2 cents.
>
> > The average metric is meaningless without standard deviation.
> > If your sandard deviation is high then your higher percentiles will be much
> > higher than the average.
>
> > Look at the following table to see what I mean:
> > 0 4 1 5 2 5 3 5 4 4 5 5 6 6 7 5 8 5 9 5 10 6 *Average* *5*
> > *5* *75th percentile* *7.75* *5* *STD* *3.316625* *0.632456*
> >>> make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsubsc...@googlegroups.com>
> >>> .
> >>> .
> >> .
> > .
>
> - Show quoted text -
Standard deviation tells you the variation of data points from the
average. The higher it is, the more spread out the values are. The
lower it is, the more close to the average the values are. So what you
want is the lowest standard deviation possible, as that would mean you
stay true to the average and there is not high variability.
http://en.wikipedia.org/wiki/Standard_deviation
I looked at a couple of tests that we are running at Catchpoint and
here is some interesting stats from Amazon and Newegg, based on
testing from New York City (3 day time frame):
Amazon Homepage (Server Wait)
Average 310ms
95 Percentile 340ms
85 Percentile 327ms
Standard Deviation 18ms
Median 311ms
Geometrical Mean 309ms
Newegg Homepage (Server Wait)
Average 56ms
95 Percentile 107ms
85 Percentile 86ms
Standard Deviation 39ms
Median 77ms
Geometrical Mean 36ms
Amazon has a lower Standard Deviation, you can clearly see the
variation between Average, Median, 85%, and 95% is low; however, its
server perform worst than Newegg.
First, I would suggest you look at standard deviation, to see if you
have too much variance. Also you can look at 85th and 95th to see if
they are trending up, that would be a bad sign.
Secondly, make sure you compare apples to apples, and what I mean is
that you monitor the same page - or similar pages that have about the
same amount of processing/work load. A search results page, has
completely different performance from a product details page.
drit
http://twitter.com/dritans
On Aug 5, 12:25 pm, Jonathan Klein <jonathan.n.kl...@gmail.com> wrote:
> Right I agree, but like you said there is a strong correlation between
> standard deviation and percentile distribution. So you can get the same
> information by looking at percentiles that you can by looking at standard
> deviation. I'm just curious about what an acceptable standard deviation is,
> and it looks like we are *kind of* in the acceptable range.
>
> -Jonathan
>
>
>
> On Tue, Aug 3, 2010 at 5:13 PM, Guy <mug...@gmail.com> wrote:
> > my 2 cents.
>
> > The average metric is meaningless without standard deviation.
> > If your sandard deviation is high then your higher percentiles will be much
> > higher than the average.
>
> > Look at the following table to see what I mean:
> > *5* *75th percentile* *7.75* *5* *STD* *3.316625* *0.632456*
>
> > On Tue, Aug 3, 2010 at 11:02 AM, Jonathan Klein <
> > jonathan.n.kl...@gmail.com> wrote:
>
> >> Hey Pat,
>
> >> Thanks for your response, I guess I should have just posted this on the
> >> webpagetest forums ;-). I actually realized after I sent the email that the
> >> numbers I gave were for E2E latency, and thus included network time as
> >> well. If I just look at host time the numbers look like this:
>
> >> Average: 1.3 seconds
> >> 80th: 1.7 seconds
> >> 95th: 3 seconds
>
> >> So they scale similarly, but our host times aren't *quite* as bad as they
> > On Tue, Aug 3, 2010 at 11:02 AM, Jonathan Klein <
> > jonathan.n.kl...@gmail.com> wrote:
>
> >> Hey Pat,
>
> >> Thanks for your response, I guess I should have just posted this on the
> >> webpagetest forums ;-). I actually realized after I sent the email that the
> >> numbers I gave were for E2E latency, and thus included network time as
> >> well. If I just look at host time the numbers look like this:
>
> >> Average: 1.3 seconds
> >> 80th: 1.7 seconds
> >> 95th: 3 seconds
>
> >> looked originally. I picked this page specifically because it is by far the
> >> slowest on our sites, and I agree that a 1.3 second host time is still
> >> pretty large. This is a complex page that has to interact heavily with a
> >> database, and as a result it takes quite a while for our servers to cook the
> >> page. We monitor the CPU/memory utilization on our servers very closely,
> >> and our DB slaves are consistently under 20% while the web servers hang out
> >> around 40%, so I don't think additional hardware would make much of a
> >> difference.
>
> >> Alois - We are a SQL Server/IIS/Classic ASP shop, and while people are
> >> always quick to lambast Classic ASP we find that most of the load time is
> >> taking place on the DB side (I'm still pushing for a switch, but there is a
> >> fair amount of inertia). I believe this means that we can't use App
> >> Visibility.
>
> >> A 30% increase in load time from the average to the 80th percentile sounds
> >> like it might be slightly concerning, I'll dig into this a bit and see what
> >> I come up with. If anyone else has numbers to share that would be great as
> >> well.
>
> >> -Jonathan
>
> >> slowest on our sites, and I agree that a 1.3 second host time is still
> >> pretty large. This is a complex page that has to interact heavily with a
> >> database, and as a result it takes quite a while for our servers to cook the
> >> page. We monitor the CPU/memory utilization on our servers very closely,
> >> and our DB slaves are consistently under 20% while the web servers hang out
> >> around 40%, so I don't think additional hardware would make much of a
> >> difference.
>
> >> Alois - We are a SQL Server/IIS/Classic ASP shop, and while people are
> >> always quick to lambast Classic ASP we find that most of the load time is
> >> taking place on the DB side (I'm still pushing for a switch, but there is a
> >> fair amount of inertia). I believe this means that we can't use App
> >> Visibility.
>
> >> A 30% increase in load time from the average to the 80th percentile sounds
> >> like it might be slightly concerning, I'll dig into this a bit and see what
> >> I come up with. If anyone else has numbers to share that would be great as
> >> well.
>
> >> -Jonathan
>
> >>> .
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Make the Web Faster" group.
> >>> To post to this group, send email to
> >>> make-the-...@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsubsc...@googlegroups.com>
> >>>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups
> >>> "Make the Web Faster" group.
> >>> To post to this group, send email to
> >>> make-the-...@googlegroups.com.
> >>> To unsubscribe from this group, send email to
> >>> .
> >>> For more options, visit this group at
> >>>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Make the Web Faster" group.
> >> To post to this group, send email to make-the-...@googlegroups.com
> >> .
> >>>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "Make the Web Faster" group.
> >> To post to this group, send email to make-the-...@googlegroups.com
> >> .
> >> To unsubscribe from this group, send email to
> >> make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsubsc...@googlegroups.com>
> >> .
> >> For more options, visit this group at
> >>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Make the Web Faster" group.
> > To post to this group, send email to make-the-...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsubsc...@googlegroups.com>
> >>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Make the Web Faster" group.
> > To post to this group, send email to make-the-...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/make-the-web-faster?hl=en.- Hide quoted text -
>
> - Show quoted text -
Jonathan Klein
Aug 17, 2010, 10:28:41 AM8/17/10
to make-the-...@googlegroups.com
Thanks, I am familiar with standard deviation. I am looking specifically at a product browsing page - a page that returns 24 products based on filters you select (e.g. http://www.newegg.com/Store/SubCategory.aspx?SubCategory=147&name=Desktop-Memory if we are going with the Newegg example). Here are the host time numbers for the last three hours:
Average: 1.1 secs
95 Percentile: 2.5 secs
85 Percentile: 1.8 secs
Std Dev: 741 ms
Median: Not given in our reports
Geometric Mean: Not given in our reports
This standard deviation seems quite high - which is why I brought up this point in the first place. Especially when you compare the numbers to Amazon and Newegg it seems clear that our variance is higher than desirable. However I believe that Newegg is an Akamai customer, and I think Amazon uses it's own CDN? That could certainly help to explain why their numbers are so good, especially for the homepages which are basically guaranteed to be cached at an edge server near you (or near the Catchpoint testing location). I would be interested to see the numbers for a highly filtered product browsing page where the server has to do a lot more work and the page has a smaller chance of being cached.
-Jonathan
Average: 1.1 secs
95 Percentile: 2.5 secs
85 Percentile: 1.8 secs
Std Dev: 741 ms
Median: Not given in our reports
Geometric Mean: Not given in our reports
This standard deviation seems quite high - which is why I brought up this point in the first place. Especially when you compare the numbers to Amazon and Newegg it seems clear that our variance is higher than desirable. However I believe that Newegg is an Akamai customer, and I think Amazon uses it's own CDN? That could certainly help to explain why their numbers are so good, especially for the homepages which are basically guaranteed to be cached at an edge server near you (or near the Catchpoint testing location). I would be interested to see the numbers for a highly filtered product browsing page where the server has to do a lot more work and the page has a smaller chance of being cached.
-Jonathan
To unsubscribe from this group, send email to make-the-web-fa...@googlegroups.com.
Drit
Aug 17, 2010, 11:30:53 AM8/17/10
to Make the Web Faster
Jonathan,
The numbers I provided above are based on the server wait time of the
primary URL. This number does not include DNS Reolution, Connection to
IP, Sending the data, receiving the entire HTML content, or the
loading of the webpage. Hence the small standard deviation compared to
your tests.
Server Wait time is one of the best metrics to use to understand how
fast the processing is on the server. This number would not be
impacted by use of CDNs, adserver, JavaScript etc.
I can definetly provide you with more detailed numbers. I just want to
make sure we are clear on the numbers. Email me at
drit__AT__catchpoint__DOTCOM ignore the underscores.
thanks
drit
On Aug 17, 10:28 am, Jonathan Klein <jonathan.n.kl...@gmail.com>
wrote:
> > > >http://groups.google.com/group/make-the-web-faster?hl=en.-Hide quoted
>
> read more »- Hide quoted text -
The numbers I provided above are based on the server wait time of the
primary URL. This number does not include DNS Reolution, Connection to
IP, Sending the data, receiving the entire HTML content, or the
loading of the webpage. Hence the small standard deviation compared to
your tests.
Server Wait time is one of the best metrics to use to understand how
fast the processing is on the server. This number would not be
impacted by use of CDNs, adserver, JavaScript etc.
I can definetly provide you with more detailed numbers. I just want to
make sure we are clear on the numbers. Email me at
drit__AT__catchpoint__DOTCOM ignore the underscores.
thanks
drit
On Aug 17, 10:28 am, Jonathan Klein <jonathan.n.kl...@gmail.com>
wrote:
> Thanks, I am familiar with standard deviation. I am looking specifically at
> a product browsing page - a page that returns 24 products based on filters
> you select (e.g.http://www.newegg.com/Store/SubCategory.aspx?SubCategory=147&name=Des...
> a product browsing page - a page that returns 24 products based on filters
> > text -
>
> > > - Show quoted text -
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Make the Web Faster" group.
> > To post to this group, send email to make-the-...@googlegroups.com.
> > To unsubscribe from this
>
> ...
>
> > > - Show quoted text -
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Make the Web Faster" group.
> > To post to this group, send email to make-the-...@googlegroups.com.
> > To unsubscribe from this
>
>
> read more »- Hide quoted text -
Patrick Meenan
Aug 17, 2010, 10:50:30 AM8/17/10
to make-the-...@googlegroups.com
For server response time the edge caches won't help as the HTML itself isn't usually cached, just the static assets. For a product search page it is going to come down to how you have the products stored and how quickly your back-end can filter the results (most will augment a well-indexed RDBMS with a free-text index like SOLR).
-Pat
Jonathan Klein
Aug 23, 2010, 10:37:06 AM8/23/10
to make-the-...@googlegroups.com
I am also looking at server wait time. When I say "host time" I am using Coradiant's term for server processing time. They break a page up into host time, network time, and page render time. More explicitly, host time is defined as the time between the last packet of the request and the first packet of the response as seen by Coradiant, which is effectively a network tap sitting in our datacenter.
Pat - there are plenty of Akamai customers that are caching HTML/dynamic content (we used to be one of them :-)). I can't say for sure if Newegg is following that model, but even if they aren't they may be using some sort of dynamic content caching/varnish-like solution for their heavily trafficked pages. We do implement SOLR for our product keyword search, but this page is more of a browsing page where you filter through predefined product attributes. Even on our SOLR driven keyword search page we are seeing an average of 621.3ms with a standard deviation of 548ms - this again being server wait time.
-Jonathan
Pat - there are plenty of Akamai customers that are caching HTML/dynamic content (we used to be one of them :-)). I can't say for sure if Newegg is following that model, but even if they aren't they may be using some sort of dynamic content caching/varnish-like solution for their heavily trafficked pages. We do implement SOLR for our product keyword search, but this page is more of a browsing page where you filter through predefined product attributes. Even on our SOLR driven keyword search page we are seeing an average of 621.3ms with a standard deviation of 548ms - this again being server wait time.
-Jonathan
Patrick Meenan
Aug 23, 2010, 11:26:08 AM8/23/10
to make-the-...@googlegroups.com
Your results were for one of your worst pages, not the front-door, right? Even without something like Akamai caching the base page I'd expect the main landing page to largely be cached on the server side (memcache or otherwise). If the Amazon and Newegg results were for unauthenticated (i.e. not personalized versions of the page) then I'd honestly be surprised if they weren't cached.
That said, even without looking at the variance, the raw times look a lot higher than I would expect. Do you log the response time in you access logs? That's generally where I'm used to looking at it and generally anything over a couple hundred milliseconds for a high-profile page is going to be something to worry about.
I have really easy access to 95th percentile times from access logs (sites to remain anonymous but of very significant scale):
- Site #1 - Apache/Tomcat/Java : 200-400ms (known optimization work we have to do around back-end caching)
- Site #2 - Apache/PHP : 100-300ms
- Site #3 - Apache/PHP : 50-150ms
- Site #4 - Apache/Tomcat/Java : 50-80ms
All of these sites have fairly complex back-ends so it's not a matter of serving static files quickly.
Thanks,
-Pat
Jonathan Klein
Aug 23, 2010, 12:53:45 PM8/23/10
to make-the-...@googlegroups.com
Unfortunately these sorts of load times are common across many pages, we can't touch the load times you quote on any of our pages at the 95th percentile. Even looking at our static content which is literally coming off a SAN and being served as quickly as possible through a couple of BSD/Lighttpd boxes we see a 334ms response time at the 95th percentile (again, this is a host time measurement - time between the last packet of the request and the first packet of the response). Here are some numbers for the rest of our pages at the 95th percentile:
Homepages: 1738ms
Keyword: 1929ms
Product Detail Pages: 1344ms
Product Browsing Pages: 3206ms
These seemed abysmal to me at first, but I guess I've gotten used to seeing these kinds of server response times for our pages. We created a varnish-like internal caching solution to mitigate some of the problem here, but our cache hit can't get as high as we would like because we have so many unique URLs (running ~220 sites). We also do some webserver level caching with a locally installed COM object, but we are currently not using any sort of centralized cache layer between the DB servers and the webservers.
I'm extremely surprised that you are seeing numbers that good at the 95th percentile - you said that the sites have fairly complex back-ends, but there must be some pretty aggressive caching in place with a high hit rate to get numbers that low (or our stack really sucks).
-Jonathan
Homepages: 1738ms
Keyword: 1929ms
Product Detail Pages: 1344ms
Product Browsing Pages: 3206ms
These seemed abysmal to me at first, but I guess I've gotten used to seeing these kinds of server response times for our pages. We created a varnish-like internal caching solution to mitigate some of the problem here, but our cache hit can't get as high as we would like because we have so many unique URLs (running ~220 sites). We also do some webserver level caching with a locally installed COM object, but we are currently not using any sort of centralized cache layer between the DB servers and the webservers.
I'm extremely surprised that you are seeing numbers that good at the 95th percentile - you said that the sites have fairly complex back-ends, but there must be some pretty aggressive caching in place with a high hit rate to get numbers that low (or our stack really sucks).
-Jonathan
Patrick Meenan
Aug 23, 2010, 2:01:35 PM8/23/10
to make-the-...@googlegroups.com
Have you tried turning on the response times in your access logs and comparing those results to what you are seeing in the Coradiant data? Just on the off chance that something else is going on with the Coradiant results. Otherwise I'd be REALLY concerned about those times for your static objects off of SAN, particularly with Lighttpd which is worlds more consistent and scalable than Apache.
Jonathan Klein
Aug 24, 2010, 8:13:06 AM8/24/10
to make-the-...@googlegroups.com
Our Ops guys enabled response times in the access logs, but apparently Lighttpd logs times in integer values of seconds, which isn't too helpful (if anyone knows how to change this let me know). We took a TCP dump of the server traffic so I am going to analyze that today and try to get some raw numbers. I'll let you know what I find - thanks for the continued assistance with this.
-Jonathan
-Jonathan
Patrick Meenan
Aug 24, 2010, 6:29:28 PM8/24/10
to make-the-...@googlegroups.com
Well that's useless: http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModAccessLog - they don't support the ms directive (and if you're measuring things in seconds, what's the point?). I suppose switching to nginx is probably more than the ops team wants to consider (everything I've heard/read claims it works better than lighttpd - particularly with the pesky memory leak in lighttpd) and they log in ms :-) - http://wiki.nginx.org/NginxHttpLogModule
Jonathan Klein
Aug 25, 2010, 10:06:40 AM8/25/10
to make-the-...@googlegroups.com
My sentiments exactly. We actually just tested a dev box with Nginx and it appears to be noticeably faster. We ran some tests using WebPagetest and our internal Nginx box compares quite well with requests that are hitting Akamai edge servers while the Lighttpd box appears to be consistently slower. We are going to convert our production static content servers over to Nginx and see what happens to our reports (and the access logs).
Luckily for me the ops guys love to try new things and make stuff as fast as possible :-).
-Jonathan
Luckily for me the ops guys love to try new things and make stuff as fast as possible :-).
-Jonathan
Jonathan Klein
Aug 26, 2010, 11:49:34 AM8/26/10
to make-the-...@googlegroups.com
Ok we put an Nginx box in production and I have some data. Looking at a block of ~50K requests in the access log this is what I find:
Average Request time: 12ms
Standard Deviation: 350ms
That standard deviation seems unbelievably huge. Here are the slowest 10 requests in seconds:
Almost 44,000 of the requests are under 1ms, and the rest of them pull the average up quite a bit and create the huge standard deviation we are seeing. For a similar period Coradiant shows these numbers:
Average: 34.5ms
Normalized Average: 185ms
Standard Deviation: 159ms
I feel like a 12ms average is great (or even a 34ms average), but I'm not super psyched about the variability in the data. I'm not sure what's going on with those super slow requests - any ideas? It seems unlikely that it would be BSD/nginx, which points to the SAN as the possible bottleneck. Does anyone else have numbers for static content that they are willing to share? My attitude is basically that I think it should be possible to get a ~10ms average with a very low (~20ms) standard deviation. Is this possible?
-Jonathan
Average Request time: 12ms
Standard Deviation: 350ms
That standard deviation seems unbelievably huge. Here are the slowest 10 requests in seconds:
| 4.9572 |
| 4.9722 |
| 4.9992 |
| 5.6862 |
| 5.9552 |
| 7.3482 |
| 10.1172 |
| 10.17421 |
| 39.4552 |
| 62.6752 |
Almost 44,000 of the requests are under 1ms, and the rest of them pull the average up quite a bit and create the huge standard deviation we are seeing. For a similar period Coradiant shows these numbers:
Average: 34.5ms
Normalized Average: 185ms
Standard Deviation: 159ms
I feel like a 12ms average is great (or even a 34ms average), but I'm not super psyched about the variability in the data. I'm not sure what's going on with those super slow requests - any ideas? It seems unlikely that it would be BSD/nginx, which points to the SAN as the possible bottleneck. Does anyone else have numbers for static content that they are willing to share? My attitude is basically that I think it should be possible to get a ~10ms average with a very low (~20ms) standard deviation. Is this possible?
-Jonathan
Jonathan Klein
Aug 26, 2010, 11:49:02 AM8/26/10
to make-the-...@googlegroups.com
Ok we put an Nginx box in production and I have some data. Looking at a block of ~50K requests in the access log this is what I find:
Average Request time: 12ms
Standard Deviation: 350ms
That standard deviation seems unbelievably huge. Here are the slowest 10 requests in seconds:
Almost 44,000 of the requests are under 1ms, and the rest of them pull the average up quite a bit and create the huge standard deviation we are seeing. For a similar period Coradiant shows these numbers:
Average: 34.5ms
Normalized Average: 185ms
Standard Deviation: 159ms
I feel like a 12ms average is great (or even a 34ms average), but I'm not super psyched about the variability in the data. I'm not sure what's going on with those super slow requests - any ideas? It seems unlikely that it would be BSD/nginx, which points to the SAN as the possible bottleneck. Does anyone else have numbers for static content that they are willing to share? My attitude is basically that I think it should be possible to get a ~10ms average with a very low (~20ms) standard deviation. Is this possible?
-Jonathan
Average Request time: 12ms
Standard Deviation: 350ms
That standard deviation seems unbelievably huge. Here are the slowest 10 requests in seconds:
| 4.9572 |
| 4.9722 |
| 4.9992 |
| 5.6862 |
| 5.9552 |
| 7.3482 |
| 10.1172 |
| 10.17421 |
| 39.4552 |
| 62.6752 |
Almost 44,000 of the requests are under 1ms, and the rest of them pull the average up quite a bit and create the huge standard deviation we are seeing. For a similar period Coradiant shows these numbers:
Average: 34.5ms
Normalized Average: 185ms
Standard Deviation: 159ms
I feel like a 12ms average is great (or even a 34ms average), but I'm not super psyched about the variability in the data. I'm not sure what's going on with those super slow requests - any ideas? It seems unlikely that it would be BSD/nginx, which points to the SAN as the possible bottleneck. Does anyone else have numbers for static content that they are willing to share? My attitude is basically that I think it should be possible to get a ~10ms average with a very low (~20ms) standard deviation. Is this possible?
-Jonathan
Patrick Meenan
Aug 26, 2010, 1:53:43 PM8/26/10
to make-the-...@googlegroups.com
How big is your store of static content? Just wondering if you'd be better off distributing the files to the edge nodes and having nginx read from local disk (or SSD :-)) instead of having it all go over SAN. You have to worry about making sure you don't have stale content and that the nodes are all up to date but it eliminates a lot of moving parts from the actual file serving side of things.
Jonathan Klein
Aug 30, 2010, 9:13:42 AM8/30/10
to make-the-...@googlegroups.com
This particular server is dealing with many millions of files totaling around 500GB. I couldn't get the exact number of files, but it's on the order of 100 million. Local storage is not a route that the IT-Ops group wants to go because of the challenges involved with replicating the files. It's looking more and more like our issues are related to congestion on the SAN share, or perhaps on the network connection to the SAN. We actually do have a set of servers that are serving static content off a FusionIO, and those numbers look better (but still not amazing). Unfortunately it's not really apples to apples because those servers are doing dynamic content caching (think Varnish) and there is a fair amount of logic that goes into each request, it's not simply serving a file off disk. We are thinking of using those boxes to serve some images/css/js off another port using Nginx just to see what happens.
Drit
Sep 2, 2010, 7:45:28 PM9/2/10
to Make the Web Faster
The "Response Time" on Nginx Logs, is it "Last Byte of Request" to
"First Byte of Response" (like coradiant) or "Last Byte of Request" to
"Last Byte of Response"?
drit
On Aug 30, 9:13 am, Jonathan Klein <jonathan.n.kl...@gmail.com> wrote:
> This particular server is dealing with many millions of files totaling
> around 500GB. I couldn't get the exact number of files, but it's on the
> order of 100 million. Local storage is not a route that the IT-Ops group
> wants to go because of the challenges involved with replicating the files.
> It's looking more and more like our issues are related to congestion on the
> SAN share, or perhaps on the network connection to the SAN. We actually do
> have a set of servers that are serving static content off a FusionIO,
> <http://www.fusionio.com/>and those numbers look better (but still not
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Thursday, August 26, 2010 11:49 AM
>
> > *To:* make-the-...@googlegroups.com
> > *Subject:* Re: [make-the-web-faster:307] Re: Percentile Performance
> > On Tue, Aug 24, 2010 at 6:29 PM, Patrick Meenan <patmee...@gmail.com>
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Tuesday, August 24, 2010 8:13 AM
>
> > *To:* make-the-...@googlegroups.com
>
> > *Subject:* Re: [make-the-web-faster:303] Re: Percentile Performance
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Monday, August 23, 2010 12:54 PM
>
> > *To:* make-the-...@googlegroups.com
>
> > *Subject:* Re: [make-the-web-faster:301] Re: Percentile Performance
>
> > -Jonathan
>
> > On Mon, Aug 23, 2010 at 11:26 AM, Patrick Meenan <patmee...@gmail.com>
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Monday, August 23, 2010 10:37 AM
> > *To:* make-the-...@googlegroups.com
> > *Subject:* Re: [make-the-web-faster:299] Re: Percentile Performance
>
> read more »- Hide quoted text -
"First Byte of Response" (like coradiant) or "Last Byte of Request" to
"Last Byte of Response"?
drit
On Aug 30, 9:13 am, Jonathan Klein <jonathan.n.kl...@gmail.com> wrote:
> This particular server is dealing with many millions of files totaling
> around 500GB. I couldn't get the exact number of files, but it's on the
> order of 100 million. Local storage is not a route that the IT-Ops group
> wants to go because of the challenges involved with replicating the files.
> It's looking more and more like our issues are related to congestion on the
> SAN share, or perhaps on the network connection to the SAN. We actually do
> have a set of servers that are serving static content off a FusionIO,
> amazing). Unfortunately it's not really apples to apples because those
> servers are doing dynamic content caching (think Varnish) and there is a
> fair amount of logic that goes into each request, it's not simply serving a
> file off disk. We are thinking of using those boxes to serve some
> images/css/js off another port using Nginx just to see what happens.
>
>
>
> On Thu, Aug 26, 2010 at 1:53 PM, Patrick Meenan <patmee...@gmail.com> wrote:
> > How big is your store of static content? Just wondering if you'd be better
> > off distributing the files to the edge nodes and having nginx read from
> > local disk (or SSD :-)) instead of having it all go over SAN. You have to
> > worry about making sure you don't have stale content and that the nodes are
> > all up to date but it eliminates a lot of moving parts from the actual file
> > serving side of things.
>
> > -Pat
>
> > *From:* make-the-...@googlegroups.com [mailto:
> servers are doing dynamic content caching (think Varnish) and there is a
> fair amount of logic that goes into each request, it's not simply serving a
> file off disk. We are thinking of using those boxes to serve some
> images/css/js off another port using Nginx just to see what happens.
>
>
>
> On Thu, Aug 26, 2010 at 1:53 PM, Patrick Meenan <patmee...@gmail.com> wrote:
> > How big is your store of static content? Just wondering if you'd be better
> > off distributing the files to the edge nodes and having nginx read from
> > local disk (or SSD :-)) instead of having it all go over SAN. You have to
> > worry about making sure you don't have stale content and that the nodes are
> > all up to date but it eliminates a lot of moving parts from the actual file
> > serving side of things.
>
> > -Pat
>
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Thursday, August 26, 2010 11:49 AM
>
> > *To:* make-the-...@googlegroups.com
> > *Subject:* Re: [make-the-web-faster:307] Re: Percentile Performance
> > wrote:
>
> > Well that's useless:
> >http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModAccessLog-
> > they don't support the ms directive (and if you're measuring things in
> > seconds, what's the point?). I suppose switching to nginx is probably more
> > than the ops team wants to consider (everything I've heard/read claims it
> > works better than lighttpd - particularly with the pesky memory leak in
> > lighttpd) and they log in ms :-) -
> >http://wiki.nginx.org/NginxHttpLogModule
>
> > *From:* make-the-...@googlegroups.com [mailto:
>
> > Well that's useless:
> >http://redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModAccessLog-
> > they don't support the ms directive (and if you're measuring things in
> > seconds, what's the point?). I suppose switching to nginx is probably more
> > than the ops team wants to consider (everything I've heard/read claims it
> > works better than lighttpd - particularly with the pesky memory leak in
> > lighttpd) and they log in ms :-) -
> >http://wiki.nginx.org/NginxHttpLogModule
>
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Tuesday, August 24, 2010 8:13 AM
>
> > *To:* make-the-...@googlegroups.com
>
> > *Subject:* Re: [make-the-web-faster:303] Re: Percentile Performance
> > Degradation
>
> > Our Ops guys enabled response times in the access logs, but apparently
> > Lighttpd logs times in integer values of seconds, which isn't too helpful
> > (if anyone knows how to change this let me know). We took a TCP dump of the
> > server traffic so I am going to analyze that today and try to get some raw
> > numbers. I'll let you know what I find - thanks for the continued
> > assistance with this.
>
> > -Jonathan
>
> > On Mon, Aug 23, 2010 at 2:01 PM, Patrick Meenan <patmee...@gmail.com>
>
> > Our Ops guys enabled response times in the access logs, but apparently
> > Lighttpd logs times in integer values of seconds, which isn't too helpful
> > (if anyone knows how to change this let me know). We took a TCP dump of the
> > server traffic so I am going to analyze that today and try to get some raw
> > numbers. I'll let you know what I find - thanks for the continued
> > assistance with this.
>
> > -Jonathan
>
> > wrote:
>
> > Have you tried turning on the response times in your access logs and
> > comparing those results to what you are seeing in the Coradiant data? Just
> > on the off chance that something else is going on with the Coradiant
> > results. Otherwise I'd be REALLY concerned about those times for your
> > static objects off of SAN, particularly with Lighttpd which is worlds more
> > consistent and scalable than Apache.
>
> > -Pat
>
> > *From:* make-the-...@googlegroups.com [mailto:
>
> > Have you tried turning on the response times in your access logs and
> > comparing those results to what you are seeing in the Coradiant data? Just
> > on the off chance that something else is going on with the Coradiant
> > results. Otherwise I'd be REALLY concerned about those times for your
> > static objects off of SAN, particularly with Lighttpd which is worlds more
> > consistent and scalable than Apache.
>
> > -Pat
>
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Monday, August 23, 2010 12:54 PM
>
> > *To:* make-the-...@googlegroups.com
>
> > *Subject:* Re: [make-the-web-faster:301] Re: Percentile Performance
> > Degradation
>
> > Unfortunately these sorts of load times are common across many pages, we
> > can't touch the load times you quote on any of our pages at the 95th
> > percentile. Even looking at our static content which is literally coming
> > off a SAN and being served as quickly as possible through a couple of
> > BSD/Lighttpd boxes we see a 334ms response time at the 95th percentile
> > (again, this is a host time measurement - time between the last packet of
> > the request and the first packet of the response). Here are some numbers
> > for the rest of our pages at the 95th percentile:
>
> > Homepages: 1738ms
> > Keyword: 1929ms
> > Product Detail Pages: 1344ms
> > Product Browsing Pages: 3206ms
>
> > These seemed abysmal to me at first, but I guess I've gotten used to seeing
> > these kinds of server response times for our pages. We created a
> > varnish-like internal caching solution to mitigate some of the problem here,
> > but our cache hit can't get as high as we would like because we have so many
> > unique URLs (running ~220 sites). We also do some webserver level caching
> > with a locally installed COM object, but we are currently not using any sort
> > of centralized cache layer between the DB servers and the webservers.
>
> > I'm extremely surprised that you are seeing numbers that good at the 95th
> > percentile - you said that the sites have fairly complex back-ends, but
> > there must be some pretty aggressive caching in place with a high hit rate
> > to get numbers that low (or our stack *really* sucks).
>
> > Unfortunately these sorts of load times are common across many pages, we
> > can't touch the load times you quote on any of our pages at the 95th
> > percentile. Even looking at our static content which is literally coming
> > off a SAN and being served as quickly as possible through a couple of
> > BSD/Lighttpd boxes we see a 334ms response time at the 95th percentile
> > (again, this is a host time measurement - time between the last packet of
> > the request and the first packet of the response). Here are some numbers
> > for the rest of our pages at the 95th percentile:
>
> > Homepages: 1738ms
> > Keyword: 1929ms
> > Product Detail Pages: 1344ms
> > Product Browsing Pages: 3206ms
>
> > These seemed abysmal to me at first, but I guess I've gotten used to seeing
> > these kinds of server response times for our pages. We created a
> > varnish-like internal caching solution to mitigate some of the problem here,
> > but our cache hit can't get as high as we would like because we have so many
> > unique URLs (running ~220 sites). We also do some webserver level caching
> > with a locally installed COM object, but we are currently not using any sort
> > of centralized cache layer between the DB servers and the webservers.
>
> > I'm extremely surprised that you are seeing numbers that good at the 95th
> > percentile - you said that the sites have fairly complex back-ends, but
> > there must be some pretty aggressive caching in place with a high hit rate
>
> > -Jonathan
>
> > On Mon, Aug 23, 2010 at 11:26 AM, Patrick Meenan <patmee...@gmail.com>
> > wrote:
>
> > Your results were for one of your worst pages, not the front-door, right?
> > Even without something like Akamai caching the base page I'd expect the
> > main landing page to largely be cached on the server side (memcache or
> > otherwise). If the Amazon and Newegg results were for unauthenticated (i.e.
> > not personalized versions of the page) then I'd honestly be surprised if
> > they weren't cached.
>
> > That said, even without looking at the variance, the raw times look a lot
> > higher than I would expect. Do you log the response time in you access
> > logs? That's generally where I'm used to looking at it and generally
> > anything over a couple hundred milliseconds for a high-profile page is going
> > to be something to worry about.
>
> > I have really easy access to 95th percentile times from access logs (sites
> > to remain anonymous but of very significant scale):
>
> > - Site #1 - Apache/Tomcat/Java : 200-400ms (known optimization work we have
> > to do around back-end caching)
>
> > - Site #2 - Apache/PHP : 100-300ms
>
> > - Site #3 - Apache/PHP : 50-150ms
>
> > - Site #4 - Apache/Tomcat/Java : 50-80ms
>
> > All of these sites have fairly complex back-ends so it's not a matter of
> > serving static files quickly.
>
> > Thanks,
>
> > -Pat
>
> > *From:* make-the-...@googlegroups.com [mailto:
>
> > Your results were for one of your worst pages, not the front-door, right?
> > Even without something like Akamai caching the base page I'd expect the
> > main landing page to largely be cached on the server side (memcache or
> > otherwise). If the Amazon and Newegg results were for unauthenticated (i.e.
> > not personalized versions of the page) then I'd honestly be surprised if
> > they weren't cached.
>
> > That said, even without looking at the variance, the raw times look a lot
> > higher than I would expect. Do you log the response time in you access
> > logs? That's generally where I'm used to looking at it and generally
> > anything over a couple hundred milliseconds for a high-profile page is going
> > to be something to worry about.
>
> > I have really easy access to 95th percentile times from access logs (sites
> > to remain anonymous but of very significant scale):
>
> > - Site #1 - Apache/Tomcat/Java : 200-400ms (known optimization work we have
> > to do around back-end caching)
>
> > - Site #2 - Apache/PHP : 100-300ms
>
> > - Site #3 - Apache/PHP : 50-150ms
>
> > - Site #4 - Apache/Tomcat/Java : 50-80ms
>
> > All of these sites have fairly complex back-ends so it's not a matter of
> > serving static files quickly.
>
> > Thanks,
>
> > -Pat
>
> > make-the-...@googlegroups.com] *On Behalf Of *Jonathan Klein
> > *Sent:* Monday, August 23, 2010 10:37 AM
> > *To:* make-the-...@googlegroups.com
> > *Subject:* Re: [make-the-web-faster:299] Re: Percentile Performance
> > Degradation
>
> > I am also looking at server wait time. When I say "host time" I am using
> > Coradiant's term for server processing time. They break a page up into host
> > time, network time, and page render time. More explicitly, host time is
> > defined as the time between the last packet of the request and the first
> > packet of the response as seen by Coradiant, which is effectively a network
> > tap sitting in our datacenter.
>
> > Pat - there are plenty of Akamai customers that are caching HTML/dynamic
> > content (we used to be one of them :-)). I can't say for sure if Newegg is
> > following that model, but even if they aren't they may be using some sort of
> > dynamic content caching/varnish-like solution for their heavily trafficked
> > pages. We do implement SOLR for our product keyword search, but this page
> > is more of a browsing page where you filter through predefined product
> > attributes. Even on our SOLR driven keyword search page we are seeing an
> > average of 621.3ms with a standard deviation of 548ms - this again being
> > server wait
>
> ...
>
> > I am also looking at server wait time. When I say "host time" I am using
> > Coradiant's term for server processing time. They break a page up into host
> > time, network time, and page render time. More explicitly, host time is
> > defined as the time between the last packet of the request and the first
> > packet of the response as seen by Coradiant, which is effectively a network
> > tap sitting in our datacenter.
>
> > Pat - there are plenty of Akamai customers that are caching HTML/dynamic
> > content (we used to be one of them :-)). I can't say for sure if Newegg is
> > following that model, but even if they aren't they may be using some sort of
> > dynamic content caching/varnish-like solution for their heavily trafficked
> > pages. We do implement SOLR for our product keyword search, but this page
> > is more of a browsing page where you filter through predefined product
> > attributes. Even on our SOLR driven keyword search page we are seeing an
> > average of 621.3ms with a standard deviation of 548ms - this again being
> > server wait
>
>
> read more »- Hide quoted text -
Jonathan Klein
Sep 3, 2010, 11:33:08 AM9/3/10
to make-the-...@googlegroups.com
No, the Nginx logs are just monitoring how long it took Nginx to work on the request (see $request_time here http://wiki.nginx.org/NginxHttpLogModule). So we expect the numbers to be different from the Coradiant report, but not at the level that we are seeing. We just ran another test serving only static content off FusionIO through Nginx with no additional logic. The times in the access logs are stupidly fast:
(all numbers in seconds)
These are the numbers for the Coradiant report monitoring the same traffic:
So clearly the Coradiant box is seeing some latency that Nginx doesn't, but the average correlates well. This Coradiant box is sitting on a span port on our switch, and it is monitoring traffic going to this server only. So there are no strange hops - the path of traffic should look like this:
1. Coradiant sees the last packet of the request as it gets sent to the server, starts timer
2. Server works on the request and starts sending the response (logs how long it took to work on the request).
3. Coradiant sees the first packet of the response from the server and stops the timer.
I have a hard time believing that sending packets inside a datacenter through gigabit networking hardware is causing that much added latency. We are not experiencing any packet loss or packets out of order either according to Coradiant. So at this point we have to either question how Coradiant is getting/measuring data or there is some other step in there that we aren't considering.
-Jonathan
| Average | 0.000141 |
| Std Dev | 0.003319 |
| 80th | 0 |
| 85th | 0 |
| 90th | 0 |
| 95th | 0 |
| 99th | 0.001 |
These are the numbers for the Coradiant report monitoring the same traffic:
| Average | 0.0001 |
| Standard Deviation | 0.069 |
| 80th | 0.109 |
| 85th | 0.116 |
| 90th | 0.123 |
| 95th | 0.13 |
| 99th | 0.136 |
| MAX | 0.137 |
So clearly the Coradiant box is seeing some latency that Nginx doesn't, but the average correlates well. This Coradiant box is sitting on a span port on our switch, and it is monitoring traffic going to this server only. So there are no strange hops - the path of traffic should look like this:
1. Coradiant sees the last packet of the request as it gets sent to the server, starts timer
2. Server works on the request and starts sending the response (logs how long it took to work on the request).
3. Coradiant sees the first packet of the response from the server and stops the timer.
I have a hard time believing that sending packets inside a datacenter through gigabit networking hardware is causing that much added latency. We are not experiencing any packet loss or packets out of order either according to Coradiant. So at this point we have to either question how Coradiant is getting/measuring data or there is some other step in there that we aren't considering.
-Jonathan
Reply all
Reply to author
Forward
0 new messages