Real User Monitoring Tools

[Jonathan Klein]

Hello,

I'm curious what tools people use for real user monitoring on the server side - specifically automated packet sniffers/network taps with reporting capabilities.  We are currently using Coradiant, but we are concerned that it will become prohibitively expensive as we scale and need to buy more hardware from them to handle the traffic/reporting (we are essentially wondering if there are less expensive tools out there with the same general feature set).  I know dynaTrace is popular among some, but they don't support our current stack.  Does anyone know of other good RUM tools?  I'd also be happy to share information about our experience with Coradiant if anyone is interested.  Thanks.

-Jonathan

[Andy Davies]

We use Oracle (ex Moniforce) RUEI which at a guess has the same price model / range and Coradiant (also have Cacti in the mix for some lower level metrics)

RUEI is both a fabulous and frustrating tool, it's easy to get at some information that's really informative e.g. the top three graphs on my dashboard are page load time, server response time and client throughput, I've also used it to dive into client sessions and determine the reason them kept getting errors was that their proxy was setting the length field in a post header to zero.

The out of the box cubes are quite good at exploring what's going on but get frustrating, in that data gets consolidated so some the edge case errors lose detail, also because the raw data goes you also lose the ability to try "what if" scenarios e.g. we wanted to play around with the time for satisfied page page loading time but there's no way to ask the question "if it was x how many pages would fall into satisfied / tolerating and frustrated"

The UI is a bit clunky and although they propose you could use it as a general business tool I don't think it's quite there yet.

I've been keeping an eye out for alternatives to the "vendor solution" and found a few interesting ideas, Pion from atomiclabs.com might be worth a play with but the Community version looks quite limited.

The mod_log_spread approach Theo Schossnagle outlines in Scalable Internet Architectures (http://amzn.to/d0dhgK) looks interesting, although I think it only works with Apache at the moment (not sure about that though)

Theo illustrates using RRDTool with spread but you could also run map/reduce over the data for other analysis and of course it then needs presenting back to the user in a format they can understand (or feeding into another app they use)

So there are possibilities outside Coradiant / Oracle etc. but I guess it depends on where you want to need to go, how much information you need, what time / resources you have.

Tap-based solutions do have the advantage of being able to track things at the TCP as well as HTTP level which can provide some more useful data, but miss out on the javascript level e.g. using TCP you can tell when the data has been ACKed but you can't tell when onload fired, so perhaps we need a mixture of tap and beacon based solutions

I would be interested in learning more Coradiant.

Cheers

Andy

 

@andydavies

--
You received this message because you are subscribed to the Google Groups "Make the Web Faster" group.
To post to this group, send email to make-the-...@googlegroups.com.
To unsubscribe from this group, send email to make-the-web-fa...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/make-the-web-faster?hl=en.

[Jonathan Klein]

Ryan - Those look like nice tools, and I'm definitely a fan of projects that are trying to standardize on/move to the HAR format.  For single waterfall charts I think your second project is going to be the way things go in the future - convert to HAR and import into your favorite tool (HttpWatch, Firebug, etc.).  Too bad about the lack of aggregation, but oh well. 

Andy - Thanks for the info, "fabulous and frustrating" is probably the same way I would describe Coradiant.  All in all it's a pretty great tool and it gives a high level of visibility into what's happening on the server side.  You can look at percentiles, filter to your heart's desire, set SLA's, monitor very specific sections of traffic, etc.  The UI can be a bit clunky, but once you get used to it there are a lot of options.  They also have a "render time" measurement that we are beta testing which is essentially supposed to capture the time between the delivery of the HTML and the onload event.  In our experience this hasn't been too accurate though, and depends fairly heavily on server side performance (for example if a server side event bumps host time up by 1 second for a small window we see the same bump in render time).

Our main complaints with Coradiant are as follows:

- It can take a really long time to drill down into requests.  When you stop looking at high level aggregated statistics and want to look at a group of individual requests (say all image requests taking over X seconds) the report takes multiple minutes to run, which isn't bad if you do this once in a while but definitely gets frustrating if you want to look at a lot of reports quickly.

- Coradiant is great at letting you know when things slow down, but not great at letting you know why.  This is not really a problem with the tool, it has no idea why a given packet (or set of packets) took longer to arrive, but this is the benefit that a product like DynaTrace provides.  When you are in the middle of an incident Coradiant gives you very limited information about what is causing the problem (depending on the nature of the incident).  It can report on client IP, resource size, user agent, etc. but once you start drilling into that kind of data things slow down quite a bit which makes it tough to diagnose an ongoing issue. 

- Coradiant is fairly expensive, and as we move to multiple data centers and our traffic keeps increasing it is only going to get more pricey. 

I'm starting to feel like a lot of the larger companies probably just build something in house to monitor their performance.  For client side performance I really like the idea of Boomerang, but haven't gotten the chance to actually implement it on our sites yet.  The server side is really where we want a good, scalable tool.  Like I said above Coradiant has been really great for us so far, we are just getting to the size where it makes sense to see what other people are doing to see if we can build something in house or if there are other viable options out there. 

Thanks for all the input,

Jonathan

[sabernic]

you can have a look at compuware vantage rum as well. Same price
point, however the toolset is software based and multiple collectors
can talk to one report server, so you can do more with less The
collectors are rated for 10000 or so http hits per second, which is by
far the most scalable rum product on the market. The Coradiant
truesight boxes are rated for a little bit under 2000 hits/sec. The
next version of the release will be able to handle 2 1/2 more times
more traffic. Additionally the tool supports protocols other than
http(s), including database decodes such as sql, orcl, ibm and
middleware like mq, and jolt.


Regards,

~S

> > On 4 October 2010 20:50, Jonathan Klein <jonathan.n.kl...@gmail.com>wrote:
>
> >> Hello,
>
> >> I'm curious what tools people use for real user monitoring on the server
> >> side - specifically automated packet sniffers/network taps with reporting
> >> capabilities.  We are currently using Coradiant, but we are concerned that
> >> it will become prohibitively expensive as we scale and need to buy more
> >> hardware from them to handle the traffic/reporting (we are essentially
> >> wondering if there are less expensive tools out there with the same general
> >> feature set).  I know dynaTrace is popular among some, but they don't
> >> support our current stack.  Does anyone know of other good RUM tools?  I'd
> >> also be happy to share information about our experience with Coradiant if
> >> anyone is interested.  Thanks.
>
> >> -Jonathan
>
> >> --
> >> You received this message because you are subscribed to the Google Groups
> >> "Make the Web Faster" group.
> >> To post to this group, send email to make-the-...@googlegroups.com
> >> .
> >> To unsubscribe from this group, send email to

> >> make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsu...@googlegroups.com>
> >> .

> >> For more options, visit this group at
> >>http://groups.google.com/group/make-the-web-faster?hl=en.
>
> >  --
> > You received this message because you are subscribed to the Google Groups
> > "Make the Web Faster" group.
> > To post to this group, send email to make-the-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsu...@googlegroups.com>
> > .

[Peter L]

Can anyone share ballpark pricing info on these two RUM solutions? Coradiant and Oracle?

Thanks in advance,

Peter

To unsubscribe from this group, send email to make-the-web-fa...@googlegroups.com.

[Timothy Fisher]

Here is where you can find more information about the Gomez RUM solution, ActualXF.  Gomez is now owned by Compuware also.

http://goo.gl/PZqb


Tim

[Andy Davies]

I'll see what I can dig out on the Oracle front...

>> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252Buns...@googlegroups.com>

>> >
>> > >> .
>> > >> For more options, visit this group at
>> > >>http://groups.google.com/group/make-the-web-faster?hl=en.
>> >
>> > > --
>> > > You received this message because you are subscribed to the Google
>> Groups
>> > > "Make the Web Faster" group.
>> > > To post to this group, send email to
>> make-the-...@googlegroups.com.
>> > > To unsubscribe from this group, send email to
>> > > make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsu...@googlegroups.com>

>> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252Buns...@googlegroups.com>

[Jonathan Klein]

It's tough to give you a number for Coradiant since the cost could vary widely depending on your specific setup.  Typically these sorts of solutions are in the low tens of thousands and up depending on scale.  It also depends on if you are purchasing a piece of hardware from the company or if you are using something that's already in your datacenter.  Hopefully Andy can give you more specifics as far as Oracle goes, but if you are fairly serious about it I recommend giving the folks at Coradiant a call - I'm sure they will be happy to turn around quickly with a quote.  Everyone we have dealt with there has been extremely helpful and friendly.  Good luck.

-Jonathan

[Peter L]

Thanks for the info, guys.

Peter

[Jared Rosoff]

Hi Jonathan,

We've been working on a solution that uses javascript + google
analytics custom variables to measure real user performance. I just
published our our proof of concept and some early data to our blog.
It's not a packet sniffer and it definitely needs lots of work, but
it's a good start and has the benefit of being 100% free.

Check out the post here:

http://blog.yottaa.com/2010/10/how-to-measure-page-load-time-with-google-analytics/

Love to hear your feedback.

BTW sorry to have missed boston speed meetup this week. Looking
forward to the next one!

-j

[Jonathan Klein]

Hi Jared,

Thanks for the link, there have been a couple of posts recently about using GA to measure load time, but I think yours is the most in depth and comprehensive one to date.  It's definitely something I'd like to try on our sites, we get render time numbers from Coradiant but it's always nice to have more data.  I particularly like the quick and easy correlation between load time and bounce rate.  Hopefully I'll get a chance to set this up in the next couple weeks and share the results. 

No problem about the Meetup, I'm glad you are planning on attending in the future.  It would be great to have someone from Yottaa speak at some point about your service (and perhaps about the scalability challenges).  Let me know if that sounds interesting to you.

Cheers,

Jonathan

[Peter L]

I missed the Meetup notice. Would have liked to have gone to the one in Boston also.

Was it sent to this list?

Thanks,

Peter

[Jonathan Klein]

I actually didn't send it to this list - sorry about that, sounds like I need to work on publicizing the Meetups a little better.  For future reference this is the link to the Meetup group:

http://www.meetup.com/Web-Performance-Boston/

I'll be sure to reach out to more channels once I schedule the next Meetup. 

-Jonathan

[Martin Borthiry]

Hi Jonathan,  

I wrote time ago a similar post. But I was focused on take many real user measures, like < head > load time, ads load time, etc.

http://tips.freedev.com.ar/en/wpo/google-analytics-web-performance-tracker/

The real problem of using GA as stats store is the charts. They sucks. 

Furthermore, Steve souder remember me his Episodes project (http://stevesouders.com/episodes/). I know his is workling on a new version. 

Good luck!

Martín

[Timothy Fisher]

You might also be interested in checking out the Boomerang project from Yahoo.  Boomerang is a JavaScript tag that lets you add performance monitoring to your web pages relatively painlessly.  It falls into the category of Real User Monitoring also.

http://github.com/yahoo/boomerang

Tim

On Fri, Oct 22, 2010 at 8:39 AM, Jonathan Klein <jonathan...@gmail.com> wrote:

[Jonathan Klein]

Yeah I've looked at Boomerang (and Philip Tellis actually spoke at the Boston Web Performance Meetup about it), and I think it's a great tool.  We haven't had the chance to implement it on our sites since we are all swamped with other stuff, but it's on my to do list.  I love that it gives bandwidth/latency measurements and that it uses the Web Timing API whenever possible - definitely useful stuff.  Now we just need everyone to upgrade to browsers that support the API so we can get really good data :-).  Maybe in 5-10 years...

-Jonathan

[Andy Davies]

The list price on our RUEI installation in 75,000 Euro pet CPU but I'm assured we didn't pay anything like that.

Still expensive, and I reckon there's an opportunity for a good OS solution.

It can't be that difficult to scoop data via a tap, stick it on  DB and run map/reduce on it.

Andy

>>> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252Buns...@googlegroups.com>
> <make-the-web-faster%252Buns...@googlegroups.com<make-the-web-faster%25252Bun...@googlegroups.com>

>>
>>> >
>>> > >> .
>>> > >> For more options, visit this group at
>>> > >>http://groups.google.com/group/make-the-web-faster?hl=en.
>>> >
>>> > > --
>>> > > You received this message because you are subscribed to the Google
>>> Groups
>>> > > "Make the Web Faster" group.
>>> > > To post to this group, send email to
>>> make-the-...@googlegroups.com.
>>> > > To unsubscribe from this group, send email to
>>> > > make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsu...@googlegroups.com>
> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252Buns...@googlegroups.com>
>>

>>> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252Buns...@googlegroups.com>
> <make-the-web-faster%252Buns...@googlegroups.com<make-the-web-faster%25252Bun...@googlegroups.com>

>>
>>> >
>>> > > .
>>> > > For more options, visit this group at
>>> > >http://groups.google.com/group/make-the-web-faster?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "Make the Web Faster" group.

>>> To post to this group, send email to make-the-...@googlegroups.com

[Peter L]

"It can't be that difficult to scoop data via a tap, stick it on  DB and run map/reduce on it."

I like how simple you make that sound. What algorithm would you use to analyze the data? How would map/reduce be employed? Seriously I'm interested in working on this but am curious to know what you see as how to implement using parallel/distributed techniques.

Thanks,

Peter

To unsubscribe from this group, send email to make-the-web-fa...@googlegroups.com.

[Andy Davies]

On 29 October 2010 15:56, Peter L <bizzb...@gmail.com> wrote:

"It can't be that difficult to scoop data via a tap, stick it on  DB and run map/reduce on it."

I like how simple you make that sound. What algorithm would you use to analyze the data? How would map/reduce be employed? Seriously I'm interested in working on this but am curious to know what you see as how to implement using parallel/distributed techniques.

Yeh, I did make it sound rather simple!

 

I saw a presentation on how last.fm used Hadoop to analyse their logs, and then remembered this post from earlier in the year - http://blog.johngoulah.com/2010/01/using-mongo-and-map-reduce-on-apache-access-logs/  - using MongoDB gives a nice environment to experiment and prototype with map/reduce (same probably applies to CouchDB and others)

By keeping the raw data and analysing it (rather than rolling it up like some of the tools do), it's possible to add new analysis later and try what-if situations.

The first analysis I'd probably do are the common ones seen in most webstats tools e.g. count the number of visits to individual URLs over a set periods of time, rank URLs by their ApDex etc.

What I've not really though about is whether I'd use this process for live monitoring e.g. current average page load times, or live ApDex etc.

It's one of those things I'd like to have a play with but I have other "fish to fry" on the technical front at the moment so haven't got round to it.

I've also not thought very hard about the network tap side of things and the challenge of scooping up large amounts of data keeping the bits you need and then chucking the rest away.

Cheers

Andy

[Peter L]

Thanks for the ideas, Andy. I'll let you know if I make any progress in this direction and what I find.

Peter

[Andy Davies]

One of our ops guys thinks I'm nuts for even contemplating it mind...

>> make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsu...@googlegroups.com>

[TheOpsMgr]

Hi Andy,

Os-based network Tap solution with intelligent protocol aware
clickstream analysis...

Atomic Labs Pion.

At a cost of 1/10 the REUI solution.

Enjoy!

Steve

[Andy Davies]

Yeh, I'd come across it but not had a play.

Have you used it anywhere?

Cheers

Andy

[Oren]

I have been following the information in this thread closely and I
think there are a lot of good valid point brought up.
The costs of Real User Monitoring solution are extremely expensive,
while only delivering a part of the overall required solution. Somehow
the technology became increasingly commoditized, but the prices still
remain quite high.
I wanted to note that our company has recently released a fully
functional enterprise grade RUM solution completely free for one year.
This solution is software based and provides browser rendering as well
as network latency.
Next year we will probably offer two or three editions to this, but
the prices will still remain much lower than any of the figures noted
in these posts.
If you are interested in additional information, you can look at
www.real-user-monitoring.com, or just reply with any questions.

-Oren

On Oct 28, 4:04 pm, Andy Davies <dajdav...@gmail.com> wrote:
> The list price on our RUEI installation in 75,000 Euro pet CPU but I'm
> assured we didn't pay anything like that.
>
> Still expensive, and I reckon there's an opportunity for a good OS solution.
>
> It can't be that difficult to scoop data via a tap, stick it on  DB and run
> map/reduce on it.
>
> Andy

> On 11 Oct 2010 22:54, "Andy Davies" <dajdav...@gmail.com> wrote:
>
>
>
>
>
>
>
> > I'll see what I can dig out on the Oracle front...

> > On 11 Oct 2010 17:59, "Peter L" <bizzbys...@gmail.com> wrote:
> >> Can anyone share ballpark pricing info on these two RUM solutions?
> > Coradiant
> >> and Oracle?
>
> >> Thanks in advance,
>
> >> Peter
>

> >>> > >> make-the-web-fa...@googlegroups.com<make-the-web-faster%2Bunsu bsc...@googlegroups.com>
>
> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252 Bunsub...@googlegroups.com>
>
> > <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252 Bunsub...@googlegroups.com>
>
> <make-the-web-faster%252Buns...@googlegroups.com<make-the-web-faster%2 5252Buns...@googlegroups.com>
>
> >>> <make-the-web-faster%2Bunsu...@googlegroups.com<make-the-web-faster%252 Bunsub...@googlegroups.com>
>
> <make-the-web-faster%252Buns...@googlegroups.com<make-the-web-faster%2 5252Buns...@googlegroups.com>
>
>
>
>
>
>
>
>
>
> > <make-the-web-faster%252Buns...@googlegroups.com<make-the-web-faster%2 5252Buns...@googlegroups.com>
>
> ...
>
> read more »