Varying Response Times - Controller Vs Manual

[Arvind Sagar]

Hi All,

I have run out of ideas and it would be really helpful if you can
share some fresh perspective in this scenario:

I am noticing largely varying response times, when a script is
executed from the controller, comparing to the same test case being
executed manually.

Scenario:

The page is developed in jsf and has flash components.
The test case under discussion involves launching the home page of the
target application and logging in.

We have two machines at disposal, one is a machine (Mach-L) located in
my workplace and the other is a machine (Mach-R), that is
geographically seperated.

Mach-L : 3GB Ram, 2 Ghz Dual Core
Mach-R : 4GB Ram, i3 processor

The target application's server and Mach-R are at proximity to each
other compared to Mach-L.

Protocol used:

Web + AMF (Multi Protocol)

What I have done so far:


1) Comparing response times:

These timelines are based on the same test case, for the same
credentials.

Mach-L - Manual : ~120 seconds
Mach-L - Vugen : ~120 seconds
Mach-L - Controller : ~115 seconds

Mach-R - Manual : ~13 seconds
Mach-R - Vugen : ~13 seconds
Mach-R - Controller : 1.8 seconds

2) Proxy settings:

I have tried altering the proxy settings in runtime settings, to match
with the browser and also to directly connect to the internet, and by
manually specifying the proxy....No changes

3) Trying different Load Generators:

Localhost : 1.8 seconds
Mach-A : ~115 seconds

4) Regenerating to URL / HTTP :

Not the best of options, but still gave it a try. Same output

5) Comparing flash player versions :

Both the machines are running at the same flash versions - 11.3.02

6) Comparing LoadRunner versions :

Mach-L runs on 9.5 and Mach-R runs on 11.0

7) Verifying checkpoints

The checkpoints have been placed for the dynamic data as well as page
titles.


Is there something that I am missing?

[Mital Majmundar]

Hello arvind,

Configure the "Network delay graph" graph for the machine which is farther away from the application server. It seams you have an issue with network delay.

--
You received this message because you are subscribed to the Google "LoadRunner" group.
To post to this group, send email to LR-Loa...@googlegroups.com
To unsubscribe from this group, send email to
LR-LoadRunne...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/LR-LoadRunner?hl=en

[James Pulley]

It is unclear from your description whether Mach-L and Mach-R are load
generators or are both load generator and controller combinations. But, if
these are controllers then you should not be executing virtual users on
them, it is considered bad practice. You're also throwing between
LoadRunner versions. You need to simply your diagnoses, keep the machines
constant, keep the version of LoadRunner constant and change one element.
As you are changing multiple elements concurrently you are having an
expectedly difficult time trying to remove the influence of one item to
check the influence of others. This is a testing process issue.

The second set of issues surround your test bed conditions. By not having
identical load generators you make your job of comparing one to another.
One load generator may be overloaded while the other may be healthy from a
resource perspective. It is considered a best practice to have matching
load generators with matching initial conditions on each prior to your test
execution.

You have a deliberate inclusion of an uncontrolled element in your test
design, which is the complex network between the application under test and
Mach-L. What you are seeing on Mach-L could very well be representative of
what a user would see from the same location given the number of application
turns and the amount of data being sent back and forth at the network layer.
But because you have this uncontrolled element in your test design your job
of determining application scalability vs network scalability vs LAN|WAN
antagonistic characteristics of your application difficult. Collapse your
network for your application test. If you must involve locations across
and uncontrolled network element then load a single virtual user as a
control element in those locations for the measurement of network affect on
the applications.

Draw back to basics on test construction. Minimize the number of
uncontrolled elements in your test design and your test bed. Purchasing and
installing additional load generators is far less expensive than the cycles
you have likely already burned trying to figure out why this is occurring
and it will certainly be less than the cycles burned by the application
group if you provide them a spurious performance defect. As your company
has invested in performance testing tools and a performance testing staff,
then should be willing to invest in a set of performance test infrastructure
to fortify the test results.

James Pulley

[Arvind Sagar]

Hello Mital,


Actually my problem statement is that, I am seeing extremely varying
response times with the controller in Mach-R and when manually tried
in Mach-R.

What I mean to say is, the controller's response time should be
atleast reasonably comparable to what is achieved manually, which is
not happening in my case.

I am not able to position network delay in this scenario. Could you
kindly elaborate a little more?

Thanks for the help!


Hi James,

Thank you for the detailed response.

I am just trying to collate and paraphrase the highlights of what you
have mentioned:

1) Controller as Load Generator : It is not a good idea to use a
machine hosting the controller, as a load generator.

Please do correct me if am wrong, but I am just trying a smoke test
using a single user. I am under the impression that using the
controller machine as an LG for one user shouldn't actually mess up
the results.

2) Varying LR versions : Both the machines have housed different
loadrunner versions

Once again James, I am in dire need of your expertise with this. One
of the main purpose of the LoadRunner tool is to provide accurate
response times, immaterial of the version. Is it actually possible
that different versions of LoadRunner can provide varying response
times for the same script?

3) Identical Load generators: Both the generators (viz., the
controller machines) are powered by different system configurations

Yes, this will be an ideal candidate that can produce different
results. But even this will come into picture only when I am
performing an actual load test using a good number of users correct?
And not when I am running using a single Vuser.

4) The geographical split up : the network between the AUT and the
Mach-L could be a plausible culprit

My actual problem is that, the response times are not being
consistent in terms of a ratio on Mach-R and Mach-L. With the network
delay in picture, the Mach-L produced almost similar response times
when tried manually, using Vugen and Controller, whereas, the Mach-R,
produced similar response times when tried manually and using Vugen,
but a different response time using the controller.

What I am exactly trying to say is this:
Mach-R Response time / Mach-L Response time = ~Constant, as long as
the methodology is the same.

I am really feeling confident now that I have another fresh pair of
eyes looking at this problem too.

These are the next steps that I am having inline:

1) Try the script using only Web Protocol and compare response times
(Mach-L Vs Mach-R)
2) Trace the response times of individual web_url calls (Mach-L Vs
Mach-R)

It would really mean a lot if you can respond to this when you find
time.