Feature Request: Directory Structure

[rstens]

When working on a few project at a time I would like to be able to
start multi-mechanize with a parameter that indicates my project.

Like: python multi-mechanize.py project1

I suggest the following directory structure to support this:

multi-mechanize
lib
tools
multi-mechanize.py
projects
project1
config.cfg
test-scripts
results
project2
config.cfg
test-scripts
results

So every project has their own config.cfg file.
Maybe the actual projects dierectory could be made a configurable
element for multi-mechanize?
Or you can simply refer to the project directory with the start up
parameter.

[Amol Jadhav]

May be once we standardize on the directory structure we may also be
able to write a Web/Desktop application which will allow us to browse
through the test results, comparing them and so on. For example, I
often asked to compare test results of two scenarios ran with
different number of virtual users.

Regards,
Amol

[Corey Goldberg]

> I suggest the following directory structure to support this:

> So every project has their own config.cfg file.

I agree with the need for better structure and a way to have multiple
active projects without a lot of copying config files around.

I will look over the proposed directory structure and implement
something that makes sense.

more updates soon....

-Corey

[Corey Goldberg]

> May be once we standardize on the directory structure we may also be
> able to write a Web/Desktop application which will allow us to browse
> through the test results, comparing them and so on.

Amol,
thanks for the input.
I wasn't planning on writing a "results viewer", but I can see how it
might be useful for browsing and comparing results.

I'm gonna push this off for a while and hopefully get back to working
on something like this in a few weeks... unless you wanted to
contribute it yourself, in which case I can help sooner.

-Corey

[corey goldberg]

> I suggest the following directory structure to support this:

I just implemented the new directory structure pretty much exactly as
Roland described.

The new code is in SVN if anyone wants to try it. I checked in a
"default_project" with the code base.... so to run the new code, do:

>python multi-mechanize.py default_project

I will be updating the documentation and doing a release soon.

-Corey

[Amol Jadhav]

> I'm gonna push this off for a while and hopefully get back to working
> on something like this in a few weeks... unless you wanted to
> contribute it yourself, in which case I can help sooner.

I would love to work on this and many other things, I also have done
comparison of two results of some different tool using Matplotlib. I
have attached a sample graph of the work, check out if you like it.
I'd also have to understand the current report format and design
accordingly. Please provide any inputs on your expectations.

Thanks,
Amol

[Corey Goldberg]

> I would love to work on this and many other things,

great! i'm very open to ideas and collaboration in any areas of the project.

> have attached a sample graph of the work, check out if you like it.
> I'd also have to understand the current report format and design
> accordingly. Please provide any inputs on your expectations.

your graph looks great... can you explain more about what you are proposing?

let's say you have completed several test runs and you have a pile of
results directories... how would you go about viewing/comparing
results?

-Corey

[corey goldberg]

> I would love to work on this and many other things, I also have done
> comparison of two results of some different tool using Matplotlib.

also, check this thread: http://groups.google.com/group/multi-mechanize/browse_thread/thread/df525ad6fdb8bf85

we are discussing adding a db layer as an optional output method. so
basically, a database gets populated with your results as a test
runs. I think we would want to build any results comparison or
results viewing features on top of the database. thoughts?

-Corey

[Brian Knox]

Corey

Just a heads up, I'm doing test runs of the DB layer this evening and things are working great.  I have a little bit of fiddling here and there to do before I'm satisfied, but we're definitely a go.

Is there any particular format you would like the changes in?  Do you just want a tarball of my copy of 1.007 with the changes I've made, or, patch files... whatever is easiest on you is fine by me.

Brian

[corey goldberg]

> Is there any particular format you would like the changes in?  Do you just
> want a tarball of my copy of 1.007 with the changes I've made

yeah, a tarball of your copy would be great.

you can email it to me: http://www.goldb.org has my contact info

i will probably either try it with mysql or sql server. along with
the code, let me know if you have any additional tips or guidance for
setting it up.

thanks!

-Corey

[Amol Jadhav]

>
> let's say you have completed several test runs and you have a pile of
> results directories... how would you go about viewing/comparing
> results?
>

Right now I'm thinking of simple console application which will take
two (or more) results directories as input. Once this initial work has
been done we can write a wrapper around this which will scan the
result directory of a project and allow user to select which results
to compare with each other. I will get this console done by this
weekend.

Same is the case with database stored results we can defer this by
abstracting away the results reader (from directories/files or from
db).

~ Amol

[Amol Jadhav]

Also, right-now I'm only thinking comparison of response time as the
only graph provided we will refine this as we get more clear on this,
is that fine?

[Corey Goldberg]

> Also, right-now I'm only thinking comparison of response time as the
> only graph provided we will refine this as we get more clear on this,
> is that fine?

Amol,
that sounds good. my advice is to start small. just get some basic
comparisons going, so we can review your code and see how it fits with
the overall structure of the project.

-Corey

[Amol Jadhav]

Hi Corey,

I'm almost done with comparing two results having same number of
user-groups. However, I'm facing some problems with parsing results.

I'm using your 'results' module to parse the results. The module is
perfectly fine, but it requires run_time, which comes from the config.
This config may not be the same after multiple test runs and one
cannot create Result object without correct run_time.

One option to get correct run_time is to parse the result.html. But
HTML parsing is hard and will make code hard to understand.

I couldn't get the database thing working because MySQLdb module
couldn't get installed on my machine. I'm working on this but can you
suggest any other way of getting run_time and other config specific
options?

Thanks,
Amol

[Amol Jadhav]

>
> One option to get correct run_time is to parse the result.html. But
> HTML parsing is hard and will make code hard to understand.

I'm trying to get it done using ElementTree.

[Brian Knox]

Just a heads up that the run time is stored in the database (if you go with the results database option which is in subversion but not in a release yet) as run_id in the table.  You could also parse the directory name the results are stored as a possible option.

year, month, day, hour, minute, second = 'results_2010.02.26.14.10.58'.split('_')[1].split('.')

[Amol Jadhav]

> year, month, day, hour, minute, second =
> 'results_2010.02.26.14.10.58'.split('_')[1].split('.')

This gives me test start time (or more precisely result directory
creation time) and not the duration (run_time) for which test ran.

Right I'm thinking of this

>>> import xml.etree.ElementTree as ET
>>> tree = ET.parse('results.html')
>>> div = tree.find('.//{http://www.w3.org/1999/xhtml}div')
>>> div.getchildren()[4].tail # this gives me runtime.

Now, I want to get total number of threads for the test. I will use
them to differentiate between two test. Can anyone thinking of better
way to differentiate?

[Brian Knox]

Oh! you're looking for the length the test ran, my apologies.  I'm still on my first cup of coffee this morning!

I think if there's additional data beyond what's stored in the results.csv that's needed for reporting, we might want
to take a look at getting that data into the results.csv file itself (or some additional csv files in the results directory). 
I personally think using reporting html as a source of data is probably the wrong road to go down as far as a permanent solution,
although it's certainly a workable ad-hoc solution.

Corey, thoughts?

[Amol Jadhav]

> Oh! you're looking for the length the test ran, my apologies.  I'm still on
> my first cup of coffee this morning!

:-)

>
> I think if there's additional data beyond what's stored in the results.csv
> that's needed for reporting, we might want
> to take a look at getting that data into the results.csv file itself (or
> some additional csv files in the results directory).

May be we can copy the config into the results directory. This will
also be a reference point for the user if he/she wants to find out
what config gave corresponding results.

> I personally think using reporting html as a source of data is probably the
> wrong road to go down as far as a permanent solution,

agreed!

[Brian Knox]

I think having the config setting stored in the results directory after a run would be useful as well.  Perhaps multi-mechanize should copy the config to the results directory for that run, and I could also modify resultsloader.py to store that data in the results database for people who want to use SQL to query their results.

Brian

[Corey Goldberg]

> May be we can copy the config into the results directory. This will
> also be a reference point for the user if he/she wants to find out
> what config gave corresponding results.

+1 i think its a good idea to have config.cfg in the results
directory.. as a reference point to look back on, and also as a data
source for whatever post processing/comparison happens.

so you can assume that config.cfg will be in the results directory,
and develop from there.

(brian already sent me a patch for this so i will add this
functionality in soon).

thanks guys!

-Corey

[Corey Goldberg]

> May be we can copy the config into the results directory.

just checked in the code for this. you can grab the latest SVN trunk.

-Corey

[corey goldberg]

> I couldn't get the database thing working because MySQLdb module
> couldn't get installed on my machine.

i've been testing with SQLite. no setup and it works great. give
that a try if you have any mysql headaches.

-Corey

[Dalibor Aleksov]

HI guys, I know this is an old post but I couldn't find anything else regarding the test result comparison.

So, is there any tool (anything) that can collate multiple results and analyze them or just combine the raw results and put them all toghter