Summarizing my thoughts on Testerman from a private email thread

[Saravanan Shanmugham]

Some next steps that might be needed to gather momentum for Testerman
    1. It would be good to have a mailing group started at groups.google.com 
possibly. [Done] :-)
    2. Move your code base to code.google.com[Use the mercurial SCM instead of SVN]. Would highly recommend it. 
Allowing others to pull branches and contribute more easily. 
    3. Take a look at how WAF documentation and release. I am not the author but I contribute to WAF and it does source code 
management, release and documentation well. http://code.google.com/p/waf/. Would be a good model to follow for documentation. 
Thomas, the author does very good documentation. You are pretty good yourself. But I would suggest looking at it for some ideas to help your documentation be better.
   4. Testerman  needs to be more pythonic in its design of Syntax. In an attempt to model closely TTCN-3 it has lost its way in syntax and is less readable and less simpler than it can be. Once all the constructs are available in a very pythonic syntax one can always write a separate compiler that takes pure TTCN-3 and translate it to Testerman's pythonic syntax. I have some suggestions for a more pythonic syntax below.
 
I took the example of Testerman script at
http://www.testerman.fr/testerman/wiki/TestermanTTCN3

and gave it a try being more pythonic.  

ParisWeekendWeatherRequest=Template( 
    location = 'Paris', 
    date = '15/06/2006', 
    kind = 'actual', )
ParisResponse=Template( 
    location = 'Paris', 
    date = '15/06/2006', 
    kind = 'actual', 
    temperature' = between(15, 30),    
    windVelocity' = between(0, 20),    
    conditions' = 'sunny',
)  

@runson(MTCType)  # May be?
@testcase    # Nose and Py.test does not need either and is able know test functions without this. 
def testWeather(self):    
    weatherOffice = self.mtc['weatherOffice']    
    weatherOffice.send(ParisWeekendWeatherRequest)    
    with alternate() as ctx:        
        @choice(ctx, weatherOffice.receive, ParisResponse)        
        def body(): setverdict('pass')            

        @choice(ctx, weatherOffice.receive)        
        def body(): setverdict('fail')  

#  Is this really necessary? Nose and Py.Test seem to able to do without it. Why is this needed in TTCN-3?
#  testWeather().execute()

Thx,
Sarvi

[Sebastien Lefevre]

Hi Shavi, and big thanks for your support !

2. Actually, I was thinking about moving to git instead of mecurial; in any case, it's clear we need a distributed SCM and I'll try to migrate during the next coming weeks. In the meanwhile, just ask me if you want an SVN account.
Regarding the hosting, I'll probably use another provider to keep the source integrated with the trac wiki, as most of the test adapters (probes) and codecs wiki pages are generated dynamically from the source code.

3. I'll have a look, I didn't spend a lot of time on documentation these days, focusing on introducing new features - not really a best practice, but we need to have several things work first :-)

4. svn commit #378 introduced a way to define "language APIs" on server side (conf/language-apis.conf, restart the server (bin/testerman-admin restart server) to take the change into account) that can then be referenced by an ATS.
From QTesterman, just update the "language API" Script property to the new API you declared on server side.
This way, you can use an alternative TTCN3 adapter module instead of the default TestermanTTCN3, and it should be easier to try new syntax approaches without breaking the current ATSes.

If you want to start experimenting a decorator-oriented syntax as you suggest, you may, for instance, copy core/TestermanTTCN3.py as core/PythonicTTCN3.py, and add a line in conf/language-apis.conf:

testerman.te.python.module.api.2 = PythonicTTCN3

When creating an ATS from QTesterman, just set the language API to "2", and your specialized module will be used.

I'll try to experiment myself later during the week.


Just some hints:

when defining a template like:

ParisWeekendWeatherRequest=Template( 
    location = 'Paris', 
    date = '15/06/2006', 
    kind = 'actual', )

you'll have a problem when you'll have to match fields whose name contains dashes, dots, etc.

Let's imagine you have an XML structure received on the network as:
<something>value</something>
<something-else>another value</something-else>

You'll probably want to match it with a template that would look like:
t = Template(
    something = 'value',
    something-else = 'another value'
)

but this won't be a valid Python syntax, unfortunately. Adopting a convention to turn dashes, dots, etc into a character suitable to define a Python identifier would prevent from creating some tests and is not a solution.

--

Function decorators, especially to declare testcases, sound a very good idea. To investigate.
There is probably something to do with the alt() statement too. I'm not happy nor proud with the current "lambda in a list" model, especially because it forces to implement a TTCN3/alternative in an external function as soon as you need to assign a value to a variable (Testerman provide a StateManager object to help that, though), or a conditions, loops, etc.

--

Regarding the "control part", TTCN-3 makes it mandatory (if I remember well).
Actually, it is convenient to programmatically select the test cases to run in an ATS, depending on several external parameters (TTCN-3 can group them, too) or at runtime (useless to run some testcases if one particular failed, etc), as well as running the same testcase multiple times with different arguments, effectively testing different aspects.

For instance:

TC_ERROR_TEST(id_suffix = 'NUMERIC_INPUT').execute(invalidValueToSend = "12345", expectedError = "Invalid characters")
TC_ERROR_TEST(id_suffix = 'TOO_SHORT').execute(invalidValueToSend = "abc", expectedErrorCode = "You must enter at least 5 characters")
TC_ERROR_TEST(id_suffix = 'TOO_LONG').execute(invalidValueToSend = "abcdefghjklm", expectedErrorCode = "You must enter at most 10 characters")

this will generate 3 different test case IDs in execution reports, thanks to the "id_suffix" arg:
TC_ERROR_TEST_NUMERIC_INPUT
TC_ERROR_TEST_TOO_SHORT
TC_ERROR_TEST_TOO_LONG


thanks,
-seb

[sarvi]

some comments inline

On Sep 26, 2:39 pm, Sebastien Lefevre <seb.lefe...@gmail.com> wrote:
> Hi Shavi, and big thanks for your support !
>
> 2. Actually, I was thinking about moving to git instead of mecurial; in
> any case, it's clear we need a distributed SCM and I'll try to migrate
> during the next coming weeks. In the meanwhile, just ask me if you want
> an SVN account.
> Regarding the hosting, I'll probably use another provider to keep the
> source integrated with the trac wiki, as most of the test adapters
> (probes) and codecs wiki pages are generated dynamically from the source
> code.

I did a SCM comparison for my team last year and went with Mercurial.
1. Windows Support
2. Much much simpler workflow/interface
did it for Mercurial.
Git is faster, but in practical terms Mercurial is pretty close.

Your a pythonista like me anyway. So I thought you might like
Mercurial :-))

anyway, no sweat. whatever works for you.

Wherever you host it, you can always autosync a copy into your current
local machine to do your wiki generation.

>
> 3. I'll have a look, I didn't spend a lot of time on documentation these
> days, focusing on introducing new features - not really a best practice,
> but we need to have several things work first :-)
>
> 4. svn commit #378 introduced a way to define "language APIs" on server
> side (conf/language-apis.conf, restart the server (bin/testerman-admin
> restart server) to take the change into account) that can then be
> referenced by an ATS.
>  From QTesterman, just update the "language API" Script property to the
> new API you declared on server side.
> This way, you can use an alternative TTCN3 adapter module instead of the
> default TestermanTTCN3, and it should be easier to try new syntax
> approaches without breaking the current ATSes.
>
> If you want to start experimenting a decorator-oriented syntax as you
> suggest, you may, for instance, copy core/TestermanTTCN3.py as
> core/PythonicTTCN3.py, and add a line in conf/language-apis.conf:
>
> testerman.te.python.module.api.2 = PythonicTTCN3
>
> When creating an ATS from QTesterman, just set the language API to "2",
> and your specialized module will be used.
>
> I'll try to experiment myself later during the week.

Thx, will give it a shot.

>
> Just some hints:
>
> when defining a template like:
> ParisWeekendWeatherRequest=Template(
>      location = 'Paris',
>      date = '15/06/2006',
>      kind = 'actual', )
>
> you'll have a problem when you'll have to match fields whose name
> contains dashes, dots, etc.
>
> Let's imagine you have an XML structure received on the network as:
> <something>value</something>
> <something-else>another value</something-else>
>
> You'll probably want to match it with a template that would look like:
> t = Template(
>      something = 'value',
>      something-else = 'another value'
> )
>
> but this won't be a valid Python syntax, unfortunately. Adopting a
> convention to turn dashes, dots, etc into a character suitable to define
> a Python identifier would prevent from creating some tests and is not a
> solution.

t=dict({'hello-how' : 'value'})
works.

So, If Template was derived from dict() class, the following should
work for the above case.
t = Template({
     'something' : 'value',
     'something-else' : 'another value',
})
and the following should work for everything else.

t = Template(
     something = 'value',

     somethingelse = 'another value'
)

>
> --
>
> Function decorators, especially to declare testcases, sound a very good
> idea. To investigate.
> There is probably something to do with the alt() statement too. I'm not
> happy nor proud with the current "lambda in a list" model, especially
> because it forces to implement a TTCN3/alternative in an external
> function as soon as you need to assign a value to a variable (Testerman
> provide a StateManager object to help that, though), or a conditions,
> loops, etc.
>
> --
>
> Regarding the "control part", TTCN-3 makes it mandatory (if I remember
> well).
> Actually, it is convenient to programmatically select the test cases to
> run in an ATS, depending on several external parameters (TTCN-3 can
> group them, too) or at runtime (useless to run some testcases if one
> particular failed, etc), as well as running the same testcase multiple
> times with different arguments, effectively testing different aspects.

If it were upto me, I would pick the best of the various frameworks
and not just stick to just TTCN-3 only.
TTCN-3 does not always get it right, though it does in a quite a few
areas.


TTCN-3 Control is not my favourite way to orgnize test cases.
The nose and python detect and organize tests without 'Control'
section.
And where they need hints, something like the nose @attr() helps.
Unittest2 also has the suite concept, which is also worth a look.

In general less code is better :-))

>
> For instance:
>
> TC_ERROR_TEST(id_suffix = 'NUMERIC_INPUT').execute(invalidValueToSend =
> "12345", expectedError = "Invalid characters")
> TC_ERROR_TEST(id_suffix = 'TOO_SHORT').execute(invalidValueToSend =
> "abc", expectedErrorCode = "You must enter at least 5 characters")
> TC_ERROR_TEST(id_suffix = 'TOO_LONG').execute(invalidValueToSend =
> "abcdefghjklm", expectedErrorCode = "You must enter at most 10 characters")
>
> this will generate 3 different test case IDs in execution reports,
> thanks to the "id_suffix" arg:
> TC_ERROR_TEST_NUMERIC_INPUT
> TC_ERROR_TEST_TOO_SHORT
> TC_ERROR_TEST_TOO_LONG

Nose and py.tests do generative and parametric test case very well.
You might want to take a look at that.
http://somethingaboutorange.com/mrl/projects/nose/0.11.2/writing_tests.html#test-generators
http://tetamap.wordpress.com/2009/05/13/parametrizing-python-tests-generalized/
http://www.voidspace.org.uk/python/weblog/arch_d7_2009_06_13.shtml

unittest2 is something worth a look too.
http://www.voidspace.org.uk/python/articles/unittest2.shtml

Thx,
Sarvi
>
> thanks,
> -seb
>
> Le 26/09/2010 19:29, Saravanan Shanmugham a �crit :

>
>
>
> > Some next steps that might be needed to gather momentum for Testerman
> >     1. It would be good to have a mailing group started at

> > groups.google.com <http://groups.google.com>

> > possibly. [Done] :-)
> >     2. Move your code base to code.google.com

> > <http://code.google.com>[Use the mercurial SCM instead of SVN]. Would

[Sébastien Lefèvre]

Just a partial response below.

Templates are already implemented as basic built-in Python types in testerman.
When you declare a new template, it can be:
- a simple value
- a dict (equivalent to TTCN3 record / ASN.1 sequence)
- a list (TTCN3 record of / ASN.1 sequence of)
- a couple (mapping a TTCN3 union / ASN.1 choice). You just cannot use a 1-tuple or N-tuple with N > 2 to define a template (you'll get an exception when trying to send it through a port).
No need for an additional "Template" class to define them.

For instance:
template = { 'something': 'value', 'something-else': 'another value' }
is a valid template.

template = ('choiceName', { 'something': 'value', 'something-else': 'another value' })
is a valid template matching a choice structure whose the branch named "choiceName" has been selected, and whose value is a record/sequence, i.e. a dict.

(see http://www.testerman.fr/testerman/wiki/TestermanTTCN3#TypeEquivalences for additional details)

Are you sure adding explicit classes would add any readability and/or would be more pythonic ?

thanks,
-seb

[sarvi]

Ok may be I misunderstood how templates are defined based the
following example
def ParisWeekendWeatherRequest():
return {
'location': 'Paris',
'date': '15/06/2006',
'kind': 'actual'
}

Didn't get why the template ParisWeekendWeatherRequest needed to be a
function that returned a dictionary instead of just being a dictionary

Sarvi

> (seehttp://www.testerman.fr/testerman/wiki/TestermanTTCN3#TypeEquivalence...

[Sébastien Lefèvre]

Indeed, there's no need for a function, this is just a suggested best practice to defined parameterized templates.

-seb

2010/9/27 sarvi <sarv...@gmail.com>

--
Sebastien Lefevre
seb.l...@gmail.com