Re: GTF: A generic Test Automation Framework based on golang

[egon]

I can't really understand what real-world problem this solves that go test & goconvey don't.

+ Egon

On Wednesday, 3 December 2014 14:12:04 UTC+2, Zhang Qin wrote:

Hi All,

GTF: https://github.com/ZhangQin3/GTF

GTF is a generic test automation framework based on golang, it is named as Golang Test Framework(GTF). Now, GTF is in its very early stage, but it has provide most necessary features that a mature testing framework shall provide, it can be used in unit test, integration test, system test and acceptance test with only a little simple adaption.

With GTF, one test script is a golang package in a single .go file, all the test scripts are put in the gtf/scripts directory. Related test scripts can get together and are put into a test suite, the test suite is also an .go file and a golang package located in the gtf/testsuites directory.

GTF generates one executable file(.exe) for each test suite, the executable file is put in the $gopath/bin/ directory. To generate an executable file for a test suite, you should build and run ignition.go package in gtf/drivers/ignition directory, this will generate a .exe file and puts it in $gopath/bin/.

Run the executable file( execute.exe for now) in $gopath/bin/, all the tests related with the test suite will be executed and logs for each test scripts will be generated and put in $gopath/bin/.

GTF Provides: 1) Test case level and test script level and test suite level setup and teardown. 2) HTML log output for each test script and the log file gives logs for each test case. 3) A simple way for creating customized test libraries(such as, Selenium, Telnet, SSH) with golang std libraries and gtf/log.

TBD: data driven/keyword driven support. log page internal jump.

Only support windows now.

Enjoy it.

BRs,

Zhang Qin

[egon]

On Thursday, 4 December 2014 03:53:19 UTC+2, Zhang Qin wrote:

On Wednesday, December 3, 2014 9:30:30 PM UTC+8, egon wrote:

I can't really understand what real-world problem this solves that go test & goconvey don't.

+ Egon

I did not compare GTF to others. In fact, I did not know goconvey before, will study it later. I just wrote GTF according to my experience and some requirements in my area, and study golang with the project. ^_^

OK. Basically the code looks way too complicated for Go - it looks like someone coming from Java background trying to write Go. Basically - your code is very in pieces and putting together the big picture, what it does, is very hard.

I'm not able to understand the problems it solves, which means I don't know how to restructure the code better.

I suggest inspecting other Go code, such as 

http://golang.org/pkg/testing/

https://github.com/golang

https://github.com/campoy?tab=repositories

https://github.com/bradfitz?tab=repositories

To figure out better ways of writing.

+ Egon

 

[egon]

On Thursday, 4 December 2014 12:12:29 UTC+2, Zhang Qin wrote:

On Thursday, December 4, 2014 9:53:19 AM UTC+8, Zhang Qin wrote:

On Wednesday, December 3, 2014 9:30:30 PM UTC+8, egon wrote:

I can't really understand what real-world problem this solves that go test & goconvey don't.

+ Egon

I did not compare GTF to others. In fact, I did not know goconvey before, will study it later. I just wrote GTF according to my experience and some requirements in my area, and study golang with the project.

^_^

- ZhangQin

After a quick scanning to goconvey, I found that goconvey mainly focuses on Golang unit test, it just tests Golang code, is that right?

As mentioned above, GTF is a “generic” "test automation framework" based on Golang, it is writen in Golang, but not only for Golang. It focuses on "generic", just uses Golang as a "script language", and puts more focus on function testing. It can be used to test functions of a router, a cable modem, Wifi, a web APP, and so on.

Compare with goconvey, GTF has:
1. generic testing
   GTF is not only for unit test, it focuses on function test as what mentioned above.
2. testcase definition
   With testcase definition, the applicability, requirements, priority and feature can be defined,

I believe the testcase definition could be significantly improved via an DSL. Currently the program expects Go compiler to be present, but if it's a standalone tool it probably should work without Go compiler.

Try to first figure out the best way of writing testcases.. then convert it to a clear DSL, hook it up to Go (so it would be possible to add plugins), and then use it as a standalone tool.

Of course bootstrapping with using Go, isn't a bad idea.

then testcases can be filterd and run by these attributes, you can just run parts of script , and will not run a testcase if its requirements are not setisfied. Test definition also gives the testcase a brief description.
3. setup and teardown for various levels
   Test case level, test script level and test suite level setup and teardown.
4. step-lized tests
   There are clear steps in the testcase, every step has its function and description, this is import for test results analysis.

The name Golang Test Framework(GTF) may has some misleading, but I think it's OK.

Hmm, yes that might be one reason it really confused me - I though it was a Go specific package/tool.

Maybe something like TestCoordinator/TestOrchestrator? :)

I.e. I would try to remove "Golang" from it's name to make it obvious that it's not a Go package, but instead a separate application - probably some example how the program is used would help here as well.

--ZhangQin

[egon]

So, if you did not know what it was and what difficulty it faced, and you did read the code carefully, you should not give you comment arbitrarily by just a quick glance and you even did not read my post carefully.

If a glance doesn't reveal the intent of the program then that already suggests that there is a problem. Let's take an arbitrary example:

if x && z {

z = w

if t {

print(r)

}

}

At first look it's obvious that it could be clearer, even if you don't understand what it does, why it was written that way or how the rest of the code is structured. The same way I look at your current code... it's not obvious what it does and how it does it - which means it can be improved.

I'm not saying the features it provides are unnecessary, rather that there should be a better way to write it. As an example of a recent similar situation:

https://groups.google.com/d/msg/golang-nuts/grLcPuPyAyU/lCwIvRG2_-YJ

You can compare:

https://github.com/johnwilson/bytengine/tree/4ff8b445b4a8db5594b792287c1863ff5f0751db

https://github.com/johnwilson/bytengine

The last version has a single proper idea - an Engine, that is supported by different parts of the engine (bytestore, statestore, authentication, filesystem, parser). With different concrete part implementations in their respective folders. The first one was pieces of the central idea scattered around, the later version is a core idea, supported by other packages.

Uncle Bob goes over the similar problems in this video https://www.youtube.com/watch?v=HhNIttd87xs

Basically, is there some sort of real-world example of a script you use/have used? Maybe seeing it being used would clarify things.

Just to clarify how I see the GTF:

Ideally when using some sort of generic testing framework, the first thing I would like to write in my console is:

gtf monitor

-> this would start monitoring my testscripts folder for modifications/changes etc.

This means it should also rerun the test cases when I've made in any of the dependencies. When writing the testscript, I would like to write something like:

>>>

var (

  $exe = "bin/something.exe"

)

// this is here, so the tool can figure out when to rerun the tests

func Dependencies() []string {

  return []string{ $exe }

}

// Display help when there are no arguments

func CaseT123(c gtf.Case){

  output := cmd.Run($exe)

  if !strings.Contains($output, "usage") { c.Fail("didn't contain usage!") }

}

<<<

Of course, I can understand that this isn't exactly your use case... So, I'm trying to understand the full set of use cases for the GTF; that way I can give any meaningful comments that actually take into account all the necessary complexities.

+ Egon