Robot Framework Listener for TestRail available on GitHub

[Geist Eng]

Folks,

I have completed implementation of robot LIsteners to integrate with TestRail.  One listener is used to do a dryrun across robot test data to add Testrail Cases to Testrail Test Suites and the other listener runs during normal test runs to add Milestones, Plans, Runs, Tests, and Results to Testrail as the run executes.

I have placed these, along with the Testrail API lib used on a public GitHub repo.  Feel free to check it out and ask questions or provide feedback.

Robotframework-Testrail-Listener

Soon I will be working prerunmodifier class to create Plans/Runs in Testrail and then run them with robot.

Cheers, Paul

[Tatu Aalto]

Ugh

Cool and it's nice that you did published it as open source project. Would you think that project is mature enough to be added to the http://robotframework.org/#tools ?

-Tatu
Send from my mobile

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Geist Eng]

The code is mature enough.  I have been testing and refactoring for a couple of weeks in my test rig.

I need to add to the documentation on how to use and will do that over the next few days.

I would be honored to be added to RF tools section and give back to the community that has supported my efforts.

Thx.

[Tatu Aalto]

Well, it's for you to make a PR in: https://github.com/robotframework/robotframework.github.com

-Tatu
Send from my mobile

--

[phaneendra kumar Gondi]

Hi Paul,

  

I'm trying to integrate robot frame work to test rail. 

my aim is to update the robot frame work test result to Test rail so among the three libraries (TestRailCasesListener.py, TestRailListener.py, TestRailRunListener.py) 

which one I should be using for my case. I'm guessing 3rd one but please confirm.

Thank you,

Phaneendra 

[phaneendra kumar Gondi]

Hi Eng,

could you please provide steps to how to execute robotframe work file with robot listener files?

Thank you

Phaneendra

On Thursday, 23 August 2018 04:52:35 UTC+1, Geist Eng wrote:

[Bryan Oakley]

Everything you need to know is in the user guide:

http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#taking-listeners-into-use

[Geist Eng]

Hello Phaneendra,

Yes I will.  I have been meaning to do this since I posted the files but have failed to do so.

I will add detail on github project but for now let me give a synopsis:

• TestRailListener.py is the base class for the other two Listeners that do the work
  
• TestRailCasesListener.py is used to create the Cases in Testrail (NOT the actual Testcases in a plan or run)
  
• TestRailRunListener.py is used to update the status of each test in Testrail as it is run by RF
  
• It also creates the Testcases in the actual Testrail Run or Plan at the beginning of each RF suite

To run:

1. Update TestRailListener.py with the Testrail user login credentials and the Testrail project ID to run in.
1. This is at the very top of the file
2. Run TestRailCasesListener.py with robot with --dryrun flag on the suites you want in Testrail
1. The config is where the most work has to be done to run in your environment.
2. Then running  with this Listener will cause robot to traverse your RF tests and make API calls to TestRail to create the Cases
3. Config and run TestRailRunListener.py  on each actual run of robot.
1. The config is where the most work has to be done to run in your environment.
1. Review the function init_site_specific_info()
1. This method decides how test results are entered into test rail.
2. The current code will end up setting:
1. TR Milestone name it will create or add to
2. TR Plan name it will create or add to
3. TR Run it will create in the TR Plan above for the current run
2. Then every run of robot with this Listener will create a new TR Run in a specific Plan, add Tests to the run, then update the Tests as each is run

Examples:

Creating the test cases first on an RF suite named 'system'

robot --listener TestRailCasesListener --dryrun system

Running the test cases to get results:

robot --listener TestRailRunListener system

Make sure you create a temp Testrail project to test this in.  One you can delete later.

I will be happy to answer more questions as you try the code.

[phaneendra kumar Gondi]

Hi Paul,

getting below error 

[ ERROR ] Taking listener 'tu-tests/regressionpack/resources/TestRailLCasesListener' into use failed: Importing listener 'tu-tests/regressionpack/resources/TestRailLCasesListener' failed: ImportError: Import by filename is not supported.

my command is

 robot --listener tu-tests/regressionpack/resources/TestRailLCasesListener --dryrun tu-tests/regressionpack/tests/tu_features

please help on this 

thank you

[phaneendra kumar Gondi]

Hi Paul,

this time I tried with below command, no error showed regarding TestRailCasesListener file but all test cases failed (hope this is expected.

but did not created test cases in test rail 

 robot --listener TestRailCasesListener --dryrun -v BROWSERS:gc -v URL:https://origin-customer-tu-test.tuinfradev.com/ -v platform:darwin --loglevel DEBUG -d results tu-tests/regressionpack/tests/tu_features

Tu Features                                                           | FAIL |

71 critical tests, 0 passed, 71 failed

71 tests total, 0 passed, 71 failed

Thank you,

Phaneendra

[phaneendra kumar Gondi]

Hi Paul please ignore my last 2 mails 

command used

robot --listener tu-tests/regressionpack/resources/TestRailCasesListener.py --dryrun -v BROWSERS:gc -v platform:darwin --loglevel DEBUG -d results tu-tests/regressionpack/tests/tu_features

gettingerror  

20181024 17:22:29.811 | ERROR | Taking listener 'tu-tests/regressionpack/resources/TestRailCasesListener.py' into use failed: Importing listener '/Users/phaneendra.gondi/tu-app/tu-tests/regressionpack/resources/TestRailCasesListener.py' failed: Creating instance failed: ValueError: No JSON object could be decoded

Traceback (most recent call last):

  File "/Users/phaneendra.gondi/tu-app/tu-tests/regressionpack/resources/TestRailCasesListener.py", line 13, in __init__

    super(TestRailCasesListener, self).__init__()

  File "/Users/phaneendra.gondi/tu-app/tu-tests/regressionpack/resources/TestRailListener.py", line 55, in __init__

    self.auto_type = self.testrail.get_automated_test_case_type()

  File "/Users/phaneendra.gondi/tu-app/tu-tests/regressionpack/resources/TestRailAPIClient.py", line 274, in get_automated_test_case_type

    response = self.send_get(uri)

  File "/Users/phaneendra.gondi/tu-app/tu-tests/regressionpack/resources/TestRailAPIClient.py", line 35, in send_get

    return self.__send_request('GET', uri, None)

  File "/Users/phaneendra.gondi/tu-app/tu-tests/regressionpack/resources/TestRailAPIClient.py", line 70, in __send_request

    result = json.loads(response)

  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 339, in loads

    return _default_decoder.decode(s)

  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 364, in decode

    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "/usr/local/Cellar/python@2/2.7.15_1/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 382, in raw_decode

    raise ValueError("No JSON object could be decoded")

Please help on this 

Thank you,

Phaneendra 

On Wednesday, 24 October 2018 15:36:28 UTC+1, Geist Eng wrote:

[Geist Eng]

At that point in the code you are still in the API code from TestRail itself.  I did not write it.  The error looks like what was returned from Testrail was not JSON.

Have you enabled the API in TestRail?  Do you have the correct TestRail user login credentials in TestRailListener.py?

Basically the failure is the Listener cannot log into TestRail.

[Geist Eng]

Here is the TestRail page on testing you have API correct:

http://docs.gurock.com/testrail-api2/accessing#example_request

And from Intro TR page:

The API is part of TestRail and can be enabled in TestRail's administration area under Administration > Site Settings > API.

[Geist Eng]

Worked with Phaneendra off line.  Initial issue was he was using a hosted Testrail so HTTPS was used and my code by default uses http on a self-hosted TR install.

Have updated files on GitHub so config now includes what web protocol is being used.  Also moved config out of main py files so users can update their installs and not overwrite server config.

Still need to do the same for some other data that is used to name TestRail Milestones, Plans, and Runs 

[Geist Eng]

Moved all site specific code to external file from Listener class.  Set up so updates from Github to a user's local repo will not override site specific changes.

Flushed out docs.

Attempted to make a pull request to add to RF tools section but having issues addressed in another thread.

Phaneendra, you should update to this new version as any issues we work on in the future will have to have these changes.  Review the README.md on GitHub for details.  But basically all the custom site stuff you added is now in a file by itself.

[Saikat Debnath]

Hi Geist,

I am new in Robot Framework. I have taken clone of your github project. I have like 1000 Automated Robot test cases. Each test cases have one unique test cases id ( Example- C45876). Test cases are in Testrail. Now what i want to do is after running the Test suit it should go to testrail and update the test case status(Pass/Fail) based on the Test case Status. I am having difficulty to understand the codes for the below. Where in my Robot test cases i have to include these listener?

• TestRailListener.py is
  
• TestRailCasesListener.py
  
• TestRailRunListener.py

[Geist Eng]

Sorry I am late to answer you.  I never saw the email for this.  My mistake.

My listener does not use the IDs from Testrail.  It used the names of the Cases as they are in the containing Sections in TestRail BECAUSE they match the layout of suites and tests in robot.

Again you need to understand that how the Sections and Cases are in Testrail  are created is how robot finds them in a run.

This way there is no need to have special IDs between robot and Testrail.  And as your robot test cases add/change/move those changes are picked up in Testrail.

The magic is the TestRailCasesListener.py  This listener is used to CREATE the Sections and Cases in Testrail.  It does this by calling the TestRail API as it runs each test in robot.  That is why the docs say to use the --dryrun option

robot --listener TestRailCasesListener --dryrun <path to your suites>

So every time you need to update Testrail you run the command above.  It will know which robot tests are new or have moved and create new Cases.  Same with Sections, it creates those as it goes.

Now once Testrail is setup just run the TestRailRunListener.py and it will update testrail.  Note it does this by creating or updating TestRail entities: Milestone, Test Plan, Test Run. Tests.

robot --listener TestRailRunListener  <path to your suites>

Note on actual run process:  at the start of each suite the listener will add the testrail Cases in this robot suite to the Testrail Run.  Then as each robot test is done the listener will update the Testrail Test with results.

In this way TestRails represents the actual time and progress of the runs.  If you create a listener that updates in batches then the execution time in Testrail will all be the same for that batch.  This makes the graphs in Testrail incorrect.

Note Testrail does not provide a way to set execution time.  It defaults to when the result was added.

[T Rohit kumar sai]

Hi Geist,

I am working on exact scenario mentioned by you. You can use testrail api's to do so. There is a helper class available for the same. Now you need to create method and that after every testcase teardown. You need to pass tc id and tc status to the method and will accordingly update. Now all of these works on robot's environment variables.

--

You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/44aa0d01-262f-43de-9b37-faeb33544d42%40googlegroups.com.

[Geist Eng]

I am a bit unclear.  Are you saying you are updating Testrail in the robot testcase teardown?

And what is helper class you speak of?

On Sunday, August 25, 2019 at 1:30:12 AM UTC-5, T Rohit kumar sai wrote:

Hi Geist,

[T Rohit kumar sai]

Hi Geist,

My bad i was unclear, Apologies for that!!

I meant that to update every test case in testrail post testcase completion you can hit testrail api's with runid and testcase id. To do that you can get the script/bindings specific to a language. For testrail api documentation please refer here .

Now what you need to do is download the required language binding from here and call its different methods available from a separate helper method according to youre requirement.

Now this helper method needs to be called after every testcase teardown in robot framework with few mandatory parameters like  testcase_id, test_status, run_id etc.

--

You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/robotframework-users/2b2eefe7-4b0f-4d24-b292-c629cef58089%40googlegroups.com.

[Geist Eng]

Hi T Rohit,

Thx for reply.  Now I understand your setup.

For clarity  my Listener also uses the Testrail API bindings to update results, etc.  The python module, TestRailAPIClient.py, in my implementation flushes out this class (similar to many out there.)

The difference is your calling the API in robot test teardown.  Which is fine.  But the down side could be EVERY run of robot ends updating TestRail.  

I only want to update Testrail during specific runs, such as a CI/CD automation run.  I would not want to touch Testrail during dev runs that have nothing to do with Testrail.

Of course in your case you can have a global flag to enable/disable the Testrail API calls in robot test teardown set by your startup script.

But it can be argued that the Listener interface is a more elegant solution as it separates test execution logic from the actual test case logic.

On Sunday, August 25, 2019 at 10:54:53 AM UTC-5, T Rohit kumar sai wrote:

Hi Geist,

[Brand Testing]

Hi Geist, 

Hopefully you're still active on this thread/library.

So I am getting following error, and not exactly sure where to look at to fix it. 

My Testrail api is enabled, created project inside testrail, and using Python 3.9 and RF 4.0.

Thanks for your time!

[Geist Eng]

The code on GitHub was written for python 2.7, not 3.x so I suspect that is the source of the issue.

The company I worked for at the time has been purchased and the new owners do not as yet support putting code out for open source so I can not update github.

But looking at latest version of TestRailAPIClient.py I see I converted this to bytes. In fact this part of the code is from Testrail itself so I can post what it looks like now. 

Notice what is passed to b64encode is now a bytes object :

    def __send_request(self, method, uri, data):
        url = self.__url + uri

        auth = str(
            base64.b64encode(
                bytes('%s:%s' % (self.user, self.password), 'utf-8')
            ),
            'ascii'
        ).strip()
        headers = {'Authorization': 'Basic ' + auth}

[Brand Testing]

Thanks for the quick response!

Looks as issue is solved, though another one popped out:

Might be due to importing this urlib2 library, since without it, it wouldn't work with python 3x:

import urllib.request as urllib2

import   json, base64

class TestRailAPIClient:

[Geist Eng]

Could be as I no longer import urllib2.request.  This part of the code is from Testrail itself.  They have it on Github too and give both the 2.x and 3.x version.  Here is the link to the 3.x version:

https://github.com/gurock/testrail-api/blob/master/python/3.x/testrail.py