Halting execution from fatal error in Listener

[Geist Eng]

I am writing a python Listener that is used during a 'dryrun' of RF suite to create in TestRail Testsuites, test cases Sections, and Cases.

If I get a failure from the TestRail api call that is done in the Listener I want to abort execution.  I did some research and not sure I can call a keyword like 'Fatal Error' as the code might not be in a RF test case context when error occurs.

How does one stop an RF run from Listener?

[adam.wyp...@gmail.com]

I think you can check what RIDE test listener is doing [1] check all 'kill' references :)

[1] https://github.com/robotframework/RIDE/blob/master/src/robotide/contrib/testrunner/testrunner.py 

[Geist Eng]

Thanks for the pointer.  But RIDE seems much more low level than my code is at using RF's Listener interface.  I have no pid to kill or sockets to send a 'kill' command down.

But if I misunderstand I am open to more info.

I do know raising an exception will not do it.  (As I suspected before I tried from what I researched; but can't hurt to try. ;-) )

[adam.wyp...@gmail.com]

Ok, a little bit description - when you add listener to running RF test you will have pid, because your code will be inside running Robot Framework test :) 

About socket is used just to communicate with listener and send some commands to execute i.e. pause test, resume test or stop it :)

Best Regards,

-Adam.

[Pekka Klärck]

If you want to stop everything, using `sys.exit()` ought to work. If you wanted to stop execution gracefully and to get log and report, you would need to send SIGINT signal to the running Robot process. I think that's what RIDE does. I guess we could also consider making it possible by using some special exception out-of-the-box. Would need someone to implement it, though.

Sent from my mobile.

--
You received this message because you are subscribed to the Google Groups "robotframework-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
To post to this group, send email to robotframe...@googlegroups.com.
Visit this group at https://groups.google.com/group/robotframework-users.
For more options, visit https://groups.google.com/d/optout.

[Geist Eng]

Abrupt or graceful abort with exit or signal is what I need.  I see no need for a special exception.

I am willing to add an error handling section to the User's Guide for the Listener section if this would be welcomed.

Thanks, Paul.

[Geist Eng]

Hmm...so it seems when running robot with DRYRUN flag the SIGINT handler in RF does not abort gracefully with a single SIGINT.  

Is this expected behavior or a bug?

As expected providing a second SIGINT abruptly aborts run.  Both from the CLI and within python Listener.

[Bryan Oakley]

What do you mean by "does not abort gracefully"? Are you saying it does not abort at all, or that it does abort in an ungraceful manner?

I think it's defined to abort gracefully on the first SIGINT, and the second SIGINT causes it to abort ungracefully. 

[Geist Eng]

Sorry, let me be clear.  It does not abort at all.

Note it does print to console on the first SIGINT that second SIGINT will abort but it continues to execute all remaining suites and tests.

[Bryan Oakley]

Yes, I believe that is what it's designed to do. Sending SIGINT is the same as pressing control-c on the command line. The behavior is described here: http://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#stopping-test-execution-gracefully

--

[Geist Eng]

Yea, the behavior does not follow per the User's Guide.  With DRYRUN the signal is not handled until execution crosses a directory boundary.  Due to difference in behavior I suspect this is a bug when DRYRUN is enabled.

On a run without the DRYRUN option sending one Ctrl-C does this:

==============================================================================
Smoke                                                                         
==============================================================================
Smoke.System Test                                                             
==============================================================================
Smoke.System Test.Smoke Api Conf Verify :: Full test of GeistAPIVariables.py  
==============================================================================
Test API Valid Single Value                                           | PASS |
------------------------------------------------------------------------------
Test Variable Number Of Args By Primary Name                          Second signal will force exit.
| FAIL |
Execution terminated by signal
------------------------------------------------------------------------------
Test Variable Number Of Args By Secondary Name                        | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test With Expected Value                                              | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Add Supported Camera Host Only                             | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Set URL On Supported Camera                                | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Set Username On Supported Camera                           | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Add Second Camera                                          | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Delete Of Camera                                                 | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Add Email Target                                           | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Set Email Target After Add                                 | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Delete Email Target                                        | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Add Two Email Targets                                      | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Test Valid Set Email Target After Two Added                           | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------

But with the DRYRUN option sending one Ctrl-C does this:

==============================================================================
Smoke                                                                         
==============================================================================
Smoke.System Test                                                             
==============================================================================
Smoke.System Test.Smoke Api Conf Verify :: Full test of GeistAPIVariables.py  
==============================================================================
Test API Valid Single Value                                           | PASS |
------------------------------------------------------------------------------
Test Variable Number Of Args By Primary Name                          | PASS |
------------------------------------------------------------------------------
Test Variable Number Of Args By Secondary Name                        | PASS |
------------------------------------------------------------------------------
Test With Expected Value                                              | PASS |
------------------------------------------------------------------------------
Test Valid Add Supported Camera Host Only                             | PASS |
------------------------------------------------------------------------------
Test Valid Set URL On Supported Camera                                | PASS |
------------------------------------------------------------------------------
Test Valid Set Host On Supported Camera                               | PASS |
------------------------------------------------------------------------------
Test Valid Set Username On Supported Camera                           | PASS |
------------------------------------------------------------------------------
Test Valid Add Second Camera                                          | PASS |
------------------------------------------------------------------------------
Test Delete Of Camera                                                 Second signal will force exit.
| PASS |
------------------------------------------------------------------------------
Test Valid Add Email Target                                           | PASS |
------------------------------------------------------------------------------
Test Valid Set Email Target After Add                                 | PASS |
------------------------------------------------------------------------------
Test Valid Delete Email Target                                        | PASS |
------------------------------------------------------------------------------
Test Valid Add Two Email Targets                                      | PASS |
------------------------------------------------------------------------------
Test Valid Set Email Target After Two Added                           | PASS |
------------------------------------------------------------------------------

But on the DRYRUN test run it gets more complicated.  It appears the signal is eventually handled.  In this run once execution comes back up from a lower suite I start to see abort of test cases:

==============================================================================
Smoke.System Test.smoke Utility lib                                           
==============================================================================
Test Del Test Users                                                   | PASS |
------------------------------------------------------------------------------
Test Get IPv4 GW                                                      | PASS |
------------------------------------------------------------------------------
Test API Failure Stats Error 1                                        | PASS |
------------------------------------------------------------------------------
Test API Failure Stats Error 2                                        | PASS |
------------------------------------------------------------------------------
Test App Restart Stats Error 1                                        | PASS |
------------------------------------------------------------------------------
Test App Restart Stats Error 2                                        | PASS |
------------------------------------------------------------------------------
Smoke.System Test.smoke Utility lib                                   | PASS |
6 critical tests, 6 passed, 0 failed
6 tests total, 6 passed, 0 failed
==============================================================================
Smoke.System Test                                                     | FAIL |
135 critical tests, 134 passed, 1 failed
135 tests total, 134 passed, 1 failed
==============================================================================
Smoke.Testrigs                                                                
==============================================================================
Smoke.Testrigs.smoke SensorDatabase                                           
==============================================================================
DB Init                                                               | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------
Add                                                                   | FAIL |
Test execution stopped due to a fatal error.
------------------------------------------------------------------------------

I did not see this when first posting as that run was not crossing directory boundaries.  Here is a snippet of the dir tree layout to give idea of when signal is finally handled.

smoke
├── system_test
│   ├── __init__.robot
│   ├── smoke_api_conf_verify.robot
│   ├── smoke_api_set_data.robot
│   ├── smoke_test_system_suite_setup.robot
│   └── smoke_Utility_lib.robot
├── testrigs
│   ├── __init__.robot
│   ├── smoke_SensorDatabase.robot
│   ├── smoke_SensorSwitch_port_ctrl.robot
│   ├── smoke_SensorSwitch.robot
│   └── smoke_TestRig.robot
└── toucan_gui

[Geist Eng]

Well it seems there might be a need for a special exception.

It seems that sending just one SIGINIT  is not handled fast enough.

Example from making an API call to TestRail:

try:

    resp = self.testrail.add_milestone(self.project_id, self.milestone) 

except TestRailAPIError as e:

    self.log('LISTENER FATAL ERROR: add milestone error: {}: {}\n'.format(e.code, e.error), console=True)

    self.signal_quit()

self.milestone_id = resp['id']

signal_quit() will send the SIGINT and then returns.  At this point the code attempts to access the API response dict and throws an exception since it is not a valid python dict.

For now I am just abruptly aborting with sending two SIGINTs.

Should I open an enhancement bug for this exception or is there something I am missing?

[Pekka Klärck]

Hi,

This looks like a valid enhancement for me so please submit an issue.
Due to the very limited development resources I cannot make any
promises when, if ever, this is going to be implemented, though. If
you are willing to implement it yourself, I can review pull requests.
Alternatively sponsoring the work either directly or via the Robot
Framework Foundation is possible.

Cheers,
.peke

> --
> You received this message because you are subscribed to the Google Groups "robotframework-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to robotframework-u...@googlegroups.com.
> To post to this group, send email to robotframe...@googlegroups.com.
> Visit this group at https://groups.google.com/group/robotframework-users.
> For more options, visit https://groups.google.com/d/optout.

--
Agile Tester/Developer/Consultant :: http://eliga.fi
Lead Developer of Robot Framework :: http://robotframework.org

[Geist Eng]

I have opened 2940 Abort execution gracefully from Listener with special exception

I will try my hand at working this issue.  

Thx.