How to distinct errors and failures in tests

[Nick N]

Hey, guys!

I have a freestyle project. It builds an app and run tests (python, unittest). It is quite straightforward if return codes are 0 then build is successful, otherwise it is failed. Earlier I returned amount of errors and failures as a returncode to see tests results on the dashboard (passed or failed). But test results may contain failures and errors, and I want to know if there were errors on tests or they all finished, but found some bugs. Actually I need to get last build (via jenkins api) without errors, no matter if there were failures.

I've tried to look at "lastStableBuild", but it seems it sticks to "lastSuccessfulBuild". I've tried to find information about build statuses but had no success. Particularly what are criteria of each of them? Maybe it is completely wrong approach and there's a better way?

[Ullrich Hafner]

Typically you do not set the status of a build by setting the return code. Your build should never alter the return code just because the tests are failing. So I would suggest that your build always returns 0 if the build is successful (ignoring the test results). The JUnit plugin will do the build status handling for you: if there are test failures, then the JUnit plugin will set the build status to unstable. If not, the status remains successful. You can then navigate with lastSuccessful or lastStable to the last build without testfailures.

BTW: I don’t think that there is a way to differentiate between test errors and failures in Jenkins yet.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CACA70iDR0X9RTWv2051vTuO3pC5odS10kssRYLQe6%2B1A2DjGDg%40mail.gmail.com.

[nikola...@gmail.com]

Thank you for the answer. If I don't mistake, JUnit plugin just publishes the report, but unittest can't produce JUint-compatible xml. Anyway, we agree to consider builds with failed tests like failed. Tests generate some artifacts, and I want to know all artifacts exist before trying to download them.

среда, 21 апреля 2021 г. в 15:03:35 UTC+3, ullrich...@gmail.com:

[JonathanRRogers]

I suggest you start using a test runner which can produce JUnit/XUnit output. I've used both nose and pytest, both of which can run unittest tests and produce JUnit-style XML output.

[Nick N]

Thank you Jonathan!

Sure, pytest is more advanced framework and provide more features. I will definitely rewrite tests to pytest when my boss give me time. Though it doesn't provide requested feature.

By the way unless there is better solution I've implemented a workaround. My framework just writes the empty file if there were no errors, so I have to request 1 file to be sure all other exist.

You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/ku3MQxMvfl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a4a3b64e-d718-44b8-992a-55b4bae0159bn%40googlegroups.com.

[Jonathan Rogers]

On 4/23/21 5:26 PM, Nick N wrote:
> Thank you Jonathan!
>
> Sure, pytest is more advanced framework and provide more features. I will definitely rewrite tests to pytest when my boss give me time. Though it doesn't provide requested feature.

What prevents you from using pytest with your existing tests? It is designed to run unittest-style tests without any changes. Which requested feature do you need?

https://docs.pytest.org/en/latest/how-to/unittest.html

--
Jonathan Rogers

[Nick N]

The problem is I don't actually use pure unittest, I have a custom runner on top of the unittest.

The requested feature in the subject.

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/ku3MQxMvfl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/9c081724-4635-702c-b604-5decd39ff188%40gmail.com.