Testing external programs from LCC

[Alexandre]

Hi all,

I'm using LCC to test a CLI program that has quite a lot of options.

I'm wondering if i'm doing it right (i'm kinda python noob), and if LCC could maybe add an API entry to encapsulate it better.

Here's how i do it:

return_code = subprocess.run(["./cli-app", "arg1", "arg2", ....], capture_output=True)

lcc.log_info("Got STDOUT: %s" % return_code.stdout)

lcc.log_info("Got STDERR: %s" % return_code.stderr)

lcc.check_that("Help return code", return_code.returncode, equal_to(0))

Other than that, LCC really fit my needs on other topics, API calls and GitLab integration (JUnit reporting). Great work!

++

Alex

[Nicolas Delon]

Hello Alexandre and thanks for your kind feedback.

About the command line : yes, you doing it right but I'm not sure about adding something "domain" specific (CLI handling in that case) to a generic-purpose testing framework.

That's why lemoncheesecake-requests is a dedicated Python package for instance.

One may imagine in the future to implement a dedicated package for CLI testing but it would need to provide more than a helper function.

Anyway, I keep your (good) request feature in mind.

About helper function and encapsulation, I wrote for you a very basic helper (based on what you provided) as an example of what a generalization could look like:  https://gist.github.com/ndelon/bd73faa72ecd871da6afa79194a48e12

And here is the result of the test sample:

I hope that it answers your question.

Regards,

Nicolas.

[Alexandre]

Hi Nicolas,

Thanks very much for your reply! I integrated your code in the test suite, and it's working great! I just modified it to add "env" as parameter.

To answer your question, i don't think it has its place in LCC itself, but it could be interesting to make a dedicated Python package for that, especially that the behaviour depends on the version of Python (subprocess.run is only Python > 3.5 IIRC).

Also i was wondering how to add details to the JUnit failure report. By default it contains "failed check in step '<failing_test>'", but how can i add more meaningful details there?

++

Alex

[Nicolas Delon]

Hello Alexandre,

You're welcome.

You're right about JUnit reports, they are not providing much details. I'll improve that soon in a future release and keep you updated.

Regards,

Nicolas.

[Alexandre]

Hi Nicolas,

Just some feedback in case someone finds it useful. I ended up with the following helpers:

https://gist.github.com/aloopkin/a50a2c45735b5819c7d4612225b0a5cf

The write_to_file method is there because in some cases it's just more simple to generate a bash file and execute it, and also in some cases it's the only way to pass parameters to programs. Anyway at the end of the day i was able to do everything that i wanted and integrate most of my test cases. Thanks very much for LCC and for your help!

++
Alex

[Nicolas Delon]

Hello Alex,

About returning the stdout content, you could simply update the check_command function and add:

"""

return normalized_stdout, normalized_stderr

"""

at the end of the function, it would avoid creating a new function for this.

About "write_to_file", please also note that if you want to log something bulkier, you can log it as an attachment: see http://docs.lemoncheesecake.io/en/latest/logging.html#attachments

About your previous feedback on junit, I've just released a 1.10.2 of lemoncheesecake that provides more error details (https://github.com/lemoncheesecake/lemoncheesecake/blob/master/CHANGELOG.md#1102-2021-10-28).

I'm glad that the project helped you with your work !

Regards,

Nicolas.