Existing hermetic end-to-end testing libraries for CLI applications?

[Tom Payne]

Hi,

tl;dr Are there any existing end-to-end testing libraries for CLI applications? Specifically, what I'm looking for is a library that makes it easy to test that "running this command should produce this output" without fear that a buggy application could corrupt the filesystem.

Background:

I have a few CLI applications (written in Go) that are becoming quite complex. I'd like to add some end-to-end and integration tests to ensure that I don't accidentally break existing functionality while refactoring or adding new features. The CLI applications can make arbitrary changes to the filesystem and I don't want it to be possible for a buggy application to leave any trace on the filesystem after the test terminates. I want something like:

- Easy setup of a chroot environment or Docker container or similar with my Go binary copied somewhere into $PATH.

- Easy way to specify what commands should be run.

- Easy way to write tests that check exit codes and ensure that the stdout/stderr output matches a string or regular expression.

- Automatic tidy-up of the chroot/container after the test is complete.

- Scripted in Go and integrated with Go's testing package, of course :)

I don't need controls over CPU usage or networking as my applications are not CPU bound and do not need network access, but I wouldn't complain if they existed.

A hypothetical test would look something like: https://play.golang.org/p/7L6PZIjvyy6.

I know about github.com/ory/dockertest which is fantastic for writing integration tests with third-party services but not set up for testing your own applications.

Do you know of any existing libraries that implement this sort of functionality?

Many thanks,

Tom

[Paul Jolly]

Hi Tom,

> tl;dr Are there any existing end-to-end testing libraries for CLI applications? Specifically, what I'm looking for is a library that makes it easy to test that "running this command should produce this output" without fear that a buggy application could corrupt the filesystem.

I'd suggest looking at:

* https://pkg.go.dev/github.com/rogpeppe/go-internal/testscript?tab=doc
for the CLI testing aspect
* https://pkg.go.dev/mvdan.cc/dockexec?tab=overview for the isolation piece

There are lots of examples of where testscript is being used:
https://github.com/rogpeppe/go-internal/wiki/Users-of-testscript

govim is one such an example where we also provide an example of how
to run those tests via dockexec:

https://github.com/govim/govim/wiki/govim-tests#running-tests---using-cmdgo

In our case it's not so much for isolation, rather providing dependencies.

The #tools channel on slack (or mailing list
https://groups.google.com/forum/#!forum/golang-tools) is also a good
place to chat about this more in case you had any follow-ups.

Best,


Paul

[Tom Payne]

Many thanks for this Paul - this is exactly what I was looking for.

Cheers,

Tom