expect_failure for unittest

[Marcel Kirsche]

Hello,

I just experimented with the Bazel testing capabilities and wondered if it's possible to also assert a fail() in a regular unittest (not analysistest)? I Couldn't find anything on that.

Thanks,

Marcel

[Alex Humesky]

Are you looking for this?

https://github.com/bazelbuild/bazel-skylib/blob/main/docs/unittest_doc.md#unittestfail

(https://github.com/bazelbuild/bazel-skylib/blob/ecc11f9a4c06400d4797e7132e71fe8702490cf0/lib/unittest.bzl#L581)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/dc2d4470-e77b-45d2-9c6b-6115e5dd989an%40googlegroups.com.

[Alex Humesky]

+bazel-discuss 

forgot to add bazel-discuss for posterity

On Thu, Feb 3, 2022 at 9:26 PM Alex Humesky <ahum...@google.com> wrote:

Gotcha, I see. Unless I missed something, I think you'd need to write an intermediate rule that just calls the function, and write an analysis test around that. Something like this:

BUILD:

load(":myhelper_tests.bzl", "myhelper_tests")

myhelper_tests()

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "bazel_skylib",
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz",
        "https://github.com/bazelbuild/bazel-skylib/releases/download/1.1.1/bazel-skylib-1.1.1.tar.gz",
    ],
    sha256 = "c6966ec828da198c5d9adbaa94c05e3a1c7f21bd012a0b29ba8ddbccb2c93b0d",
)
load("@bazel_skylib//:workspace.bzl", "bazel_skylib_workspace")
bazel_skylib_workspace()

myhelper.bzl:

def myhelper(value):
  if value == 1:
    fail("1 is not allowed")
  if value == 2:
    fail("2 is not allowed")
  return "abc"

myhelper_tests.bzl:

load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest")
load(":myhelper.bzl", "myhelper")

def _myhelper_failure_test_impl(ctx):
  env = analysistest.begin(ctx)
  asserts.expect_failure(env, ctx.attr.expected_failure)
  return analysistest.end(env)

myhelper_failure_test = analysistest.make(
  _myhelper_failure_test_impl,
  expect_failure = True,
  attrs = {
    "expected_failure": attr.string(mandatory = True),
  },
)

def _myhelper_caller_impl(ctx):
  myhelper(ctx.attr.myhelper_param)
  return []

_myhelper_caller = rule(
  implementation = _myhelper_caller_impl,
  attrs = {
    "myhelper_param": attr.int(mandatory = True),
  }
)

def _make_myhelper_test(name, param, expected_failure):
  caller_target = "_%s_myhelper_caller" % name
  _myhelper_caller(
      name = caller_target,
      myhelper_param = param,
      tags = ["manual"],
  )
  myhelper_failure_test(
    name = "_%s_myhelper_failure_test" % name,
    target_under_test = caller_target,
    expected_failure = expected_failure,
  )

def myhelper_tests():
  _make_myhelper_test("1", 1, "1 is not allowed")
  _make_myhelper_test("2", 2, "2 is not allowed")

That's a lot of boilerplate. This helps reduce it a bit:

failure_unittest.bzl:

load("@bazel_skylib//lib:unittest.bzl", "asserts", "analysistest")

def _failure_test_impl(ctx):
  env = analysistest.begin(ctx)
  asserts.expect_failure(env, ctx.attr.expected_failure)
  return analysistest.end(env)

_failure_test = analysistest.make(
  _failure_test_impl,
  expect_failure = True,
  attrs = {
    "expected_failure": attr.string(mandatory = True),
  },
)

def make_caller_rule(func, *args, **kwargs):
  def _caller_impl(ctx):
    func(*args, **kwargs)
    return []
  return rule(
    implementation = _caller_impl,
  )

def make_failure_unittest(name, caller_rule, expected_failure):
  caller_target = "_%s_caller" % name
  caller_rule(
      name = caller_target,
      tags = ["manual"],
  )
  _failure_test(
    name = "_%s_failure_test" % name,
    target_under_test = caller_target,
    expected_failure = expected_failure,
  )

myhelper_tests.bzl:

load("failure_unittest.bzl", "make_failure_unittest", "make_caller_rule")
load(":myhelper.bzl", "myhelper")

myhelper_caller_1 = make_caller_rule(myhelper, 1)
myhelper_caller_2 = make_caller_rule(myhelper, 2)

def myhelper_tests():
  make_failure_unittest("1", myhelper_caller_1, "1 is not allowed")
  make_failure_unittest("2", myhelper_caller_2, "2 is not allowed")

Having to instantiate the rule classes for each case ("myhelper_caller_1 = make_caller_rule(myhelper, 1)") separately from creating the test targets ("make_failure_unittest(...)") is annoying, but it seems to be a limitation of how rule classes work. I tried this:

def _make_caller_rule(func, *args, **kwargs):
  def _caller_impl(ctx):
    func(*args, **kwargs)
    return []
  return rule(
    implementation = _caller_impl,
  )

def make_failure_unittest(name, expected_failure, func, *args, **kwargs):
  caller_rule = _make_caller_rule(func, *args, **kwargs)
  caller_target = "_%s_caller" % name
  caller_rule(
      name = caller_target,
      tags = ["manual"],
  )
  _failure_test(
    name = "_%s_failure_test" % name,
    target_under_test = caller_target,
    expected_failure = expected_failure,
  )
 
BUILD:

def myhelper_tests():
  make_failure_unittest("1", "1 is not allowed", myhelper, 1)
  make_failure_unittest("2", "2 is not allowed", myhelper, 2)

But that failed with:

"Error in unexported rule: Invalid rule class hasn't been exported by a bzl file"

I think there's some magic that happens underneath with "name = rule(...)", where "name" becomes part of that rule.

Feel free to file a feature request to make this more convenient.

On Wed, Feb 2, 2022 at 1:57 AM Marcel Kirsche <mcl.k...@gmail.com> wrote:

I don’t think so. This is going to make the test fail right away as I understand.

I want to test that a Starlark function fails when I give it the wrong input.

On 2. Feb 2022, at 01:04, Alex Humesky <ahum...@google.com> wrote: