Testing many modules with the same interface?

[James Edington Administrator]

I'm maintaining a codebase (CFFI bindings for a 3rd-party C library), and I'm interested in adding tests to my bindings.

My code is organized into 2 modules ("pqc.kem" and "pqc.sign"), each of which has around fifteen sub-modules in it. Within each module, every sub-module has exactly the same interface, the same set of functions, which I'll need to run "the same" test suite against.

This is definitely past the quantity threshold where "DRY" becomes good, but I'm wondering what the best practices are for this when it comes to test suites. I had a bit of trouble finding anything in either the Hypothesis or pytest documentation about "interfaces".

Thanks,

--
James E. A.

[Florian Bruhin]

Hey,

> I'm maintaining a codebase (CFFI bindings for a 3rd-party C library), and
> I'm interested in adding tests to my bindings.
>
> My code is organized into 2 modules ("pqc.kem" and "pqc.sign"), each of

> which has *around fifteen* sub-modules in it. Within each module, every

> sub-module has exactly the same interface, the same set of functions, which
> I'll need to run "the same" test suite against.
>
> This is definitely past the quantity threshold where "DRY" becomes good,
> but I'm wondering what the best practices are for this when it comes to
> test suites. I had a bit of trouble finding anything in either the
> Hypothesis or pytest documentation about "interfaces".

I don't think there is much on the Hypothesis side of things for this.

But there certainly is for pytest: A parametrized fixture is what I'd
use for this, which then provides your tests with the respective Python
module object:

https://docs.pytest.org/en/8.0.x/explanation/fixtures.html
https://docs.pytest.org/en/8.0.x/how-to/fixtures.html#parametrizing-fixtures

A quick untested draft to get you started:

@pytest.fixture(params=["kem", "sign"])
def mod(request):
return importlib.import_module(f"pqc.{request.param}")

def test_one(mod):
...

def test_two(mod):
...

Which should then result in four tests being run.

Florian

[James Edington Administrator]

Thanks for the suggestion. Fixtures seem to work great for this:

@pytest.fixture(scope='package', params=['hqc_128', 'hqc_192', 'hqc_256', 'kyber512', 'kyber768', 'kyber1024', 'mceliece348864', 'mceliece460896', 'mceliece6688128', 'mceliece6960119', 'mceliece8192128'])
def kemalg(request):
    module_name = f'pqc.kem.{request.param}'
    return importlib.import_module(module_name)

def test_roundtrip_encap(kemalg):
    pk, sk = kemalg.keypair()
    ss, ct = kemalg.encap(pk)
    assert kemalg.decap(ct, sk) == ss

However, is there a way to change the behavior of a Strategy based on a pytest fixture?

For example, say that I want to test rejection of correctly sized (but fake) ciphertexts, where "correctly sized" depends on the parameters of the fixture. This does not work:

@given(
  # obviously, this raises NameError: name 'kemalg' is not defined
  # because the pytest fixture doesn't help outside the function body
  binary(min_size=kemalg.CT_SIZE, max_size=kemalg.CT_SIZE)
)
def test_bad_ciphertext(kemalg, bad_ct):
    pk, sk = kemalg.keypair()
    ss, _ = kemalc.encap(pk)
    assert kemalg.decap(bad_ct, sk) != ss

Is this something Hypothesis supports, or is that a bit beyond the scope of it?