Skip to content

Fuzzy testing with hypothesis

David Schäfer requested to merge testing into develop

I was playing around with the testing module hypothesis and am pretty convinced by now...I guess, I need to give some more in-depth details into hypothesis at another occassion, I just like to mention a few things to get you jump started reviewing this MR:

  • Hypothsis is a unit testing framework, that does not focus on the typical test style like funcToTest(input) == expected_output but more on the testing of invariants that hold under all possible inputs (so something like checkCondition(funcToTest(fuzzy_input), condition), the main example from the docs is in the lines of decode(encode(something)) == something).

  • To achieve that, hypothesis comes with features to generate fuzzy input data, that satisfies given conditions and a mechanism to simplify the input to failing tests as much as possible.

While the identification of invariants (not only in saqc, I guess) is somewhat tricky, I can also benefit from the fuzzy data generation. The following MR show cases implementations for:

  • A hypothesis rewrite of test/core/test_masking.py
  • The generation of fuzzy input data and initialized flagger
  • A generation of fuzzy function calls (including parameters). This is based on type hints (see the typing module from the standard lib), that is why I had to add these hints to saqc/func/functions.py
  • Some fixes for bugs (mostly numerical instabilities) resulting from my test runs.

There are quite some changes in this MR, I hope it is understandable anyhow. Comments and reviews are very welcome @palmb and @luenensc !

Edited by David Schäfer

Merge request reports