Fuzzy testing with hypothesis
I was playing around with the testing module hypothesis and am pretty convinced by now...I guess, I need to give some more in-depth details into hypothesis at another occassion, I just like to mention a few things to get you jump started reviewing this MR:
-
Hypothsis is a unit testing framework, that does not focus on the typical test style like
funcToTest(input) == expected_output
but more on the testing of invariants that hold under all possible inputs (so something likecheckCondition(funcToTest(fuzzy_input), condition)
, the main example from the docs is in the lines ofdecode(encode(something)) == something
). -
To achieve that, hypothesis comes with features to generate fuzzy input data, that satisfies given conditions and a mechanism to simplify the input to failing tests as much as possible.
While the identification of invariants (not only in saqc
, I guess) is somewhat tricky, I can also benefit from the fuzzy data generation. The following MR show cases implementations for:
- A hypothesis rewrite of
test/core/test_masking.py
- The generation of fuzzy input
data
and initializedflagger
- A generation of fuzzy function calls (including parameters). This is based on type hints (see the
typing
module from the standard lib), that is why I had to add these hints tosaqc/func/functions.py
- Some fixes for bugs (mostly numerical instabilities) resulting from my test runs.
There are quite some changes in this MR, I hope it is understandable anyhow. Comments and reviews are very welcome @palmb and @luenensc !