Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
### User Defined Test
User defined tests allow to specify simple quality checks directly within the
configuration.
#### Specification
- Test name: `generic`
- The parameter 'func' followed by an expression needs to be given
- Example: generic, `{func: (thisvar > 0) & ismissing(othervar)}`
#### Restrictions
- only the operators and functions listed below are available
- all checks need to be conditional expression and have to return an array of boolean values,
all other expressions are rejected. This limitation is enforced to somewhat narrow the
scope of the system and therefore the potential to mess things up and might as well be
removed in the future.
#### Syntax
- standard Python syntax
- all variables within the configuration file can be used
#### Supported Operators
- all arithmetic operators
- all comparison operators
- bitwise operators: and, or, xor, complement (`&`, `|`, `^`, `~`)
#### Supported functions
| function name | description |
|---------------|------------------------------------------------------------------|
| `abs` | absolute values of a variable |
| `max` | maximum value of a variable |
| `min` | minimum value of a variable |
| `mean` | mean value of a variable |
| `sum` | sum of a variable |
| `std` | standard deviation of a variable |
| `len` | the number of values of variable |
| `ismissing` | check for missing values (nan and a possibly user defined value) |
#### Referencing Semantics
If another variable is reference within an generic test, the flags from that variable are
propagated to the checked variable.
For example:
Let `var1` and `var2` be two variables of a given dataset and `func: var1 > mean(var1)`
the condition wheter to flag `var2`. The result of the check can be described
as `isflagged(var1) & istrue(func())`.