Newer
Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
# Generic Functions
Generic Functions provide a possibility to implement simple quality checks
directly within the configuration using a simple, Python based extension
language.
## Specification
Generic funtions are used in the same manner as their
[non-generic counterparts](docs/FunctionDescriptions.md). The basic
signature looks like that:
```sh
generic(func=<expression>, flag=<flagging_constant>)
```
where `<expression>` is composed of the [supported constructs](#supported-constructs)
and `<flag_constant>` is either one of the predefined
[flagging constants](docs/ParameterDescriptions.md#flagging-constants) or any value supported
by the flagger in use.
## Examples
### Simple comparisons
#### Task
Flag all values of variable `x` when variable `y` falls below a certain threashold
#### Configuration file
| varname | test |
|---------|-----------------------|
| `x` | `generic(func=y < 0)` |
### Calculations
#### Task
Flag all values of variable `x` that exceed 3 standard deviations of variable `y`
#### Configuration file
| varname | test |
|---------|-----------------------------------|
| `x` | `generic(func=this > std(y) * 3)` |
### Special functions
#### Task
Flag variable `x` where variable `y` is flagged and variable `x` has missing values
#### Configuration file
| varname | test |
|---------|---------------------------------------------------|
| `x` | `generic(func=this > isflagged(y) & ismissing(z)` |
## Variable References
All variables of the processed dataset are available within generic functions, so
arbitrary cross references are possible. The variable of intereset
is furthermore available with the special reference `this`, so the second
[example](#calculations) could be rewritten as:
| varname | test |
|---------|--------------------------------|
| `x` | `generic(func=x > std(y) * 3)` |
When referencing other variables, their flags will be respected during evaluation
of the generic expression. So, in the example above only previously
unflagged values of `x` and `y` are used within the expression `x > std(y)*3`.
## Supported constructs
### Operators
#### Comparsions
The following comparison operators are available:
| Operator | Description |
|----------|----------------------------------------------------------------------------------------------------|
| `==` | `True` if the values of the operands are equal |
| `!=` | `True` if the values of the operands are not equal |
| `>` | `True` if the values of the left operand are greater than the values of the right operand |
| `<` | `True` if the values of the left operand are smaller than the values of the right operand |
| `>=` | `True` if the values of the left operand are greater or equal than the values of the right operand |
| `<=` | `True` if the values of the left operand are smaller or equal than the values of the right operand |
#### Arithmetics
The following arithmetic operators are supported:
| Operator | Description |
|----------|----------------|
| `+` | addition |
| `-` | substraction |
| `*` | multiplication |
| `/` | division |
| `**` | exponantion |
| `%` | modulus |
#### Bitwise
The bitwise operators also act as logical operators in comparison chains
| Operator | Description |
|----------|-------------------|
| `&` | binary and |
| | | binary or |
| `^` | binary xor |
| `~` | binary complement |
### Functions
All functions expect a [variable reference](#variable-references)
as the only non-keyword argument (see [here](#special-functions))
| Name | Description |
|-------------|-----------------------------------|
| `abs` | absolute values of a variable |
| `max` | maximum value of a variable |
| `min` | minimum value of a variable |
| `mean` | mean value of a variable |
| `sum` | sum of a variable |
| `std` | standard deviation of a variable |
| `len` | the number of values for variable |
| `ismissing` | check for missing values |
| `isflagged` | check for flags |
### Constants
Generic functions support the same constants as normal functions, a detailed
list is available [here](docs/ParameterDescriptions.md#constants).