| isolation_range | string | | Offset string. The range, within there are no valid values allowed for a valuegroup to get flagged isolated. See condition (1) and (2).|
| continuation_range | string | `"1min"` | Offset string. The upper bound for the temporal extension of a value group to be considered an isolated group. See condition (4). Only relevant if `max_islated_group_size` > 1.|
| drop_flags | list or Nonetype| `None` | A list of flags, that are to be considered, signifying invalid values. See condition (1) and (2).|
### Description
The function flags isolated values / value groups.
Isolated values are values / value groups,
...
...
@@ -59,33 +53,28 @@ is considered "isolated", if:
## `missing`
## missing
### Signature
```
missing(nodata=NaN)
```
### Parameters
| parameter | data type | default value | description |
| filter_window_size | Nonetype or string | `None` | Options: <br/> - `None`<br/> - any offset string <br/><br/> Controlls the range of the smoothing window applied with the Savitsky-Golay filter. If None is passed (default), the window size will be two times the sampling rate. (Thus, covering 3 values.) If you are not very well knowing what you are doing - do not change that value. Broader window sizes caused unexpected results during testing phase.|
### Description
The function detects and flags spikes in input data series by evaluating the
the timeseries' derivatives and applying some conditions to them.
...
...
@@ -259,32 +229,27 @@ Data from the international Soil Moisture Network. 2013. Vadoze Zone J.
doi:10.2136/vzj2012.0097.
## `constant`
## constant
### Signature
```
constant(eps, length, thmin=None)
```
### Parameters
| parameter | data type | default value | description |
| parameter | data type | default value | description |
| ------ | ------ | ------ | ---- |
| plateau_window_min | string | | Options <br/> - any offset string <br/><br/> Minimum barrier for the duration, values have to be continouos to be plateau canditaes. See condition (1). |
| var_consec_nans | integer | `Inf` | Maximum number of consecutive nan values allowed, for a calculated variance to be valid. (Default skips the condition.) |
### Description
Function flags plateaus/series of constant values. Any set of consecutive values
$`x_k,..., x_{k+n}`$ of a timeseries $`x`$ is flagged, if:
...
...
@@ -307,9 +271,8 @@ NOTE, that when `var_total_nans` or `var_consec_nans` are set to a value < `Inf`
, plateaus that can not be calculated the variance of, due to missing values,
will never be flagged. (Test not applicable rule.)
| parameter | data type | default value | description |
| ------ | ------ | ------ | ---- |
| plateau_window_min | string | `"12h"` | Options <br/> - any offset string <br/><br/> Minimum barrier for the duration, values have to be continouos to be plateau canditaes. See condition (1).|
| parameter | data type | default value | description |
| ------ | ------ | ------ | ---- |
| soil_temp_reference | string | | A string, denoting the fields name in data, that holds the data series of soil temperature values, the to-be-flagged values shall be checked against.|
| tolerated_deviation | string | `"1h"` | An offset string, denoting the maximal temporal deviation, the soil frost states timestamp is allowed to have, relative to the data point to be flagged.|
| frost_level | integer | `0` | Value level, the flagger shall check against, when evaluating soil frost level. |
### Description
The function flags Soil moisture measurements by evaluating the soil-frost-level
in the moment of measurement (+/- `tolerated deviation`).
...
...
@@ -458,9 +410,8 @@ All parameters default to the values, suggested in this publication.
| parameter | data type | default value | description |
| ------ | ------ | ------ | ---- |
| prec_reference | string | | A string, denoting the fields name in data, that holds the data series of precipitation values, the to-be-flagged values shall be checked against. |
| ignore_missing | bool | `False` | If True, the variance of condition (2), will also be calculated if there is a value missing in the time window. Selcting Flase (default) results in values that succeed a time window containing a missing value never being flagged (test not applicable rule) |
### Description
Function flags Soil moisture measurements by flagging moisture rises that do not follow up a sufficient
precipitation event. If measurement depth, sensor accuracy of the soil moisture sensor and the porosity of the
...
...
@@ -516,9 +465,8 @@ doi:10.2136/vzj2012.0097.
All parameters default to the values, suggested in this publication.
| parameter | data type | default value | description |
| ------ | ------ | ------ | ---- |
| rel_change_rate_min | float | `0.1` | Lower bound for the relative difference, a value has to have to its preceeding value, to be a candidate for being break-flagged. See condition (2).|
| filter_window_size | Nonetype or string | `None` | Options: <br/> - `None` <br/> - any offset string <br/><br/> Controlls the range of the smoothing window applied with the Savitsky-Golay filter. If None is passed (default), the window size will be two times the sampling rate. (Thus, covering 3 values.) If you are not very well knowing what you are doing - do not change that value. Broader window sizes caused unexpected results during testing phase.|
### Description
The function flags breaks (jumps/drops) in input measurement series by
evaluating its derivatives.
...
...
@@ -570,14 +516,12 @@ Dorigo,W. et al.: Global Automated Quality Control of In Situ Soil Moisture
Data from the international Soil Moisture Network. 2013. Vadoze Zone J.
| path | string | | Path to the respective model object, i.e. its name and the respective value of the grouping variable. e.g. "models/model_0.2.pkl" |
### Description
This Function uses pre-trained machine-learning model objects for flagging.
This requires training a model by use of the [training script](../ressources/machine_learning/train_machine_learning.py) provided.
For flagging, inputs to the model are the data of the variable of interest,
...
...
@@ -600,9 +543,8 @@ the user during model training. For the model to work, the parameters
values as during training. For a more detailed description of the modeling
aproach see the [training script](../ressources/machine_learning/train_machine_learning.py).
| drop_flags | list or Nonetype |`None` | A list of flags to exclude from harmonization. See step (1) below. If `None` is passed, only BAD - flagged values get dropped. If a list is passed, the BAD flag gets added to that list by default |
| data_missing_value | any valeu |`np.nan` | The value, indicating missing data in the dataseries-to-be-flagged.|
### Description
The function "harmonizes" the data-to-be-flagged, to match an equidistant
frequency grid. In general this includes projection and/or interpolation of
...
...
@@ -720,19 +660,16 @@ Key word overview:
* `"nearest_agg"`: all flags in the range (+/- freq/2) of a grid point get
aggregated with the function passed to agg_method and assigned to it.
## `deharmonize`
## deharmonize
### Signature
```
deharmonize(co_flagging)
```
### Parameters
| parameter | data type | default value | description |
| co_flagging | boolean | | `False`: depending on the harmonization method applied, only overwrite ultimately preceeding, first succeeding or nearest flag to a harmonized flag. <br/> `True`: Depending on the harmonization method applied, overwrite all the values covered by the succeeding or preceeding sampling intervall, or, all the values in the range of a harmonic flags timestamp. |
### Description
After having calculated flags on an equidistant frequency grid, generated by
a call to a harmonization function, you may want to project
