Skip to content
Snippets Groups Projects
Commit 5566b72c authored by Peter Lünenschloß's avatar Peter Lünenschloß
Browse files

constants detection documented

parent cc4de8e0
No related branches found
No related tags found
3 merge requests!193Release 1.4,!188Release 1.4,!78doc-string doc of test functionality
Pipeline #6111 passed with stage
in 6 minutes and 31 seconds
...@@ -52,7 +52,7 @@ def breaks_flagSpektrumBased( ...@@ -52,7 +52,7 @@ def breaks_flagSpektrumBased(
data : dios.DictOfSeries data : dios.DictOfSeries
A dictionary of pandas.Series, holding all the data. A dictionary of pandas.Series, holding all the data.
field : str field : str
The fieldname of the column, holding the data-to-be-flagged. (Here a dummy, for structural reasons) The fieldname of the column, holding the data-to-be-flagged.
flagger : saqc.flagger flagger : saqc.flagger
A flagger object, holding flags and additional Informations related to `data`. A flagger object, holding flags and additional Informations related to `data`.
thresh_rel : float, default 0.1 thresh_rel : float, default 0.1
......
...@@ -12,14 +12,38 @@ from saqc.lib.tools import retrieveTrustworthyOriginal ...@@ -12,14 +12,38 @@ from saqc.lib.tools import retrieveTrustworthyOriginal
@register @register
def constants_flagBasic(data, field, flagger, thresh, window, **kwargs): def constants_flagBasic(data, field, flagger, thresh, window, **kwargs):
""" """
This functions flags plateaus/series of constant values of length `window` if
their maximum total change is smaller than thresh.
Function flags plateaus/series of constant values. Any interval of values y(t),..y(t+n) is flagged, if:
(1) n > `window`
(2) |(y(t + i) - (t + j)| < `thresh`, for all i,j in [0, 1, ..., n]
Flag values are (semi-)constant. Flag values are (semi-)constant.
:param data: dataframe Parameters
:param field: column in data ----------
:param flagger: saqc flagger obj data : dios.DictOfSeries
:param thresh: the difference between two values must be below that A dictionary of pandas.Series, holding all the data.
:param window: sliding window field : str
The fieldname of the column, holding the data-to-be-flagged.
flagger : saqc.flagger
A flagger object, holding flags and additional Informations related to `data`.
thresh : float
Upper bound for the maximum total change of an interval to be flagged constant.
window : str
Lower bound for the size of an interval to be flagged constant.
Returns
-------
data : dios.DictOfSeries
A dictionary of pandas.Series, holding all the data.
flagger : saqc.flagger
The flagger object, holding flags and additional informations related to `data`.
Flags values may have changed, relatively to the flagger input.
""" """
d = data[field] d = data[field]
# find all constant values in a row with a forward search # find all constant values in a row with a forward search
...@@ -47,24 +71,37 @@ def constants_flagVarianceBased( ...@@ -47,24 +71,37 @@ def constants_flagVarianceBased(
""" """
Function flags plateaus/series of constant values. Any interval of values y(t),..y(t+n) is flagged, if: Function flags plateaus/series of constant values. Any interval of values y(t),..y(t+n) is flagged, if:
(1) n > "plateau_interval_min" (1) n > `window`
(2) variance(y(t),...,y(t+n) < thresh (2) variance(y(t),...,y(t+n) < `thresh`
:param data: The pandas dataframe holding the data-to-be flagged. Parameters
Data must be indexed by a datetime series and be harmonized onto a ----------
time raster with seconds precision (skips allowed). data : dios.DictOfSeries
:param field: Fieldname of the Soil moisture measurements field in data. A dictionary of pandas.Series, holding all the data.
:param flagger: A flagger - object. (saqc.flagger.X) field : str
:param window: Offset String. Only intervals of minimum size "window" have the The fieldname of the column, holding the data-to-be-flagged.
chance to get flagged as constant intervals flagger : saqc.flagger
:param thresh: Float. The upper barrier, the variance of an interval mus not exceed, if the A flagger object, holding flags and additional Informations related to `data`.
interval wants to be flagged a plateau. window : str
:param max_missing: maximum number of nan values tolerated in an interval, for retrieving a valid Only intervals of minimum size "window" have the chance to get flagged as constant intervals
variance from it. (Intervals with a number of nans exceeding "max_missing" thresh : float
have no chance to get flagged a plateau!) The upper bound, the variance of an interval must not exceed, if the interval wants to be flagged a plateau.
:param max_consec_missing: Maximum number of consecutive nan values allowed in an interval to retrieve a max_missing : {None, int}, default None
valid variance from it. (Intervals with a number of nans exceeding Maximum number of nan values tolerated in an interval, for retrieving a valid
"max_missing" have no chance to get flagged a plateau!) variance from it. (Intervals with a number of nans exceeding "max_missing"
have no chance to get flagged a plateau!)
max_consec_missing : {None, int}, default None
Maximum number of consecutive nan values allowed in an interval to retrieve a
valid variance from it. (Intervals with a number of nans exceeding
"max_consec_missing" have no chance to get flagged a plateau!)
Returns
-------
data : dios.DictOfSeries
A dictionary of pandas.Series, holding all the data.
flagger : saqc.flagger
The flagger object, holding flags and additional informations related to `data`.
Flags values may have changed, relatively to the flagger input.
""" """
dataseries, data_rate = retrieveTrustworthyOriginal(data, field, flagger) dataseries, data_rate = retrieveTrustworthyOriginal(data, field, flagger)
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment