Skip to content
Snippets Groups Projects
Commit 6305859a authored by David Schäfer's avatar David Schäfer
Browse files

Update BrakeDetection.md

parent 2afd552b
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
docs/funcs/BreakDetection.md
+ 5
− 5
View file @ 6305859a
......@@ -9,20 +9,20 @@
breaks_spektrumBased(thresh_rel=0.1, thresh_abs=0.01, first_der_factor=10,
first_der_window_size="12h", scnd_der_ratio_margin_1=0.05,
scnd_der_ratio_margin_2=10, smooth_poly_order=2,
diff_method="raw", filter_window="3h")
smooth_func="raw", smooth_window="3h")
```
| parameter | data type | default value | description |
| ------ | ------ | ------ | ---- |
| thresh_rel | float | `0.1` | Minimum relative difference between two values to consider the latter as break candidate. See condition (1) |
| thresh_abs | float | `0.01` | Minimum relative difference between two values to consider the latter as break candidate. See condition (2) |
| first_der_factor | float | `10` | Factor of the second derivates "arithmetic middle bound". See condition (3). |
| first_der_factor | float | `10` | Factor of the first derivates "arithmetic middle bound". See condition (3). |
| first_der_window_size | [offset string](docs/ParameterDescriptions.md#offset-strings) | `"12h"` | Determining the size of the window, covering all the values included in the the arithmetic middle calculation of condition (3). |
| scnd_der_ratio_margin_1 | float | `0.05` | Range of the area, covering all the values of the second derivatives quotient, that are regarded "sufficiently close to 1" for signifying a break. See condition (5). |
| scnd_der_ratio_margin_2 | float | `10.0` | Lower bound for the break succeeding second derivatives quotients. See condition (5). |
| smooth_poly_order | integer | `2` | When calculating derivatives from smoothed timeseries (diff_method="savgol"), this value gives the order of the fitting polynomial calculated in the smoothing process. |
| diff_method | string | `"savgol"` | Options: <br/> - `"savgol"` <br/> - `"raw"` <br/><br/> Select "raw", to skip smoothing before differenciation. |
| filter_window | [offset string](docs/ParameterDescriptions.md#offset-strings) | `None` | Size of the smoothing window applied with the Savitsky-Golay filter. If None is passed (default), the window size will be two times the sampling rate (thus, covering 3 values). If unsure, keep the default |
| smooth_poly_order | integer | `2` | When calculating derivatives from smoothed timeseries (smooth_func="savgol"), this value gives the order of the fitting polynomial calculated in the smoothing process. |
| smooth_func | string | `"savgol"` | Options: <br/> - `"savgol"` <br/> - `"raw"` <br/><br/> Select "raw", to skip smoothing before differenciation. |
| smooth_window | [offset string](docs/ParameterDescriptions.md#offset-strings) | `None` | Size of the smoothing window applied with the Savitsky-Golay filter. If None is passed (default), the window size will be two times the sampling rate (thus, covering 3 values). If unsure, keep the default |
The function flags breaks (jumps/drops) by evaluating the derivatives of a time series.
......
saqc/funcs/break_detection.py
+ 17
− 17
View file @ 6305859a
......@@ -22,8 +22,8 @@ def flagBreaks_spektrumBased(
scnd_der_ratio_margin_1=0.05,
scnd_der_ratio_margin_2=10,
smooth_poly_order=2,
diff_method="savgol",
filter_window=None,
smooth_func="savgol",
smooth_window=None,
**kwargs
):
......@@ -51,7 +51,7 @@ def flagBreaks_spektrumBased(
constructed breaks.
Especially condition [4] and [5]! This is because smoothing distributes the harshness of the break over the
smoothing window. Since just taking the differences as derivatives did work well for my empirical data set,
the parameter "diff_method" defaults to "raw". That means, that derivatives will be obtained by just using the
the parameter "smooth_func" defaults to "raw". That means, that derivatives will be obtained by just using the
differences series.
You are free of course, to change this parameter to "savgol" and play around with the associated filter options.
(see parameter description below)
......@@ -65,15 +65,15 @@ def flagBreaks_spektrumBased(
:param flags: A dataframe holding the flags/flag-entries associated with "data".
:param field: Fieldname of the Soil moisture measurements field in data.
:param flagger: A flagger - object. (saqc.flagger.X)
:param diff_method: String. Method for obtaining dataseries' derivatives.
:param smooth_func: String. Method for obtaining dataseries' derivatives.
'raw': Just take series step differences (default)
'savgol': Smooth data with a Savitzky Golay Filter before differentiating.
:param filter_window: Offset string. Size of the filter window, used to calculate the derivatives.
(relevant only, if: diff_method='savgol')
:param smooth_window: Offset string. Size of the filter window, used to calculate the derivatives.
(relevant only, if: smooth_func='savgol')
:param smooth_poly_order: Integer. Polynomial order, used for smoothing with savitzk golay filter.
(relevant only, if: diff_method='savgol')
:param thresh_rel Float in [0,1]. See (1) of function descritpion above to learn more
:param thresh_abs Float > 0. See (2) of function descritpion above to learn more.
(relevant only, if: smooth_func='savgol')
:param thresh_rel Float in [0,1]. See (1) of function descritpion above to learn more
:param thresh_abs Float > 0. See (2) of function descritpion above to learn more.
:param first_der_factor Float > 0. See (3) of function descritpion above to learn more.
:param first_der_window_range Offset_String. See (3) of function description to learn more.
:param scnd_der_ratio_margin_1 Float in [0,1]. See (4) of function descritpion above to learn more.
......@@ -83,10 +83,10 @@ def flagBreaks_spektrumBased(
# retrieve data series input at its original sampling rate
dataseries, data_rate = retrieveTrustworthyOriginal(data, field, flagger)
if filter_window is None:
filter_window = 3 * pd.Timedelta(data_rate)
if smooth_window is None:
smooth_window = 3 * pd.Timedelta(data_rate)
else:
filter_window = pd.Timedelta(filter_window)
smooth_window = pd.Timedelta(smooth_window)
# relative - change - break criteria testing:
abs_change = np.abs(dataseries.shift(+1) - dataseries)
......@@ -96,7 +96,7 @@ def flagBreaks_spektrumBased(
breaks = breaks[breaks == True]
# First derivative criterion
smoothing_periods = int(np.ceil((filter_window.seconds / data_rate.n)))
smoothing_periods = int(np.ceil((smooth_window.seconds / data_rate.n)))
if smoothing_periods % 2 == 0:
smoothing_periods += 1
......@@ -111,7 +111,7 @@ def flagBreaks_spektrumBased(
data_slice = dataseries[slice_start:slice_end]
# obtain first derivative:
if diff_method == "savgol":
if smooth_func == "savgol":
first_deri_series = pd.Series(
data=savgol_filter(
data_slice,
......@@ -121,7 +121,7 @@ def flagBreaks_spektrumBased(
),
index=data_slice.index,
)
if diff_method == "raw":
if smooth_func == "raw":
first_deri_series = data_slice.diff()
# condition constructing and testing:
......@@ -140,7 +140,7 @@ def flagBreaks_spektrumBased(
data_slice = data_slice[slice_start:slice_end]
# obtain second derivative:
if diff_method == "savgol":
if smooth_func == "savgol":
second_deri_series = pd.Series(
data=savgol_filter(
data_slice,
......@@ -150,7 +150,7 @@ def flagBreaks_spektrumBased(
),
index=data_slice.index,
)
if diff_method == "raw":
if smooth_func == "raw":
second_deri_series = data_slice.diff().diff()
# criterion evaluation:
......
saqc/funcs/soil_moisture_tests.py
+ 2
− 3
View file @ 6305859a
......@@ -19,9 +19,8 @@ from saqc.lib.tools import (
@register("soilMoisture_spikes")
def flagSoilMoistureSpikes(
data,
field,
flagger,
raise_ffilter_windowarrier=1,
smooth_func,
raise_ffismooth_window1,
noise_window_size="12h",
noise_statistic="CoVar",
filter_window_size=None,
......
test/funcs/test_break_detection.py
+ 2
− 2
View file @ 6305859a
......@@ -20,8 +20,8 @@ def test_flagBreaks_SpektrumBased(data, flagger):
field, *_ = data.columns
data.iloc[5:15] += 100
break_positions = [5, 15]
flagger = flagger.initFlags(data)
data, flagger_result = flagBreaks_spektrumBased(data, field, flagger)
flagger smooth_funcdata)
data,smooth_windowflagBreaks_spektrumBased(data, field, flagger)
flag_result = flagger_result.getFlags(field)
test_sum = (flag_result[break_positions] == flagger.BAD).sum()
assert test_sum == len(break_positions)
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment