FunctionDescriptions.md

Implemented QC functions

range

Signature

range(min, max)

Description

missing

Signature

missing(nodata=NaN)

Description

The Function flags those values in the the passed data series, that are associated with "missing" data. The missing data indicator (np.nan by default) , can be altered to any other value by passing this new value to the parameter nodata.

sesonalRange

Signature

sesonalRange(min, max, startmonth=1, endmonth=12, startday=1, endday=31)

clear

Signature

clear()

Description

force

Signature

force()

Description

sliding_outlier

Signature

sliding_outlier(winsz="1h", dx="1h", count=1, deg=1, z=3.5, method="modZ")

Description

mad

Signature

mad(length, z=3.5, freq=None)

Description

Spikes_Basic

Signature

Spikes_Basic(thresh=7, tol=0, length="15min")

Description

A basic outlier test, that is designed to work for harmonized, as well as raw (not-harmonized) data.

The values x(n), x(n+1), .... , x(n+k) of a passed timeseries x, are considered spikes, if:

1. |x(n-1) - x(n + s)| > thresh, for all integers s in {0,1,2,...,k}

2. |x(n-1) - x(n+k+1)| < tol

3. |x(n-1).index - x(n+k+1).index| < length

By this definition, spikes are values, that, after a jump of margin thresh(1), are keeping that new value level they jumped to, for a timespan smaller than length (3), and do then return to the initial value level - within a tolerance margin of tol (2).
Note, that this characterization of a "spike", not only includes one-value outliers, but also plateau-ish value courses.

The implementation is a time-window based version of an outlier test from the UFZ Python library, that can be found here:

https://git.ufz.de/chs/python/blob/master/ufz/level1/spike.py

Spikes_SpektrumBased

Signature

Spikes_SpektrumBased(filter_window_size="3h", raise_factor=0.15, dev_cont_factor=0.2,
                     noise_barrier=1, noise_window_size="12h", noise_statistic="CoVar",
                     smooth_poly_order=2)

Description

The function detects and flags spikes in input data series by evaluating the the timeseries' derivatives and applying some conditions to it.

NOTE, that the dataseries-to-be flagged is supposed to be harmonized to an equadistant frequencie grid.

A datapoint x(k) of a dataseries x, is considered a spike, if:

1. The quotient to its preceeding datapoint exceeds a certain bound:
   • x(k)/x(k-1) > 1 + raise_factor, or:
   • x(k)/x(k-1) < 1 - raise_factor
2. The quotient of the datas second derivate x'', at the preceeding and subsequent timestamps is close enough to 1:
   • (1 - dev_cont_factor) < | x''(k-1)/x''(k+1) |, and
   • (1 + dev_cont_factor) > | x''(k-1)/x''(k+1) |
3. The dataset, surrounding x(k), within noise_window_size range, but excluding x(k), is not too noisy. Wheras the noisyness gets measured by noise_statistic:
   • 'noise_statistic'(x.index(k-'noise_window_size'),..., x.index(k+'noise_window') < noise_barrier

This Function is a generalization of the Spectrum based Spike flagging mechanism as presented in:

Dorigo,W,.... Global Automated Quality Control of In Situ Soil Moisture Data from the international Soil Moisture Network. 2013. Vadoze Zone J. doi:10.2136/vzj2012.0097.

constant

Signature

constant(eps, length, thmin=None)

Description

constants_varianceBased

Signature

constants_varianceBased(plateau_window_min="12h", plateau_var_limit=0.0005,
                        var_total_nans=Inf, var_consec_nans=Inf)

Description

SoilMoistureSpikes

Signature

SoilMoistureSpikes(filter_window_size="3h", raise_factor=0.15, dev_cont_factor=0.2,
                   noise_barrier=1, noise_window_size="12h", noise_statistic="CoVar")

Description

The Function is just a wrapper around flagSpikes_SpektrumBased, from the spike detection library and performs a call to this function with a parameter set, referring to:

Dorigo,W,.... Global Automated Quality Control of In Situ Soil Moisture Data from the international Soil Moisture Network. 2013. Vadoze Zone J. doi:10.2136/vzj2012.0097.

SoilMoistureBreaks

Signature

SoilMoistureBreaks(diff_method="raw", filter_window_size="3h",
                   rel_change_rate_min=0.1, abs_change_min=0.01, first_der_factor=10,
                   first_der_window_size="12h", scnd_der_ratio_margin_1=0.05,
                   scnd_der_ratio_margin_2=10, smooth_poly_order=2)

Description

The Function is just a wrapper around flagBreaks_SpektrumBased, from the breaks detection library and performs a call to this function with a parameter set, referring to:

Dorigo,W,.... Global Automated Quality Control of In Situ Soil Moisture Data from the international Soil Moisture Network. 2013. Vadoze Zone J. doi:10.2136/vzj2012.0097.

SoilMoistureByFrost

Signature

SoilMoistureByFrost(soil_temp_reference, tolerated_deviation="1h", frost_level=0)

Description

SoilMoistureByPrecipitation

Signature

SoilMoistureByPrecipitation(prec_reference, sensor_meas_depth=0,
                            sensor_accuracy=0, soil_porosity=0,
                            std_factor=2, std_factor_range="24h")

Description

Breaks_SpektrumBased

Signature

Breaks_SpektrumBased(diff_method="raw", filter_window_size="3h",
                     rel_change_rate_min=0.1, abs_change_min=0.01, first_der_factor=10,
                     first_der_window_size="12h", scnd_der_ratio_margin_1=0.05,
                     scnd_der_ratio_margin_2=10, smooth_poly_order=2)

Description