Miscellaneous.md

Miscellaneous

A collection of unrelated quality check functions.

Index

• flagRange
• flagSeasonalRange
• flagIsolated
• flagPattern
• flagMissing
• clearFlags
• forceFlags

flagRange

flagRange(min, max)

parameter	data type	default value	description
min	float		The upper bound for valid values
max	float		The lower bound for valid values

The function flags all values outside the closed interval

$[$

min, max

$]$

.

flagSeasonalRange

flagSeasonalRange(min, max, startmonth=1, endmonth=12, startday=1, endday=31)

parameter	data type	default value	description
min	float		The upper bound for valid values
max	float		The lower bound for valid values
startmonth	integer	1	The interval start month
endmonth	integer	12	The interval end month
startday	integer	1	The interval start day
endday	integer	31	The interval end day

The function does the same as flagRange, but only if the timestamp of the data-point lies in a defined interval, which is build from days and months only. In particular, the year is not considered in the Interval.

The left boundary is defined by startmonth and startday, the right boundary by endmonth and endday. Both boundaries are inclusive. If the left side occurs later in the year than the right side, the interval is extended over the change of year (e.g. an interval of [01/12, 01/03], will flag values in December, January and February).

NOTE: Only works for time-series-like datasets.

flagIsolated

flagIsolated(window, gap_window, group_window) 

parameter	data type	default value	description
gap_window	offset string		The minimum size of the gap before and after a group of valid values, which makes this group considered as isolated. See condition (2) and (3)
group_window	offset string		The maximum size of an isolated group of valid data. See condition (1).

The function flags arbitrary large groups of values, if they are surrounded by sufficiently large data gaps. A gap is defined as group of missing and/or flagged values.

A continuous group of values

$x_{k}, x_{k+1},...,x_{k+n}$

with timestamps

$t_{k}, t_{k+1}, ..., t_{k+n}$

is considered to be isolated, if:

1. $t_{k+n} - t_{k} \le$
   group_window
2. None of the values
   $x_i, ..., x_{k-1}$
   , with
   $t_{k-1} - t_{i} \ge$
   gap_window is valid or unflagged
3. None of the values
   $x_{k+n+1}, ..., x_{j}$
   , with
   $t_{j} - t_{k+n+1} \ge$
   gap_window is valid or unflagged

flagMissing

flagMissing(nodata=NaN)

parameter	data type	default value	description
nodata	any	NAN	A value that defines missing data

The function flags all values indicating missing data.

flagPattern

flagPattern(ref_datafield, sample_freq = '15 Min', method = 'dtw', min_distance = None)

parameter	data type	default value	description
ref_datafield	string		Name of the reference datafield = "pattern"
sample_freq	string	"15 Min"	Sample frequency to harmonize the data
method	string	"dtw "	"dtw" for Dynamic Time Warping (DTW), "wavelet" for Wavelet Pattern Recognition Algorithm
min_distance	float	None	For DTW - alogrithm: the minimum distance of two graphs in order to be classified as "different"

Implementation of the pattern recognition algorithms introduced in Pattern Recognition.

clearFlags

clearFlags()

The funcion removes all previously set flags.

forceFlags

forceFlags(flag)

parameter	data type	default value	description
flag	float/flagging constant	GOOD	The flag that is set unconditionally

The functions overwrites all previous set flags with the given flag.

flagDummy

flagDummy()

Identity function, i.e. the function does nothing.