Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
SaQC
Manage
Activity
Members
Labels
Plan
Issues
36
Issue boards
Milestones
Wiki
Code
Merge requests
8
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Admin message
UFZ GitLab and Mattermost will be
OFFLINE
on
March 27
from
8:00pm
to
10pm
due to a system migration!
Show more breadcrumbs
rdm-software
SaQC
Commits
0765faf2
Commit
0765faf2
authored
4 years ago
by
Peter Lünenschloß
Browse files
Options
Downloads
Patches
Plain Diff
sphinx bugs solved
parent
cc3c0e6e
No related branches found
Branches containing commit
No related tags found
Tags containing commit
2 merge requests
!193
Release 1.4
,
!188
Release 1.4
Pipeline
#11680
failed with stages
Stage: test
Stage: deploy
in 9 minutes and 15 seconds
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
saqc/funcs/spikes_detection.py
+45
-32
45 additions, 32 deletions
saqc/funcs/spikes_detection.py
with
45 additions
and
32 deletions
saqc/funcs/spikes_detection.py
+
45
−
32
View file @
0765faf2
...
@@ -524,7 +524,7 @@ def spikes_flagRaise(
...
@@ -524,7 +524,7 @@ def spikes_flagRaise(
**
kwargs
,
**
kwargs
,
):
):
"""
"""
The function flags rises and drops in value courses, that exceed a certain threshold
The function flags r
a
ises and drops in value courses, that exceed a certain threshold
within a certain timespan.
within a certain timespan.
The parameter variety of the function is owned to the intriguing
The parameter variety of the function is owned to the intriguing
...
@@ -554,11 +554,11 @@ def spikes_flagRaise(
...
@@ -554,11 +554,11 @@ def spikes_flagRaise(
See condition (2) of the description linked in the references. Window is inclusively defined.
See condition (2) of the description linked in the references. Window is inclusively defined.
The window defaults to 1.5 times the size of `raise_window`
The window defaults to 1.5 times the size of `raise_window`
mean_raise_factor : float, default 2
mean_raise_factor : float, default 2
See cond
ition (2) of the descrip
tion li
nk
ed in the
references
.
See
se
cond
condi
tion li
st
ed in the
notes below
.
min_slope : {None, float}, default None
min_slope : {None, float}, default None
See
condition (3) of the descrip
tion li
nk
ed in the
references
See
third condi
tion li
st
ed in the
notes below.
min_slope_weight : float, default 0.8
min_slope_weight : float, default 0.8
See
condition (3) of the descrip
tion li
nk
ed in the
references
See
third condi
tion li
st
ed in the
notes below.
numba_boost : bool, default True
numba_boost : bool, default True
Returns
Returns
...
@@ -572,17 +572,22 @@ def spikes_flagRaise(
...
@@ -572,17 +572,22 @@ def spikes_flagRaise(
Notes
Notes
-----
-----
The value :math:`x_{k}` of a time series :math:`x` with associated
The value :math:`x_{k}` of a time series :math:`x` with associated
timestamps :math:`t_i`, is flagged a rise, if:
timestamps :math:`t_i`, is flagged a raise, if:
1. There is any value :math:`x_{s}`, preceeding :math:`x_{k}` within `raise_window` range, so that:
* There is any value :math:`x_{s}`, preceeding :math:`x_{k}` within `raise_window` range, so that:
* :math:` M = |x_k - x_s | > ` `thresh` :math:` > 0`
2. The weighted average :math:`\mu^*` of the values, preceeding :math:`x_{k}` within `average_window`
* :math:`M = |x_k - x_s | >` `thresh` :math:`> 0`
range indicates, that :math:`x_{k}`$ doesnt return from an outliererish value course, meaning that:
* :math:` x_k > \mu^* + ( M ` / `mean_raise_factor` :math:`)`
* The weighted average :math:`
\\
mu^{*}` of the values, preceding :math:`x_{k}` within `average_window`
3. Additionally, if `min_slope` is not `None`, :math:`x_{k}` is checked for being sufficiently divergent from its
range indicates, that :math:`x_{k}` does not return from an
"
outlierish
"
value course, meaning that:
very predecessor $`x_{k-1}`$, meaning that, it is additionally checked if:
* :math:`x_k - x_{k-1} > ` `min_slope`
* :math:`x_k >
\\
mu^* + ( M` / `mean_raise_factor` :math:`)`
* :math:`t_k - t_{k-1} > ` `min_slope_weight`*`intended_freq`
* Additionally, if `min_slope` is not `None`, :math:`x_{k}` is checked for being sufficiently divergent from its
very predecessor :max:`x_{k-1}`$, meaning that, it is additionally checked if:
* :math:`x_k - x_{k-1} >` `min_slope`
* :math:`t_k - t_{k-1} >` `min_slope_weight` :math:`
\\
times` `intended_freq`
"""
"""
...
@@ -883,13 +888,14 @@ def spikes_flagBasic(data, field, flagger, thresh, tolerance, window, numba_kick
...
@@ -883,13 +888,14 @@ def spikes_flagBasic(data, field, flagger, thresh, tolerance, window, numba_kick
The test classifies values/value courses as outliers by detecting not only a rise in value, but also,
The test classifies values/value courses as outliers by detecting not only a rise in value, but also,
checking for a return to the initial value level.
checking for a return to the initial value level.
Values x(n), x(n+1), .... , x(n+k) of a timeseries x are considered spikes, if
Values :math:`x_n, x_{n+1}, .... , x_{n+k}` of a timeseries :math:`x` with associated timestamps
:math:`t_n, t_{n+1}, .... , t_{n+k}` are considered spikes, if
(1) |x(
n-1
)
- x
(
n + s
)
| > `thresh`, for all
s
in [0,1,2,...,k]
1. :math:`|x_{
n-1
}
- x
_{
n + s
}
| >
`
`thresh`, for all
:math:`s
\\
in [0,1,2,...,k]
`
(2) |x(
n-1
)
- x
(
n+k+1
)
| < `tolerance`
2. :math:`|x_{
n-1
}
- x
_{
n+k+1
}
| <
`
`tolerance`
(3) |x(n-1).index - x(n+k+1).index
| < `windo
o
w`
3. :math:`|t_{n-1} - t_{n+k+1}
| <
`
`window`
Note, that this definition of a
"
spike
"
not only includes one-value outliers, but also plateau-ish value courses.
Note, that this definition of a
"
spike
"
not only includes one-value outliers, but also plateau-ish value courses.
...
@@ -1025,8 +1031,10 @@ def spikes_flagSpektrumBased(
...
@@ -1025,8 +1031,10 @@ def spikes_flagSpektrumBased(
See condition (2) (or reference [2]).
See condition (2) (or reference [2]).
noise_func : {
'
CoVar
'
,
'
rVar
'
}, default
'
CoVar
'
noise_func : {
'
CoVar
'
,
'
rVar
'
}, default
'
CoVar
'
Function to calculate noisiness of the data surrounding potential spikes.
Function to calculate noisiness of the data surrounding potential spikes.
``
'
CoVar
'
``: Coefficient of Variation
``
'
rVar
'
``: Relative Variance
* ``
'
CoVar
'
``: Coefficient of Variation
* ``
'
rVar
'
``: Relative Variance
noise_window : str, default
'
12h
'
noise_window : str, default
'
12h
'
An offset string that determines the range of the time window of the
"
surrounding
"
data of a potential spike.
An offset string that determines the range of the time window of the
"
surrounding
"
data of a potential spike.
See condition (3) (or reference [2]).
See condition (3) (or reference [2]).
...
@@ -1058,17 +1066,22 @@ def spikes_flagSpektrumBased(
...
@@ -1058,17 +1066,22 @@ def spikes_flagSpektrumBased(
-----
-----
A value is flagged a spike, if:
A value is flagged a spike, if:
1. The quotient to its preceding data point exceeds a certain bound:
* The quotient to its preceding data point exceeds a certain bound:
* :math:`|
\f
rac{x_k}{x_{k-1}}| > 1 + ` `raise_factor`, or
* :math:`|
\f
rac{x_k}{x_{k-1}}| < 1 - ` `raise_factor`
* :math:`|
\\
frac{x_k}{x_{k-1}}| > 1 +` ``raise_factor``, or
2. The quotient of the second derivative :math:`x
''
`, at the preceding
* :math:`|
\\
frac{x_k}{x_{k-1}}| < 1 -` ``raise_factor``
and subsequent timestamps is close enough to 1:
* :math:` |
\f
rac{x
''
_{k-1}}{x
''
_{k+1}} | > 1 - ` `deriv_factor`, and
* The quotient of the second derivative :math:`x
''
`, at the preceding
* :math:` |
\f
rac{x
''
_{k-1}}{x
''
_{k+1}} | < 1 + ` `deriv_factor`
and subsequent timestamps is close enough to 1:
3. The dataset :math:`X = x_i, ..., x_{k-1}, x_{k+1}, ..., x_j`, with
:math:`|t_{k-1} - t_i| = |t_j - t_{k+1}| =` `noise_window` fulfills the
* :math:`|
\\
frac{x
''
_{k-1}}{x
''
_{k+1}} | > 1 -` ``deriv_factor``, and
following condition:
* :math:`|
\\
frac{x
''
_{k-1}}{x
''
_{k+1}} | < 1 +` ``deriv_factor``
`noise_func`:math:`(X) <` `noise_thresh`
* The dataset :math:`X = x_i, ..., x_{k-1}, x_{k+1}, ..., x_j`, with
:math:`|t_{k-1} - t_i| = |t_j - t_{k+1}| =` ``noise_window`` fulfills the
following condition:
* ``noise_func``:math:`(X) <` ``noise_thresh``
"""
"""
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment