Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
SaQC
Manage
Activity
Members
Labels
Plan
Issues
36
Issue boards
Milestones
Wiki
Code
Merge requests
8
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
rdm-software
SaQC
Commits
5566b72c
Commit
5566b72c
authored
4 years ago
by
Peter Lünenschloß
Browse files
Options
Downloads
Patches
Plain Diff
constants detection documented
parent
cc4de8e0
No related branches found
No related tags found
3 merge requests
!193
Release 1.4
,
!188
Release 1.4
,
!78
doc-string doc of test functionality
Pipeline
#6111
passed with stage
in 6 minutes and 31 seconds
Changes
2
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
saqc/funcs/breaks_detection.py
+1
-1
1 addition, 1 deletion
saqc/funcs/breaks_detection.py
saqc/funcs/constants_detection.py
+60
-23
60 additions, 23 deletions
saqc/funcs/constants_detection.py
with
61 additions
and
24 deletions
saqc/funcs/breaks_detection.py
+
1
−
1
View file @
5566b72c
...
@@ -52,7 +52,7 @@ def breaks_flagSpektrumBased(
...
@@ -52,7 +52,7 @@ def breaks_flagSpektrumBased(
data : dios.DictOfSeries
data : dios.DictOfSeries
A dictionary of pandas.Series, holding all the data.
A dictionary of pandas.Series, holding all the data.
field : str
field : str
The fieldname of the column, holding the data-to-be-flagged.
(Here a dummy, for structural reasons)
The fieldname of the column, holding the data-to-be-flagged.
flagger : saqc.flagger
flagger : saqc.flagger
A flagger object, holding flags and additional Informations related to `data`.
A flagger object, holding flags and additional Informations related to `data`.
thresh_rel : float, default 0.1
thresh_rel : float, default 0.1
...
...
This diff is collapsed.
Click to expand it.
saqc/funcs/constants_detection.py
+
60
−
23
View file @
5566b72c
...
@@ -12,14 +12,38 @@ from saqc.lib.tools import retrieveTrustworthyOriginal
...
@@ -12,14 +12,38 @@ from saqc.lib.tools import retrieveTrustworthyOriginal
@register
@register
def
constants_flagBasic
(
data
,
field
,
flagger
,
thresh
,
window
,
**
kwargs
):
def
constants_flagBasic
(
data
,
field
,
flagger
,
thresh
,
window
,
**
kwargs
):
"""
"""
This functions flags plateaus/series of constant values of length `window` if
their maximum total change is smaller than thresh.
Function flags plateaus/series of constant values. Any interval of values y(t),..y(t+n) is flagged, if:
(1) n > `window`
(2) |(y(t + i) - (t + j)| < `thresh`, for all i,j in [0, 1, ..., n]
Flag values are (semi-)constant.
Flag values are (semi-)constant.
:param data: dataframe
Parameters
:param field: column in data
----------
:param flagger: saqc flagger obj
data : dios.DictOfSeries
:param thresh: the difference between two values must be below that
A dictionary of pandas.Series, holding all the data.
:param window: sliding window
field : str
The fieldname of the column, holding the data-to-be-flagged.
flagger : saqc.flagger
A flagger object, holding flags and additional Informations related to `data`.
thresh : float
Upper bound for the maximum total change of an interval to be flagged constant.
window : str
Lower bound for the size of an interval to be flagged constant.
Returns
-------
data : dios.DictOfSeries
A dictionary of pandas.Series, holding all the data.
flagger : saqc.flagger
The flagger object, holding flags and additional informations related to `data`.
Flags values may have changed, relatively to the flagger input.
"""
"""
d
=
data
[
field
]
d
=
data
[
field
]
# find all constant values in a row with a forward search
# find all constant values in a row with a forward search
...
@@ -47,24 +71,37 @@ def constants_flagVarianceBased(
...
@@ -47,24 +71,37 @@ def constants_flagVarianceBased(
"""
"""
Function flags plateaus/series of constant values. Any interval of values y(t),..y(t+n) is flagged, if:
Function flags plateaus/series of constant values. Any interval of values y(t),..y(t+n) is flagged, if:
(1) n >
"
plateau_interval_min
"
(1) n > `window`
(2) variance(y(t),...,y(t+n) < thresh
(2) variance(y(t),...,y(t+n) < `thresh`
:param data: The pandas dataframe holding the data-to-be flagged.
Parameters
Data must be indexed by a datetime series and be harmonized onto a
----------
time raster with seconds precision (skips allowed).
data : dios.DictOfSeries
:param field: Fieldname of the Soil moisture measurements field in data.
A dictionary of pandas.Series, holding all the data.
:param flagger: A flagger - object. (saqc.flagger.X)
field : str
:param window: Offset String. Only intervals of minimum size
"
window
"
have the
The fieldname of the column, holding the data-to-be-flagged.
chance to get flagged as constant intervals
flagger : saqc.flagger
:param thresh: Float. The upper barrier, the variance of an interval mus not exceed, if the
A flagger object, holding flags and additional Informations related to `data`.
interval wants to be flagged a plateau.
window : str
:param max_missing: maximum number of nan values tolerated in an interval, for retrieving a valid
Only intervals of minimum size
"
window
"
have the chance to get flagged as constant intervals
variance from it. (Intervals with a number of nans exceeding
"
max_missing
"
thresh : float
have no chance to get flagged a plateau!)
The upper bound, the variance of an interval must not exceed, if the interval wants to be flagged a plateau.
:param max_consec_missing: Maximum number of consecutive nan values allowed in an interval to retrieve a
max_missing : {None, int}, default None
valid variance from it. (Intervals with a number of nans exceeding
Maximum number of nan values tolerated in an interval, for retrieving a valid
"
max_missing
"
have no chance to get flagged a plateau!)
variance from it. (Intervals with a number of nans exceeding
"
max_missing
"
have no chance to get flagged a plateau!)
max_consec_missing : {None, int}, default None
Maximum number of consecutive nan values allowed in an interval to retrieve a
valid variance from it. (Intervals with a number of nans exceeding
"
max_consec_missing
"
have no chance to get flagged a plateau!)
Returns
-------
data : dios.DictOfSeries
A dictionary of pandas.Series, holding all the data.
flagger : saqc.flagger
The flagger object, holding flags and additional informations related to `data`.
Flags values may have changed, relatively to the flagger input.
"""
"""
dataseries
,
data_rate
=
retrieveTrustworthyOriginal
(
data
,
field
,
flagger
)
dataseries
,
data_rate
=
retrieveTrustworthyOriginal
(
data
,
field
,
flagger
)
...
...
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment