Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
SaQC
Manage
Activity
Members
Labels
Plan
Issues
36
Issue boards
Milestones
Wiki
Code
Merge requests
8
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
rdm-software
SaQC
Commits
f7985688
Commit
f7985688
authored
5 years ago
by
David Schäfer
Browse files
Options
Downloads
Patches
Plain Diff
Update GettingStarted.md
parent
0449555a
No related branches found
Branches containing commit
No related tags found
Tags containing commit
Loading
Pipeline
#3247
passed with stage
in 8 minutes and 46 seconds
Changes
1
Pipelines
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
docs/GettingStarted.md
+30
-29
30 additions, 29 deletions
docs/GettingStarted.md
with
30 additions
and
29 deletions
docs/GettingStarted.md
+
30
−
29
View file @
f7985688
...
...
@@ -79,8 +79,8 @@ Now create your our own configuration file `saqc/ressources/data/myconfig.csv`
and paste the following lines into it:
varname;test;plot
SM2;
r
ange(min=10, max=60);False
SM2;spikes_
simple
Mad(window="30d", z=3.5);True
SM2;
flagR
ange(min=10, max=60);False
SM2;spikes_
flag
Mad(window="30d", z=3.5);True
These lines illustrate how different quality control tests can be specified for
different variables by following the pattern:
...
...
@@ -128,6 +128,26 @@ So, what do we see here?
only. Thus, the plot aggregates all preceeding tests (here:
`range`
) to black
points and highlights the flags of the selected test as red points.
#### Save outputs to file
If you want the final results to be saved to a csv-file, you can do so by the
use of the
`-o`
option:
```
sh
saqc
-c
ressources/data/config.csv
-d
ressources/data/data.csv
-o
ressources/data/out.csv
```
Which saves a dataframe that contains both the original data and the quality
flags that were assigned by SaQC for each of the variables:
Date,SM1,SM1_flags,SM2,SM2_flags
2016-04-01 00:05:48,32.685,OK,29.3157,OK
2016-04-01 00:20:42,32.7428,OK,29.3157,OK
2016-04-01 00:35:37,32.6186,OK,29.3679,OK
2016-04-01 00:50:32,32.736999999999995,OK,29.3679,OK
...
### Configure SaQC
#### Change test parameters
...
...
@@ -138,8 +158,8 @@ example, you could modify your `myconfig.csv` and change the parameters of the
range-test:
varname;test;plot
SM2;
r
ange(min=-20, max=60);False
SM2;spikes_
simple
Mad(window="30d", z=3.5);True
SM2;
flagR
ange(min=-20, max=60);False
SM2;spikes_
flag
Mad(window="30d", z=3.5);True
Rerunning SaQC as above produces the following plot:

...
...
@@ -156,10 +176,10 @@ are then executed sequentially and can be plotted seperately. E.g. you could do
something like this:
varname;test;plot
SM1;
r
ange(min=10, max=60);False
SM2;
r
ange(min=10, max=60);False
SM1;spikes_
simple
Mad(window="15d", z=3.5);True
SM2;spikes_
simple
Mad(window="30d", z=3.5);True
SM1;
flagR
ange(min=10, max=60);False
SM2;
flagR
ange(min=10, max=60);False
SM1;spikes_
flag
Mad(window="15d", z=3.5);True
SM2;spikes_
flag
Mad(window="30d", z=3.5);True
which gives you separate plots for each line where the plotting option is set to
`True`
as well as one summary "data plot" that depicts the joint flags from all
...
...
@@ -178,7 +198,7 @@ series. Also, you can write your own tests using a python-based
[
extension language
](
docs/GenericFunctions.md
)
. This would look like this:
varname;test;plot
SM2;harm
onize
_shift2Grid(freq="15Min");False
SM2;harm_shift2Grid(freq="15Min");False
SM2;flagGeneric(func=(SM2 < 30));True
The above executes an internal framework that harmonizes the timestamps of SM2
...
...
@@ -196,23 +216,4 @@ Also, all values where SM2 is below 30 are flagged via the custom function (see
plot below). You can learn more about the syntax of these custom functions
[
here
](
docs/GenericFunctions.md
)
.

#### Save outputs to file
If you want the final results to be saved to a csv-file, you can do so by the
use of the
`-o`
option:
```
sh
saqc
-c
ressources/data/config.csv
-d
ressources/data/data.csv
-o
ressources/data/out.csv
```
Which saves a dataframe that contains both the original data and the quality
flags that were assigned by SaQC for each of the variables:
Date,SM1,SM1_flags,SM2,SM2_flags
2016-04-01 00:05:48,32.685,OK,29.3157,OK
2016-04-01 00:20:42,32.7428,OK,29.3157,OK
2016-04-01 00:35:37,32.6186,OK,29.3679,OK
2016-04-01 00:50:32,32.736999999999995,OK,29.3679,OK
...

\ No newline at end of file
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment