Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
SaQC
Manage
Activity
Members
Labels
Plan
Issues
36
Issue boards
Milestones
Wiki
Code
Merge requests
8
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Build
Pipelines
Jobs
Pipeline schedules
Artifacts
Deploy
Releases
Container Registry
Model registry
Operate
Environments
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
CI/CD analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Admin message
UFZ GitLab and Mattermost will be
OFFLINE
on
March 27
from
8:00pm
to
10pm
due to a system migration!
Show more breadcrumbs
rdm-software
SaQC
Commits
666aba5d
Commit
666aba5d
authored
5 years ago
by
Peter Lünenschloß
Browse files
Options
Downloads
Patches
Plain Diff
updated functionDescription.md
parent
fcd92e40
No related branches found
Branches containing commit
No related tags found
Tags containing commit
No related merge requests found
Changes
1
Hide whitespace changes
Inline
Side-by-side
Showing
1 changed file
docs/FunctionDescriptions.md
+107
-5
107 additions, 5 deletions
docs/FunctionDescriptions.md
with
107 additions
and
5 deletions
docs/FunctionDescriptions.md
+
107
−
5
View file @
666aba5d
...
@@ -698,7 +698,7 @@ Key word overview:
...
@@ -698,7 +698,7 @@ Key word overview:
3. Interpolations:
3. Interpolations:
* There are available all the interpolation methods from the pandas.interpolate() method and they can be reffered to with
* There are available all the interpolation methods from the pandas.interpolate() method and they can be reffered to with
the very same keywords, that you would pass to pd.Series.interpolates's method parameter.
the very same keywords, that you would pass to pd.Series.interpolates's method parameter.
* Available interpolations:
´
"linear"
´
,
´
"time"
´
,
´
"nearest"
´
,
´
"zero"
´
,
´
"slinear"
´
,
* Available interpolations:
`
"linear"
`
,
`
"time"
`
,
`
"nearest"
`
,
`
"zero"
`
,
`
"slinear"
`
,
`
"quadratic"
`, `
"cubic"
`, `
"spline"
`, `
"barycentric"
`, `
"polynomial"
`, `
"krogh"
`,
`
"quadratic"
`, `
"cubic"
`, `
"spline"
`, `
"barycentric"
`, `
"polynomial"
`, `
"krogh"
`,
`
"piecewise_polynomial"
`, `
"spline"
`, `
"pchip"
`, `
"akima"
`.
`
"piecewise_polynomial"
`, `
"spline"
`, `
"pchip"
`, `
"akima"
`.
* If a selected interpolation method needs to get passed an order of
* If a selected interpolation method needs to get passed an order of
...
@@ -826,7 +826,7 @@ harmonize_aggregate2Grid(freq, agg_func, agg_method='nearest_agg', flag_agg_func
...
@@ -826,7 +826,7 @@ harmonize_aggregate2Grid(freq, agg_func, agg_method='nearest_agg', flag_agg_func
```
```
| parameter | data type | default value | description |
| parameter | data type | default value | description |
| --------- | --------- | ------------- | ----------- |
| --------- | --------- | ------------- | ----------- |
| freq | string | | Offset string. Determining the sampling rate of the frequency grid, the data shall be
shif
ted to. |
| freq | string | | Offset string. Determining the sampling rate of the frequency grid, the data shall be
aggrega
ted to. |
| agg_func | func | | Function. Function used for data aggregation.|
| agg_func | func | | Function. Function used for data aggregation.|
| agg_method | string | `
nearest_agg
` | Method, determining the range of data and flags aggregation. See a list of methods below. |
| agg_method | string | `
nearest_agg
` | Method, determining the range of data and flags aggregation. See a list of methods below. |
| flag_agg_func | func | max | Function used for flags aggregation.|
| flag_agg_func | func | max | Function used for flags aggregation.|
...
@@ -859,7 +859,7 @@ In detail, the process includes:
...
@@ -859,7 +859,7 @@ In detail, the process includes:
data (Thus get assigned `
NaN
` value).
data (Thus get assigned `
NaN
` value).
The original data will be dropped (but can be regained by function
The original data will be dropped (but can be regained by function
`
deharmonize
`).
`
deharmonize
`).
4. Depending on the keyword passed to agg_flag_func, the original flags get
4. Depending on the keyword passed to
`
agg_flag_func
`
, the original flags get
aggregated and assigned onto the new, harmonized data, generated in step (3).
aggregated and assigned onto the new, harmonized data, generated in step (3).
New sampling intervals, covering no data in the original dataseries or only
New sampling intervals, covering no data in the original dataseries or only
data that got excluded in step (1), will be regarded as representing missing
data that got excluded in step (1), will be regarded as representing missing
...
@@ -883,13 +883,115 @@ harmonize_linear2Grid(freq, flag_assignment_method='nearest_agg', flag_agg_func=
...
@@ -883,13 +883,115 @@ harmonize_linear2Grid(freq, flag_assignment_method='nearest_agg', flag_agg_func=
```
```
| parameter | data type | default value | description |
| parameter | data type | default value | description |
| --------- | --------- | ------------- | ----------- |
| --------- | --------- | ------------- | ----------- |
| freq | string | | Offset string. Determining the sampling rate of the frequency grid, the data shall be
shif
ted t
o
.|
| freq | string | | Offset string. Determining the sampling rate of the frequency grid, the data shall be
interpola
ted
a
t.|
| flag_assignment_method| string | "nearest_agg" |
Function. Function
used for
data
aggregation.|
| flag_assignment_method| string | "nearest_agg" |
Method keyword, signifying method
used for
flags
aggregation.
See step 4 and table below
|
| flag_agg_func | func | max | Function used for flags aggregation.|
| flag_agg_func | func | max | Function used for flags aggregation.|
| drop_flags | list or Nonetype | None | Flags to be excluded from harmonization. See description of step 2 below. |
| drop_flags | list or Nonetype | None | Flags to be excluded from harmonization. See description of step 2 below. |
Linear interpolation of an inserted equidistant frequency grid of sampling rate `
freq
`.
1. All missing values in the data, identified by `
np.nan
`,
get flagged and will be excluded from the aggregation process.
NOTE, that implicitly this step includes a call to `
missing
` onto the
data-to-be-flagged.
2. Additionally, if a list is passed to `
drop_flags
`, all the values in data,
that are flagged with a flag, listed in `
drop_list
`, will be excluded from
interpolation - meaning, that they will not affect the further
aggregation prozess.
3. Data interpolation gets carried out: since the function is a harmonization function, the interpolation will not fill
gaps in your timeseries, but only calculate an interpolation value for grid points, that are surrounded by
valid values within `
freq
` range. If there is either no valid value to the right, or to the left of a new grid point,
that new grid point gets assigned `
np.nan
` (missing.)
4. Depending on the keyword passed to `
flag_assignment_method
`, the original flags get
shifted, or aggregated with `
flag_agg_func
` onto the new, harmonized data index, generated in step (3).
New sampling intervals, covering no data in the original dataseries or only
data that got excluded in step (1), will be regarded as representing missing
data and thus get assigned the worst flag level available.
`
flag_assignment_method
` - Keywords
1. Shifts:
* `
"fshift"
`: every grid point gets assigned its ultimately preceeding flag
if there is one available in the preceeding sampling interval. If not, BAD - flag gets assigned.
* `
"bshift"
`: every grid point gets assigned its first succeeding flag
if there is one available in the succeeding sampling interval. If not, BAD - flag gets assigned.
* `
"nearest_shift"
`: every grid point gets assigned the flag in its range. ( range = +/- `
freq
`/2 ).
* Extra flag fields like "comment", just get shifted along with the flag.
Only inserted flags for empty intervals will get signified by the set flag routine of the current flagger.
Set `
set_shift_comment
` to `
True
`, to apply setFlags signification to all flags.
2. Aggregations:
* `
"fagg"
`: all flags in a sampling interval get aggregated with the function passed to `
agg_func
`
, and the result gets assigned to the last grid point.
* `
"bagg"
`: all flags in a sampling interval get aggregated with the function passed to `
agg_func
`
, and the result gets assigned to the next grid point.
* `
"nearest_agg"
`: all flags in the range (+/- freq/2) of a grid point get
aggregated with the function passed to `
agg_func
` and assigned to it.
## harmonize_interpolate2Grid
```
harmonize_interpolate2Grid(freq, interpolation_method, interpolation_order=1, flag_assignment_method='nearest_agg',
flag_agg_func=max, drop_flags=None)
```
| parameter | data type | default value | description |
| --------- | --------- | ------------- | ----------- |
| freq | string | | Offset string. Determining the sampling rate of the frequency grid, the data shall be interpolated at.|
| interpolation_method | string | | Method keyword, signifying method used for grid interpolation. See step 3 and table below|
| interpolation_order | func | 1 | If needed - order of the interpolation, carried out.|
| flag_assignment_method| string | "nearest_agg" | Method keyword, signifying method used for flags aggregation. See step 4 and table below|
| flag_agg_func | func | max | Function used for flags aggregation.|
| drop_flags | list or Nonetype | None | Flags to be excluded from harmonization. See description of step 2 below. |
Interpolation of an inserted equidistant frequency grid of sampling rate `
freq
`.
1. All missing values in the data, identified by `
np.nan
`,
get flagged and will be excluded from the aggregation process.
NOTE, that implicitly this step includes a call to `
missing
` onto the
data-to-be-flagged.
2. Additionally, if a list is passed to `
drop_flags
`, all the values in data,
that are flagged with a flag, listed in `
drop_list
`, will be excluded from
interpolation - meaning, that they will not affect the further
aggregation prozess.
3. Data interpolation with `
interpolation_method
` gets carried out: since the function is a harmonization function, the interpolation will not fill
gaps in your timeseries, but only calculate an interpolation value for grid points, that are surrounded by
valid values within `
freq
` range. If there is either no valid value to the right, or to the left of a new grid point,
that new grid point gets assigned `
np.nan
` (missing.)
4. Depending on the keyword passed to `
flag_assignment_method
`, the original flags get
shifted, or aggregated with `
flag_agg_func
` onto the new, harmonized data index, generated in step (3).
New sampling intervals, covering no data in the original dataseries or only
data that got excluded in step (1), will be regarded as representing missing
data and thus get assigned the worst flag level available.
`
interpolation_method
` - Keywords:
* There are available all the interpolation methods from the pandas.interpolate() method and they can be reffered to with
the very same keywords, that you would pass to pd.Series.interpolates's method parameter.
* Available interpolations: `
"linear"
`, `
"time"
`, `
"nearest"
`, `
"zero"
`, `
"slinear"
`,
`
"quadratic"
`, `
"cubic"
`, `
"spline"
`, `
"barycentric"
`, `
"polynomial"
`, `
"krogh"
`,
`
"piecewise_polynomial"
`, `
"spline"
`, `
"pchip"
`, `
"akima"
`.
* Be careful with pd.Series.interpolate's `
"nearest"
` and `
"pad"
`:
To just fill grid points forward/backward or from the nearest point - and
assign grid points, that refer to missing data, a nan value, the use of `
"fshift"
`, `
"bshift"
` and `
"nearest_shift"
` is
recommended, to ensure getting the result expected. (The methods diverge in some
special cases and do not properly interpolate grid-only.).
`
flag_assignment_method
` - Keywords
1. Shifts:
* `
"fshift"
`: every grid point gets assigned its ultimately preceeding flag
if there is one available in the preceeding sampling interval. If not, BAD - flag gets assigned.
* `
"bshift"
`: every grid point gets assigned its first succeeding flag
if there is one available in the succeeding sampling interval. If not, BAD - flag gets assigned.
* `
"nearest_shift"
`: every grid point gets assigned the flag in its range. ( range = +/- `
freq
`/2 ).
* Extra flag fields like "comment", just get shifted along with the flag.
Only inserted flags for empty intervals will get signified by the set flag routine of the current flagger.
Set `
set_shift_comment
` to `
True
`, to apply setFlags signification to all flags.
2. Aggregations:
* `
"fagg"
`: all flags in a sampling interval get aggregated with the function passed to `
agg_func
`
, and the result gets assigned to the last grid point.
* `
"bagg"
`: all flags in a sampling interval get aggregated with the function passed to `
agg_func
`
, and the result gets assigned to the next grid point.
* `
"nearest_agg"
`: all flags in the range (+/- freq/2) of a grid point get
aggregated with the function passed to `
agg_func
`
and assigned to it.
\ No newline at end of file
This diff is collapsed.
Click to expand it.
Preview
0%
Loading
Try again
or
attach a new file
.
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Save comment
Cancel
Please
register
or
sign in
to comment