SaQC merge requestshttps://git.ufz.de/rdm-software/saqc/-/merge_requests2024-03-20T18:33:17+01:00https://git.ufz.de/rdm-software/saqc/-/merge_requests/803added python 3.12 tests2024-03-20T18:33:17+01:00David Schäferdavid.schaefer@ufz.deadded python 3.12 tests2.6.0David Schäferdavid.schaefer@ufz.deDavid Schäferdavid.schaefer@ufz.dehttps://git.ufz.de/rdm-software/saqc/-/merge_requests/723Config class2023-12-06T14:23:03+01:00Bert Palmbert.palm@ufz.deConfig classabstraction for config:
- a config class that is a collection of tests
- a test that can be parsed and runabstraction for config:
- a config class that is a collection of tests
- a test that can be parsed and runBert Palmbert.palm@ufz.deBert Palmbert.palm@ufz.dehttps://git.ufz.de/rdm-software/saqc/-/merge_requests/693Draft: allow pandas functions in text configurations2023-07-24T13:54:02+02:00David Schäferdavid.schaefer@ufz.deDraft: allow pandas functions in text configurationsThis MR exposes pandas to the text configuration and allows, for example, things like:
```
var ; flagGeneric(func=pd.isna(var))
```This MR exposes pandas to the text configuration and allows, for example, things like:
```
var ; flagGeneric(func=pd.isna(var))
```2.5.0David Schäferdavid.schaefer@ufz.deDavid Schäferdavid.schaefer@ufz.dehttps://git.ufz.de/rdm-software/saqc/-/merge_requests/805Draft: WIP: pydantic2024-02-26T08:05:34+01:00Peter LünenschloßDraft: WIP: pydanticpydantic draft/try out.
see #457pydantic draft/try out.
see #457https://git.ufz.de/rdm-software/saqc/-/merge_requests/789make field optional2024-01-24T15:30:18+01:00Bert Palmbert.palm@ufz.demake field optional- rm field as first positional argument
- added fallback option (default on) to use the legacy call style
- and throw warning if fallback active
:warning: sits ontop of !787- rm field as first positional argument
- added fallback option (default on) to use the legacy call style
- and throw warning if fallback active
:warning: sits ontop of !787Bert Palmbert.palm@ufz.deBert Palmbert.palm@ufz.dehttps://git.ufz.de/rdm-software/saqc/-/merge_requests/797reduce the number of type checking errors2024-01-31T11:42:43+01:00David Schäferdavid.schaefer@ufz.dereduce the number of type checking errors2.6.0David Schäferdavid.schaefer@ufz.deDavid Schäferdavid.schaefer@ufz.dehttps://git.ufz.de/rdm-software/saqc/-/merge_requests/825reorganize concatflags and resampling2024-03-28T19:12:50+01:00Peter Lünenschloßreorganize concatflags and resamplingadresses #409
* establish *reindexing* base function *reindex* that maps flags and/or data between indexes and is wrapped around by the resampling and alignment library functions
* split `concatFlags` up into reindexing function (`rein...adresses #409
* establish *reindexing* base function *reindex* that maps flags and/or data between indexes and is wrapped around by the resampling and alignment library functions
* split `concatFlags` up into reindexing function (`reindex`) and flags appending (`transferFlags`) function
* `resample` and `align` now are bare wrapper around `reindex`
* deprecate the `inverse_` + method notions in favor of boolean parameter `invert_method` (as suggested in #409 )
usage examples:
```python
import pandas as pd
import saqc
import numpy as np
from saqc.constants import FILTER_NONE
# ------------- generate data with messed up timestamps -------------------------
np.random.seed(23)
idx = pd.DatetimeIndex(pd.date_range('2000', freq='1d', periods=23))
idx += pd.Index([pd.Timedelta(f'{k}min') for k in np.random.randint(-360,360,23)])
drops = np.random.randint(0,20,3)
drops.sort()
idx=idx[np.r_[0:drops[0],drops[0]+1:drops[1],drops[1]+1:drops[2],drops[2]+1:23]]
data = pd.Series(np.abs(np.arange(-10,10)), index=idx, name='data')
# ------------ linear alignment/harm to 2d grid and "deharm" ------------------------------
qc = saqc.SaQC(data)
qc = qc.reindex('data', target='linear', idx='2D', index_selection_method='fromLastNext', data_aggregation_method='linear')
# flagging:
qc = qc.setFlags('linear', data=['2000-01-16'])
# "deharm"
qc = qc.reindex('linear', idx='data', reindex_window='2D', index_selection_method='toLastNext', dfilter=FILTER_NONE)
qc = qc.transferFlags('linear', 'data')
# -------------- shift harmonization and "deharm"-shift ---------------
qc = saqc.SaQC(data)
# "harm"
qc = qc.reindex('data', idx='1D', target='shifted_data', index_selection_method='nshift')
# "deharm"
qc = qc.reindex('shifted_data', idx='data', index_selection_method='nshift')
qc = qc.transferFlags('shifted_data', 'data')
# -------------- Forward backward aggregation ------------------------
qc = saqc.SaQC(data)
# "harm"
qc = qc.reindex('data', target='sum_aggregate', idx='3D', index_selection_method='fagg', data_aggregation_method='sum')
# set some flags
qc = qc.setFlags('sum_aggregate', data=['2000-01-18', '2000-01-24'])
# different "deharms":
qc = qc.reindex('sum_aggregate', target='bagg', idx='data', index_selection_method='bagg', dfilter=FILTER_NONE)
qc = qc.reindex('sum_aggregate', target='bagg_limited', idx='data', index_selection_method='bagg', reindex_window='2D', dfilter=FILTER_NONE)
```
Reindexing options can be a bit overwhelming/hard to discern. I added this chart to the doc string for an overview. Maybe that also helps with the review:![reindexMethods](/uploads/23abfa086977df2acfc29d45a5388b5d/reindexMethods.png)Peter LünenschloßPeter Lünenschloßhttps://git.ufz.de/rdm-software/saqc/-/merge_requests/827Represent all flags by DictOfSeries2024-03-12T11:25:19+01:00David Schäferdavid.schaefer@ufz.deRepresent all flags by DictOfSeriesThis MR makes our `DictOfSeries` capable of holding `pd.DataFrame`s as well an introduces its usage for all translators. This can only be merged after https://github.com/palmb/fancy-collections/pull/2 is in and the `fancy_collections` is...This MR makes our `DictOfSeries` capable of holding `pd.DataFrame`s as well an introduces its usage for all translators. This can only be merged after https://github.com/palmb/fancy-collections/pull/2 is in and the `fancy_collections` is bumped to the to-be-released version.David Schäferdavid.schaefer@ufz.deDavid Schäferdavid.schaefer@ufz.de