Skip to content

reorganize concatflags and resampling

Peter Lünenschloß requested to merge reindexFuncReordering into develop

adresses #409 (closed)

  • establish reindexing base function reindex that maps flags and/or data between indexes and is wrapped around by the resampling and alignment library functions
  • split concatFlags up into reindexing function (reindex) and flags appending (transferFlags) function
  • resample and align now are bare wrapper around reindex
  • deprecate the inverse_ + method notions in favor of boolean parameter invert_method (as suggested in #409 (closed) )

usage examples:

import pandas as pd
import saqc
import numpy as np
from saqc.constants import FILTER_NONE
# ------------- generate data with messed up timestamps -------------------------
np.random.seed(23)
idx = pd.DatetimeIndex(pd.date_range('2000', freq='1d', periods=23))
idx += pd.Index([pd.Timedelta(f'{k}min') for k in np.random.randint(-360,360,23)])
drops = np.random.randint(0,20,3)
drops.sort()
idx=idx[np.r_[0:drops[0],drops[0]+1:drops[1],drops[1]+1:drops[2],drops[2]+1:23]]
data = pd.Series(np.abs(np.arange(-10,10)), index=idx, name='data')

# ------------ linear alignment/harm to 2d grid and "deharm" ------------------------------
qc = saqc.SaQC(data)
qc = qc.reindex('data', target='linear', idx='2D', index_selection_method='fromLastNext', data_aggregation_method='linear')
# flagging:
qc = qc.setFlags('linear', data=['2000-01-16'])
# "deharm"
qc = qc.reindex('linear', idx='data', reindex_window='2D', index_selection_method='toLastNext', dfilter=FILTER_NONE)
qc = qc.transferFlags('linear', 'data')

# -------------- shift harmonization and "deharm"-shift ---------------
qc = saqc.SaQC(data)
# "harm"
qc = qc.reindex('data', idx='1D', target='shifted_data', index_selection_method='nshift')
# "deharm"
qc = qc.reindex('shifted_data', idx='data', index_selection_method='nshift')
qc = qc.transferFlags('shifted_data', 'data')

# -------------- Forward backward aggregation ------------------------
qc = saqc.SaQC(data)
# "harm"
qc = qc.reindex('data', target='sum_aggregate', idx='3D', index_selection_method='fagg', data_aggregation_method='sum')
# set some flags
qc = qc.setFlags('sum_aggregate', data=['2000-01-18', '2000-01-24'])
# different "deharms":
qc = qc.reindex('sum_aggregate', target='bagg', idx='data', index_selection_method='bagg', dfilter=FILTER_NONE)
qc = qc.reindex('sum_aggregate', target='bagg_limited', idx='data', index_selection_method='bagg', reindex_window='2D', dfilter=FILTER_NONE)

Reindexing options can be a bit overwhelming/hard to discern. I added this chart to the doc string for an overview. Maybe that also helps with the review:reindexMethods

Edited by Peter Lünenschloß

Merge request reports