SaQC Operators

So this is a suggestive MR/Issue for the upcoming SaQC-future discussion. Figured with __getitem__, __setitem__ inplace, there are a lot of low hanging fruits available, that simplify saqc operations. Namely, to add boolean and arithmetic operators to replace generic-lambda and andGroup-orGroup combos with easily readable/communicable operator counterparts.

So !857 adds the common boolean and arithmetic operators to saqc. This is far from optimally implemented and just meant to serve as basis for discussion/try-out.

The MR currently comprises:

• logical operators:

  • & -> qc['foo'] & qc['bar']: returns SaQC object with field foo and flags min(qc['foo'].flags, qc['bar'].flags)
  • | -> qc['foo'] & qc['bar']: returns SaQC object with field foo and flags max(qc['foo'].flags, qc['bar'].flags)
  • ~ -> shortcut for squeeze

• arithmetic operators:

  • + -> qc['foo'] + qc['bar']: returns SaQC object with field foo, data qc['foo'].data + qc['bar'].data and flags min(qc['foo'].flags, qc['bar'].flags)
  • - -> qc['foo'] - qc['bar']: returns SaQC object with field foo, data qc['foo'].data - qc['bar'].data and flags min(qc['foo'].flags, qc['bar'].flags)
  • * -> qc['foo'] + qc['bar']: returns SaQC object with field foo, data qc['foo'].data * qc['bar'].data and flags min(qc['foo'].flags, qc['bar'].flags)
  • + f, - f, * f, ** f -> for example: qc['foo'] + 7: returns SaQC object with field foo, data qc['foo'].data + 7 and flags qc['foo'].flags

• history (append) operators

  • << -> qc['foo'] << qc['bar']: returns SaQC object with field 'foo' data from 'foo' and history resulting from 'bar' history appended to 'foo'
  • ^ -> qc['foo'] ^ qc['bar']: returns SaQC object with field 'foo' data from 'bar' and history resulting from 'bar' history appended to 'foo'

• magical inplacy counterparts:

  • <<= -> qc['foo'] <<= qc['bar']: shorthand/inplace for qc['foo'] = qc['foo'] << qc['bar']
  • &= -> qc['foo'] &= qc['bar']: shorthand/inplace for qc['foo'] = qc['foo'] << (qc['foo'] & qc['bar'])
  • += -> qc['foo'] += qc['bar']: shorthand/inplace for qc['foo'] = qc['foo'] ^ (qc['foo'] + qc['bar'])
  • ...

• All operators also work for multifield SaQC objects - but necessitate that all the objects have the same number of fields and will than just loop over those fields pair wise and return the operator result for any two fields in a new SaQC object. (The fields are just matched by order, not by name).

• The arithmetic operators already support operating with pd.DataFrame and np.ndarray objects. (but necessitate them to have equaling size: so no broadcasting, except for scalars.)

The implementation is made so, that consecutive operator calls wont blow up the history. So the calls:

qc['data0'] &= (qc['data1'] & qc.flagBar('data2')['data2']) | qc2['voltage']
qc['data1'] += 7*qc['data0']

Will only add one new history column to field 'data0' (and 'data1'). Also, the inplace operators (<<=, &=, ...) actually work in conjunction with the implemented __setitem__ and __getitem__ operators. So above code actually results in qc holding the results from the operations.

Some sandbox code:

import saqc
import pandas as pd
from saqc.lib.tools import periodicMask

mask_func = lambda x: ~periodicMask(x.index, '6:00:00', '18:00:00', True)
dat0 = pd.Series(range(100), name='data0', index=pd.date_range('2000', freq='4h', periods=100))
dat1 = pd.Series(0, name='data1', index=pd.date_range('2000', freq='4h', periods=100))
dat2 = pd.Series(range(-50,50), name='data2', index=pd.date_range('2000', freq='4h', periods=100))
scheme = saqc.DmpScheme()
qc = saqc.SaQC([dat0,dat1,dat2], scheme=scheme)
qc['data0'] += 1000.0
qc['data0'] <<= qc['data0'] & (qc['data2'].flagGeneric('data2', func=mask_func)

This for example results in the meta:

qc._history['data0'].meta

{'func': '(data0 + 1000.0)',
  'args': (),
  'kwargs': {'operand_0': {'dfilter': -inf, 'field': 'data0'},
   'operand_1': {'value': '1000.0',
    'type': "<class 'float'>",
    'squeezed': True}}},
 {'func': '(|(data0 + 1000.0)| & |data2|)',
  'args': (),
  'kwargs': {'operand_0': {'dfilter': -inf,
    'field': 'data0',
    'squeezed': True},
   'operand_1': {'value': '1000.0',
    'type': "<class 'float'>",
    'squeezed': True},
   'operand_2': {'squeezed': True}}}]