set flagged values to nan?
We currently replace all flagged values by np.nan
before they are passed to the test functions. Is this a good idea?
A little more context:
This was necessary to propagate flags through the evaluator. The idea was that a call like:
flagGeneric(func=this < var1 + var2)
would return an aggregate of the result of the <
condition and the flags of var1
and var2
, i.e. this < var1 + var2
can only be True
, where var1
and var2
are unflagged.
This restriction could be lifted with a function in evaluator.py
like:
def _dslAggregateFunc(func, flagger, data):
flags = flagger.isFlagged(data.name)
return func(data.mask(flags))
and the following changes to initLocalEnv
:
return {
...
"mean": partial(_dslAggregateFunc, np.nanmean, flagger),
"sum": partial(_dslAggregateFunc, np.nansum, flagger),
"std": partial(_dslAggregateFunc, np.nanstd, flagger),
...
}
Edited by David Schäfer