Follow-up from "New flagged value masking behaviour"
The following discussion from !45 (closed) should be addressed:
-
@schaefed started a discussion: (+3 comments) I am not entirely convinced, that moving the crucial part of evaluation into the module
funcs
makes it easier or more obvious to find. But the idea to make flags-to-mask a user settable options sort of requires, that the masking is part of the function wrapping.So, long story short, I think, that
register.py
should eventually move into thecore
, as it is going to become an integral part of the evaluation machinery.On more thing:
getMask
in snippet $49 is essentially aflagger.isFlagged
on steroids, so I really think, we should implement the option to pass a (possibly empty) sequence of flags toflagger.isFlagged
and reduce the snippet to:from lib.tools import toSequence def register(): def outer(func): name = func.__name__ func = Partial(func, func_name=name) FUNC_MAP[name] = func def inner(data, field, flagger, *args, **kwargs): # NOTE: would be nice to make to_mask a 'real' argument to_mask = toSequence(kwargs.get('to_mask', flagger.BAD)) data_in = maskData(data, flagger, to_mask) data_res, flagger_res = func(data_in, field, flagger, *args, **kwargs) data_out = unmaskData(data, flagger, data_res, flagger_res, to_mask) return data_out, flagger_res return inner return outer def maskData(data, flagger, to_mask): data_masked = data.copy() data_masked[flagger.isFlagged(flag=to_mask)] = np.nan return data_masked def unmaskData(data_old, flagger_old, data_new, flagger_new, to_mask): mask_old = flagger_old.isFlagged(flag=to_mask) mask_new = flagger_new.isFlagged(flag=to_mask) mask_combined = DictOfSeries() for col, left in data_new.indexes.iteritems(): right = mask_old[col].index # NOTE: ignore columns with changed indices (assumption: harmonization) if left.equals(right): # NOTE: don't overwrite flags that where unset within the called function mask_combined[col] = mask_old[col] & mask_new[col] data_new.aloc[mask_combined] = data_old[mask_combined] return data_new