Follow-up from "New flagged value masking behaviour"

The following discussion from !45 (closed) should be addressed:

  • @schaefed started a discussion: (+3 comments)

    I am not entirely convinced, that moving the crucial part of evaluation into the module funcs makes it easier or more obvious to find. But the idea to make flags-to-mask a user settable options sort of requires, that the masking is part of the function wrapping.

    So, long story short, I think, that register.py should eventually move into the core, as it is going to become an integral part of the evaluation machinery.

    On more thing: getMask in snippet $49 is essentially a flagger.isFlagged on steroids, so I really think, we should implement the option to pass a (possibly empty) sequence of flags to flagger.isFlagged and reduce the snippet to:

    from lib.tools import toSequence
    
    def register():
        def outer(func):
            name = func.__name__
            func = Partial(func, func_name=name)
            FUNC_MAP[name] = func
    
            def inner(data, field, flagger, *args, **kwargs):
                # NOTE: would be nice to make to_mask a 'real' argument
                to_mask = toSequence(kwargs.get('to_mask', flagger.BAD))
                
                data_in = maskData(data, flagger, to_mask)
                data_res, flagger_res = func(data_in, field, flagger, *args, **kwargs)
                data_out = unmaskData(data, flagger, data_res, flagger_res, to_mask)
    
                return data_out, flagger_res
    
            return inner
    
        return outer
    
    
    def maskData(data, flagger, to_mask):
        data_masked = data.copy()
        data_masked[flagger.isFlagged(flag=to_mask)] = np.nan
        return data_masked
    
    
    def unmaskData(data_old, flagger_old, data_new, flagger_new, to_mask):
        mask_old = flagger_old.isFlagged(flag=to_mask)
        mask_new = flagger_new.isFlagged(flag=to_mask)
        mask_combined = DictOfSeries()
        for col, left in data_new.indexes.iteritems():
            right = mask_old[col].index
            # NOTE: ignore columns with changed indices (assumption: harmonization)
            if left.equals(right):
                # NOTE: don't overwrite flags that where unset within the called function
                mask_combined[col] = mask_old[col] & mask_new[col]
        data_new.aloc[mask_combined] = data_old[mask_combined]
        return data_new