flagCrossStatistics dont work as expected
the latest changes to flagCrossStatictics
changed the behavior a lot !
before the change it was able to flag all columns that was given even if they have different indexes. this was the charm of it.
Now it enforce harmonized index and also fails [1] during checking this :(
i somewhat rely on the function in GECF..
[1]
Stacktrace (Click to expand)
File "scripts/qc_cross_P.py", line 43, in process
qc = qc.flagCrossStatistics(field=hasroof, thresh=3.5, flag=f.DOUBT)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/saqc/core/modules/outliers.py", line 114, in flagCrossStatistics
return self._defer("flagCrossStatistics", locals())
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/saqc/core/modules/__init__.py", line 45, in _defer
return self._wrap(FUNC_MAP[fname])(**flocals, **fkwargs)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/saqc/core/core.py", line 215, in inner
out = out._callFunction(
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/saqc/core/core.py", line 242, in _callFunction
res = function(data=self._data, flags=self._flags, field=field, *args, **kwargs)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/saqc/core/register.py", line 155, in __call__
data, flags = self.func(*args, **kwargs)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/saqc/funcs/outliers.py", line 1268, in flagCrossStatistics
if (data[src].index != data[fields[0]].index).any():
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/pandas/core/indexes/extension.py", line 156, in wrapper
return op(other)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/pandas/core/ops/common.py", line 69, in new_method
return method(self, other)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/pandas/core/arraylike.py", line 36, in __ne__
return self._cmp_method(other, operator.ne)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 1008, in _cmp_method
other = self._validate_comparison_value(other)
File "/home/palmb/.virtualenvs/gcef_pipeline/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 549, in _validate_comparison_value
raise ValueError("Lengths must match")
ValueError: Lengths must match