optimization to flag basic / implementation of a custom rolling wrapper
- As discussed, spikes_flagBasic had to be optimized.
- a factor 10 improvement comes from rewriting the inner check function and truncating the timeseries of spike candidates
- another factor 10 improvement comes from using a custom - rolling implementation that skips function application for a selection of windows (Yeah)
- (numba boosting the rolling application turned out to slow down performance for less then 200.000 spike candidates, so no performance gain from there)
- i can tidy up my benchmark script and post it, if demanded
- New custom rolling implementation
- a wrapper around pandas.rolling.apply, that allows for skipping function application for an arbitrary selection of windows by instantiating a custom pandas.BaseIndexer.
-
@schaefed : i didnt benchmark this implementation against the existing
slidingWindowIndices
implementation, but here are some reasons for maybe deprecatingslidingWindowIndices
:
- the new implementation passes window-index selection to a pandas - ".pyx" function, as i understand it, that files implement C-functions that are quite fast in comparison to uhm - non-C implementations....
- i needed arbitrary skipping of windows, which was not possible with the slidingIndices implementation that only allows for constant skipping increments (when i got it right).
- The difference between freq and periods defined windows is confusing (as pointed out in the slidingWindowIndices documentation), but can actually be controlled and neutralized by the pandas.rolling
min_periods
parameter (took me some time to notice as well, because the default (None) results in different behavior of freq and periods defined rolling windows).