Skip to content

optimization to flag basic / implementation of a custom rolling wrapper

Peter Lünenschloß requested to merge flagBasicOpt into develop
  1. As discussed, spikes_flagBasic had to be optimized.
  • a factor 10 improvement comes from rewriting the inner check function and truncating the timeseries of spike candidates
  • another factor 10 improvement comes from using a custom - rolling implementation that skips function application for a selection of windows (Yeah)
  • (numba boosting the rolling application turned out to slow down performance for less then 200.000 spike candidates, so no performance gain from there)
  • i can tidy up my benchmark script and post it, if demanded
  1. New custom rolling implementation
  • a wrapper around pandas.rolling.apply, that allows for skipping function application for an arbitrary selection of windows by instantiating a custom pandas.BaseIndexer.
  • @schaefed : i didnt benchmark this implementation against the existing slidingWindowIndices implementation, but here are some reasons for maybe deprecating slidingWindowIndices:
  1. the new implementation passes window-index selection to a pandas - ".pyx" function, as i understand it, that files implement C-functions that are quite fast in comparison to uhm - non-C implementations....
  2. i needed arbitrary skipping of windows, which was not possible with the slidingIndices implementation that only allows for constant skipping increments (when i got it right).
  3. The difference between freq and periods defined windows is confusing (as pointed out in the slidingWindowIndices documentation), but can actually be controlled and neutralized by the pandas.rolling min_periods parameter (took me some time to notice as well, because the default (None) results in different behavior of freq and periods defined rolling windows).

Merge request reports