Several fixes

A container branch for the fixes of several issues I encountered while rolling out the current SaQC on the SEEFO pipeline.

Not sure, if this needs a real review, but as some of the changes touched code I am not completely familiar with, maybe you, @palmb and @luenensc want to have a look anyhow?

added 3 commits

• b2137c2a - [FIX] excess history column for new variables through generic.flag
• 734621f7 - [FIX] outliers.flagOffset raises IndexError for all nan-data
• e880fa7d - [FIX] remove real-world memory usage by ~20% and runtime by ~3% through removal of Flags._cache

Compare with previous version

added 1 commit

• 0ffd8c04 - [FIX] reduce runtime by ~45% (!) through the removal of builtins.any

Compare with previous version

Found nothing controversial

added 1 commit

• 9e1d6d25 - [FIX] reduce the memory consumption of SaQC by >50% through Histories of type pd.Categorical

Compare with previous version

added 1 commit

• ea4d0b06 - [FIX] reduce the memory consumption of SaQC by >50% through Histories of type pd.Categorical

Compare with previous version

AFAIK, astype returns always a copy, which break copy=False.

True, that's why I also had to tweak the respective test...So with that, there are no real shallow copies anymore

changed this line in version 6 of the diff

So maybe one more general remark about the somewhat random collections of commits within in this MR:

All changes are fixes to issues I had when porting the SEEFO-pipeline to the latest develop (through !260 (merged)). Every commit is a fix to one issue I encountered (I tried hard to not intermingle changes) in chronological order. With exception of 1da172e8, which corrects faulty behavior without breakage, all changes fix an issue hardly breaking the formerly running pipeline. So, I had to come up with some sort of solutions to the underlying problems in order to get things into their former state of working. Are the proposed changed the only solution to the underlying problems? Likely not! Are these changed the best solution to the underlying problems? Probably also not...

Not sure, if this comments helps or not, I just wanted to make clear, that there are no simple refactorings incorporated. I chose the commit message prefix [FIX] as, from the point of view of the SEEFO-pipeline, these changes are fixing regressions.

With that out of the way, I am happy to discuss any objections/suggestions on any suitable channel. Cheers!

added 1 commit

• 686065dc - [FIX] reduce the memory consumption of SaQC by >50% through Histories of type pd.Categorical

Compare with previous version

mentioned in merge request !260 (merged)

Closing as all commits, will go in through !260 (merged)

closed

From some brief benchmarking, i got, that casting via df.astype(pd.SparseDtype('float', np.nan)), instead of df.astype('category'), is faster in casting (around 30 percent) and uses less memory (Factor 1-10, without initial unflagged column: factor 2-20) and is faster in column and row access and also in row wise max calculation.

So, since integrating would just mean to replace category cast by sparse cast, maybe we should give it a try?

Sure! But let's please do it after !260 (merged) was merged.

!260 (merged) is in now, so feel free to sparsify.