Memory Leak

The following discussion from !465 (merged) should be addressed:

• @palmb started a discussion: (+8 comments)

  Anyways the increase in memory usage seems to be acceptable: With data of size (10, 5_000_000) and a single test on every column increases the memory usage from ~1_730_000 to ~ 2_094_000 in steps of ~20_000 bytes per function call.

  can you provide how you test test the memory..

added BUG category: dios info / benchmark labels

assigned to @palmb

I guess we still have, because our memory increases come from on data alone! Which is not what I would expect...Anyhow, for the moment, it should work as is...

i think i dont understand what you mean. can you explain it to me again 🙏 , best with code snippets.

Running the attached script against develop outputs the following for me:

1741756
1741756
1969168
1988700
2008232
2027764
2047296
2066824
2086356
2104832

Applying the following diff and running the attached script again, however, gives a constant peak memory usage of 1742724 in every iteration.

diff --git a/saqc/core/core.py b/saqc/core/core.py
index 4830b177..d34c25e5 100644
--- a/saqc/core/core.py
+++ b/saqc/core/core.py
@@ -264,7 +264,8 @@ class SaQC(FunctionsMixin):
     def copy(self, deep=True):
         copyfunc = deepcopy if deep else shallowcopy
         new = self._construct()
-        for attr in self._attributes:
+        setattr(new, "_data", shallowcopy(getattr(self, "_data")))
+        for attr in {"_flags", "_scheme", "_attrs"}:
             setattr(new, attr, copyfunc(getattr(self, attr)))
         return new

We see an increase in peak memory usage in every iteration, even though, data is not modified at all when running test_memory.py. So for whatever reason deep copying data leaves some unfreed memory behind.

test_memory.py

hmm, i can reproduce it, but i'm somewhat confused that the peak memory is constant, because at least one flags-column is generated per call..

hmm, i can reproduce it,

What is it you can't reproduce? The leak or the constant memory situation?

but i'm somewhat confused that the peak memory is constant, because at least one flags-column is generated per call..

On first sight, I was too, but we are using pd.Categorical, which should have a rather small foot print.

What is it you can't reproduce? The leak or the constant memory situation?

no, i can reproduce it (the leak and the constant memory situation)

Sorry, misread your answer...

locked this issue

unlocked this issue

added needs-investigation label

changed milestone to %Someday

The leak still exists on 4045187d

changed title from Follow-up from "Make SaQC immutable again" to Memory Leak