Dios integration

@schaefed @luenensc

Dios is integrated, nearly all tests run again. So far i only implemented the Baseflagger interface. i merged develop into dios_integration, so that now the merge in the other direction should work without further problems.

We should do this soon, because new commits on develop need new adjustments afterwards (kick Dataframe for DictOfSeries..., fix use of df.index, and so on so on)

some test still dont run, but i find them much to complex/abstract to understand and fix them quick, ples check this.

cheerio B

added internal architecture urgent + 1 deleted label

probably, we should clean up a bit, eg. integrate dios as a module, kick unused files, stuff and crap, but for now this should work, that everyone get in touch and in use with dios...

also its quite handy for now to have dios as a submodule, as long as we need fixes there. Its quite a hack to do this with a pyPI package..

added 3 commits

• 0cfbe0fd - fixing the derivation of dates from dios
• 5546ba54 - fixing missing import
• 787ae1a4 - added dios to the ci config

Compare with previous version

added 2 commits

• 0ff6fb79 - fixed failing test
• 51f782cc - fixing the failing ML test by doing to many dios -> pandas

Compare with previous version

I fixed what I could. The only thing remaining is the harmonization, but I guess @luenensc needs to lead that integeration.

i guess that harmonisation need a whole rework now, may we just merge this and disable harmo for now?

also i suggest: merge develop into dios_integration, then dios_integration in develop.

I would like to wait for a comment of @luenensc before merging. The harmonization is such an integral part of many (most?) usage patterns, that I am not sure SaQC would be as usable without it.

If this turns out to be a mayor undertaking, than maybe we could try to incorporate the harmonization as is (side-by-side with the new stuff) and gradually adapt it to dios as we see fit and free time slots.

Hallo - yes, the harmonization needs an actual rework now - but this is more concerned with getting rid of parts that are now obsolet due to the new features of dios - Since, while doing so, i would as well like to change some other shortcomings of the harm module - mainly regarding performance - i guess this reworking would be a longer, lets say, one week+ project, that i would like to undertake after the current implementation of the oddWater algorithm, that i am concerned with right now - so it wont be tomorrow or this week.

But @palmb - by just going through the code and replacing dataframe constructions by the corresponding dios constructions - like you did in the rest of saqc i guess - it should just run like before. I guess it could be some kind of annoying to have an old version of saqc, were harmonization is available and a newer one, were it isnt.

but maybe there wont be any gcef parametrization meetings the upcoming weeks, anyway - but on the other hand, i need harmonization for the oddwater algorithm, since it is implicitly part of it. so there would be growing divergence....

I agree with @luenensc , we should not try to maintain (and develop) two different versions of SaQC in parallel (one with harmonization but without dios and the other with without harmonization but dios). So let's try to port the harmonization to dios as is, while keeping all the then unnecessary bits, and get the refactorization of the harmonization done as soon as it fits into the schedule.

added 1 commit

• ce170c12 - fixed getFlagger

Compare with previous version

added 1 commit

• d418d18f - harm next steps

Compare with previous version

added 2 commits

• 9cf3678a - harm next steps
• f8950ee6 - more harm-crap

Compare with previous version

added 1 commit

• 0d6e332b - fixed iloc bugs in tests

Compare with previous version

im on it, but fixing harmonization is really pain in the ass! and i have the feeling, that the really hard bits to fix, are thoose that will kicked out soon.. :/

added 1 commit

• f6e08d9c - close, very close

Compare with previous version

added 1 commit

• cec77ccc - dios in harmo - done - puhh

Compare with previous version

added 1 commit

• 0437d4e4 - fixed DmpFlagger

Compare with previous version

now everything is done. except plotting, that need a rework too. I recommend to merge with plotting broken. i will rework it, but nevertheless it will be breaking-new changes in plotting - so it doesnt matter. @schaefed will you do the merge?, to have a 2nd look. i know, maybe its not very easy to understand in some parts, especially in every detail, but maybe you see some obvious things i overlooked.