Skip to content
GitLab
Explore
Sign in
Primary navigation
Search or go to…
Project
D
data_progs
Manage
Activity
Members
Labels
Plan
Issues
4
Issue boards
Milestones
Wiki
Code
Merge requests
1
Repository
Branches
Commits
Tags
Repository graph
Compare revisions
Snippets
Deploy
Releases
Container Registry
Model registry
Monitor
Incidents
Service Desk
Analyze
Value stream analytics
Contributor analytics
Repository analytics
Model experiments
Help
Help
Support
GitLab documentation
Compare GitLab plans
Community forum
Contribute to GitLab
Provide feedback
Terms and privacy
Keyboard shortcuts
?
Snippets
Groups
Projects
Show more breadcrumbs
CHS
data_progs
Merge requests
!28
Dendro
Code
Review changes
Check out branch
Download
Patches
Plain diff
Merged
Dendro
hinza/data_progs:dendro
into
meteo
Overview
33
Commits
12
Pipelines
0
Changes
2
All threads resolved!
Hide all comments
Merged
Alexander Hinz
requested to merge
hinza/data_progs:dendro
into
meteo
2 years ago
Overview
33
Commits
12
Pipelines
0
Changes
1
All threads resolved!
Hide all comments
Expand
@schaefed
👍
0
👎
0
Merge request reports
Viewing commit
90b40081
Prev
Next
Show latest version
1 file
+
11
−
6
Inline
Compare changes
Side-by-side
Inline
Show whitespace changes
Show one file at a time
90b40081
dendro: adjust grouper function
· 90b40081
Alexander Hinz
authored
2 years ago
level1/do_dendro_calculation.py
+
11
−
6
Options
@@ -25,16 +25,21 @@ def filter(data, flags):
return
data
[
data
.
columns
[
data
.
columns
.
str
.
match
(
"
.*dendro(?!.*r)
"
)]]
def
grouper
(
n
,
df
,
fillvalue
=
None
):
# the customized grouper function split a dataframe, so that the output is a n columns long dataframe
# in the dendro calculation, one tree correspond to one dendrometer
# this consist of 3 columns: the calculated value, the manually read value and the flag
def
grouper
(
df
,
n
,
fillvalue
=
None
):
"""
This grouper function splits a pd.DataFrame into pd.DataFrames with `n` columns.
Background: Every dendrometer (i.e one tree) is represented by three columns:
1. calculated value
2. manually read value
3. quality flag
"""
args
=
[
iter
(
df
)]
*
n
return
zip_longest
(
fillvalue
=
fillvalue
,
*
args
)
def
fillNa
(
df
):
for
bhd_value
,
bhd_manu
,
flag
in
grouper
(
3
,
df
):
for
bhd_value
,
bhd_manu
,
flag
in
grouper
(
df
,
n
=
3
):
# setting 91 flag on all values, where calculated value is nan
df
.
loc
[
df
[
bhd_value
].
isna
(),
flag
]
=
91
df
[
bhd_value
]
=
df
[
bhd_value
].
fillna
(
df
[
bhd_manu
])
@@ -139,7+144,7 @@
start_date
=
device
.
start_date
,
end_date
=
device
.
end_date
,
reindex
=
True
,
fill
=
True
)
manflags
=
device
.
getManualFlags
()
if
df
.
empty
or
manflags
.
empty
:
logger
.
info
(
f
"
{
device
}
: no data found for period - skipping
"
)
@@ -177,7+182,7 @@
# corrections in index Order and not setted flags
final
=
final
.
sort_index
()
for
a
,
b
,
c
in
grouper
(
3
,
final
):
for
a
,
b
,
c
in
grouper
(
final
,
n
=
3
):
final
=
final
.
fillna
({
a
:
NODATA
,
b
:
NODATA
,
c
:
92
})
writeData
(
final
,
device
)
Loading