README.md 10.1 KB
Newer Older
1
# CoRNy -- A Comic-Ray Neutron processing toolbox for python
2

Martin Schrön's avatar
Martin Schrön committed
3
4
![CoRNy](docs/corny-logo.png)

5
Corny aims at making CRNS **data analysis accessible for everyone**! CRNS stands for Cosmic Ray Neutron Sensing, a method used to measure snow and soil water content of the environment with neutrons from the natural cosmic background radiation.
6

7
This project has been initially developed by Martin Schrön, Erik Nixdorf, David Schäfer, Jannis Jakobi, Carmen Zengerle, Rafael Rosolem, and Steffen Zacharias. It is, however, open for contributions from an active user community! 
8

9
## Idea
10

11
 Corny wants to be understood as a *pre-processor*, i.e., it **reads original measurement files**, performs basic quality control, correction functions, and suggests a first-order soil water conversion. The generated PDF and KML output helps to have a **first glimpse on the data**, to identify potential issues or interesting features. However, Corny does *not* perform the full publication-ready analysis for you! **The output CSV is harmonized**, i.e., it wants to be imported into your individual analysis scripts -- may it be in Python, R, MatLab,  Excel, or whatever. This allows you to **do rocket science on the basis of soundly pre-processed data**.
12

13
14
15
16
 *Note:* CoRNy is the renamed version of the older *CORNish PASDy* -- COsmic-Ray Neutron flavored Processing and Analysis of Sensor Data in pYthon -- because we love cereals, and every corn contributed by the user community adds to universality and flavour. Full generalization of this toolbox also for non-cosmic-ray data is one of the many overachieving future milestones. Then, it may be renamed to PASDy again.
 
 *Important note:* This repository will be renamed to "corny" soon. Be warned to adapt your local copy.
 
17
18
19
20
21
22
## Contribution

- Did you find a bug or are you missing a feature? Please report your experience in the [project issues](https://git.ufz.de/CRNS/cornish_pasdy/issues).
- Have you resolved errors or added new features? Please push your contributions back to GitLab such that the whole community can benefit from your work. Contact [Martin](mailto:martin.schroen@ufz.de) to receive write permission.

## Usage
23

24
The CoRNy package aims at allowing various ways of usage depending on the level of detail and interaction required by individual users. It can be used in the following ways:
25

26
27
- **Mouse click** on the file `instantPASDy.py` processes the sensor data instantly. It requires a file `config.cfg` to be present in the same folder. Use this option if you have only one type of sensor for which the exact same analysis needs to be performed over and over again.
- In the **console**, anywhere on your disk, type
28

29
        python path/to/cornish_pasdy/instantPASDy.py configfile.cfg
30

31
32
33
34
    This way you can Corny on specific config files. Use this option if you have different config files in your research folder, and you don't like GUIs or you apply batch scripting. 
- **Execute** `cornyApp.py` to open the GUI (graphical user interface). Here, any config file can be selected and ammended directly through the interface. It allows to run Corny, log the output, and interactively look at plots and maps.
- **Shortcut double click** on `Corny.bat`. Copy this file from `app/Corny.bat` to your research folder, edit it with an editor and adapt the paths. Then just double-click on the file to directly open the GUI. 
- **Import Corny** from within your own scripts (e.g., using Jupyter notebooks) to directly access specific functions, e.g.:
35

Martin Schrön's avatar
Martin Schrön committed
36
37
38
39
40
        # Add the Corny folder to the recognizable paths
        import sys
        sys.path.insert(0, '/path/to/cornish_pasdy')
        # Import
        from corny import *
41

42

43
44
45
46
47
## Demo

Demonstration for a CRNS rover dataset (works similar for CRNS station data):
![Corny demonstration](docs/corny_demonstration.gif)

48
## Install
49

50
51
Here is a suggestion of how Corny could be installed. It has not been tested *everywhere*, so please contribute to this list if it also works (or doesn't) on your system :-)

52
53
54
55
56
57
58
59
60
61
62
### 0. Quick setup for basic users

If you only want to use Corny on Windows, don't intend to change its code, and don't mind the version numbers of python packages on your system, use this quick & dirty setup.

1. Install Python 3.10.* from [Python.org](https://www.python.org/downloads/). In the first screen, check "Add Python to PATH".
2. Download the [Corny zip file](https://git.ufz.de/CRNS/cornish_pasdy/-/archive/master/cornish_pasdy-master.zip) and unpack it.
3. In this folder, go to `install/` and double-click on `Install.bat`. The process could take a few minutes. That's it!
4. Copy `example_work_folder` to your favorite place where you usually do data analysis. It contains example data, configuration, and shortcuts to easify your daily work.
5. Double-click on `Corny.bat` to quickly process the example data in the console (closes when finished), or `Corny-GUI.bat` to use the graphical user interface. There, press "Load", and when it is finished, click "Save & Run". 
6. Skip all other instructions below this item.

Martin Schrön's avatar
Martin Schrön committed
63
### 1. Download Corny
64
65
66
67
68
69
70

You can download the [latest version as a zip file](https://git.ufz.de/CRNS/cornish_pasdy/-/archive/master/cornish_pasdy-master.zip). Extract the archive and Corny is ready to go.

Alternatively, you can use *Git* (optional, but recommended). Git is a handy version control system, which would allow you to easily clone and update Corny with a single command (`git pull`). You can get Git from [git-scm.com](https://git-scm.com/downloads). After installation of Git, use the command line to run:

    git clone https://git.ufz.de/CRNS/cornish_pasdy.git

Martin Schrön's avatar
Martin Schrön committed
71
### 2. Install Python
72

73
74
75
First, get Python, e.g., from here: [Python.org](https://www.python.org/downloads/).
It is important that your system knows where Python has been installed. So on the python install dialogue, check "Add Python to PATH" and "Disable path length limit". Or manually edit your system-specific environmental variables.

76
Successfully tested on: 
Martin Schrön's avatar
Martin Schrön committed
77
78

- Windows
79
    - [Python.org](https://www.python.org/downloads/) distribution, versions `3.9`, `3.10`.
Martin Schrön's avatar
Martin Schrön committed
80

81
Does *not* work on:
Martin Schrön's avatar
Martin Schrön committed
82

83
- Windows
84
    - Anaconda or Miniconda distributions, versions `3.6`, `3.7`, `3.8`, `3.9`. Conda seems to have issues with Visual C++ runtime libraries and geo-related packages (GDAL, pyproj, Fiona, osmnx), even if they are precompiled.
85

Martin Schrön's avatar
Martin Schrön committed
86
#### 2.1 Use Virtual Environments (optional)
87

88
The concept of virtual environments protects your other python projects from interfering package versions and dependency clashes. **If you are a python developer** and want to keep your packages safe from Corny dependencies, use this option.
89

90
To create a new environment, execute in any folder:
91

Martin Schrön's avatar
Martin Schrön committed
92
    python -m venv env
93

94
95
96
Now you can install packages and mess around without disturbing your other python projects. To start working, activate the session with:

    source env/Scripts/activate
97

98
99
If the execution fails on Windows, it's probably a line-ending error. Edit the file `env/Scripts/activate` and change line endings to `LF` (Unix).
To deactivate the session, just type `deactivate`.
100

Martin Schrön's avatar
Martin Schrön committed
101
### 3. Package installation
102

Martin Schrön's avatar
Martin Schrön committed
103
#### 3.1 Linux
104

105
You have two options:
106
107
108
109
110

- Install latest versions of the required packages. This may not introduce compatibility issues, but hopefully not:

        pip install -r requirements_1-linux.txt
        pip install -r requirements_2.txt
111

112
- Find missing packages that you need to install or update. For this, once install the `pipreq` package and execute:
113

114
        pipreqs ./
115
116
117
118
119
120

    The list of required packages is saved to `requirements.txt`.

        pip install -r requirements.txt

    Please remove the file `requirements.txt` (but not the one in `env/`) after you are done, as it is user-specific and should not be part of the repo.
121
122
    
To continue the installation of `kivy garden`, please follow item 4 in the next section.
123

Martin Schrön's avatar
Martin Schrön committed
124
#### 3.2 Windows
125

126
127
128
Windows has problems with the automatic installation of GDAL and other packages. That's why we split the installation into two parts:

1. Download pre-compiled packages ("wheels") of `GDAL`, `pyproj`, `Fiona`, `Shapely`, `rasterio`, and `Cartopy` directly from <https://www.lfd.uci.edu/~gohlke/pythonlibs/>. Use `Strg+F` to search for the packages and look for the *newest* versions which fit to your python installation (e.g., `cp310` is for Python version 3.10) and your Windows system (32bit or 64bit).
129

130
2. Install them in the suggested order, either one by one:
131

132
133
134
        pip install filename.whl
        
    or using:
Martin Schrön's avatar
Martin Schrön committed
135

136
        pip install -r install/python_packages/requirements_1-windows.txt
Martin Schrön's avatar
Martin Schrön committed
137

138
    *Note:* version numbers and file names my change in the future, so you might want to adapt the file to your needs.
139

140
3. Install the remaining "less problematic" packages using:
141

142
        pip install -r install/python_packages/requirements_2.txt
143

144
4. The GUI needs further installations of the kivy garden package. By now, your console should know about the command `garden`, if not, restart the console or go directly to its containing folder (e.g., `path/to/Python39/Scripts/garden`). Then execute:
145

146
147
        garden install matplotlib
        garden install mapview
148

149
    *Note:* There is a bug in the kivy garden package that does not allow us to display plots for newer versions of matplotlib. To fix it, go to your user folder (`C:/Users/username/`) and open `.kivy/garden/garden.matplotlib/backend_kivy.py` in a text editor. Comment out the line `from matplotlib import _png` (probably line 256) by adding a `#` to the beginning of the line. Save and close.
150

151
Test the installation by running `python cornyApp.py`. Let us know whether you come over any issues.
152

Martin Schrön's avatar
Martin Schrön committed
153
### 3.3 Jupyter Lab (optional)
154

155
[Jupyter notebooks](https://jupyter.org/) are a great way to play with python, here is a short instruction of installation:
156

157
    pip install jupyterlab
158

159
Run Jupyter Lab with the command `jupyter lab`.
160

161
An alternative is using the [Hydrogen package](https://atom.io/packages/hydrogen) inside [Atom Editor](https://atom.io/). It emulates a similar interactive notebook-like experience, with the advantage of clean python code that can be more easily shared and version-controlled.
162

Martin Schrön's avatar
Martin Schrön committed
163
164
165
166
167

## Examples

Example station data:

168
    python instantPASDy.py examples/station.cfg
Martin Schrön's avatar
Martin Schrön committed
169
170
171

Example rover data:

172
173
    python instantPASDy.py examples/rover.cfg
    python instantPASDy.py examples/rover-styx.cfg