Skip to content

Commit 2002375

Browse files
authored
Merge pull request #8 from euroargodev/transition-to-the-marine-data-store
Transition to the marine data store
2 parents 600481a + a17cdca commit 2002375

File tree

4 files changed

+435
-365
lines changed

4 files changed

+435
-365
lines changed

README.md

+26-90
Original file line numberDiff line numberDiff line change
@@ -1,56 +1,49 @@
1-
|<img src="https://raw.githubusercontent.com/euroargodev/VirtualFleet_recovery/master/docs/img/logo-virtual-fleet-recovery.png" alt="VirtualFleet-Recovery logo" width="400"><br>``Virtual Fleet - Recovery`` is a CLI and webAPI to make predictions of Argo float positions|
2-
|:---------:|
3-
|[![DOI](https://zenodo.org/badge/543618989.svg)](https://zenodo.org/badge/latestdoi/543618989)|
1+
|<img src="https://raw.githubusercontent.com/euroargodev/VirtualFleet_recovery/master/docs/img/logo-virtual-fleet-recovery.png" alt="VirtualFleet-Recovery logo" width="400"><br>``Virtual Fleet - Recovery`` is a CLI to make predictions of Argo float positions|
2+
|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|
3+
| [![DOI](https://zenodo.org/badge/543618989.svg)](https://zenodo.org/badge/latestdoi/543618989) |
44

55
The goal of this repository is to provide a library to make Argo floats trajectory predictions easy, in order to facilitate recovery.
66
The library produces a prediction _patch_ or _cone_ that could be displayed on a map like here: https://floatrecovery.euro-argo.eu
77
More about Argo floats recovery in here: https://github.com/euroargodev/recovery/issues
88

9-
- [POC #1](#poc--1)
10-
* [Working design of the procedure](#working-design-of-the-procedure)
11-
* [Installation](#installation)
12-
* [Command line instructions](#command-line-instructions)
13-
+ [Usage](#usage)
14-
+ [Example](#example)
15-
* [web API](#web-api)
16-
+ [Server set-up](#server-set-up)
17-
+ [Usage](#usage-1)
18-
- [Make a prediction](#make-a-prediction)
19-
- [Visualise prediction results](#visualise-prediction-results)
20-
21-
# POC #1
22-
23-
## Working design of the procedure
9+
New version compatible with [VirtualFleet 0.4.0](https://virtualfleet.readthedocs.io/en/latest/whats-new.html#v0-4-0-2-feb-2024) and using the [Copernicus Marine Toolbox](https://help.marine.copernicus.eu/en/collections/5821001-python-library-api) to retrieve GLORYS or ARMOR3D velocity fields.
10+
11+
# Documentation (preliminary)
12+
13+
## Overall prediction procedure
2414
1. Given a specific float cycle to predict ``C``, we extract:
2515
- space/time position of the previous cycle ``C-1``,
2616
- configuration parameters of the previous cycle ``C-1``, such as parking depth, profiling depth and cycling period using the EA API (but these can be overwritten if necessary).
2717

28-
2. We download the hourly CMEMS velocity fields for a large region around the previous cycle ``C-1`` coordinates
18+
2. We download the daily CMEMS velocity fields for a region around the previous cycle ``C-1`` coordinates
2919

3020
3. We run a VirtualFleet simulation:
3121
- where we use a large number of virtual floats located with a random perturbations around the float cycle ``C-1`` position in space/time
3222
- for the cycle ``C-1`` duration
3323

3424
4. We compute the most probable position of the float cycle ``C`` and prediction metrics and figures.
3525

36-
The reason why we make random perturbations of the float cycle ``C-1`` position is not because the float position is uncertain (with GPS it is fairly accurate most of the time), but because it is a cheap way to account for errors in the velocity field. Indeed, we assume that the _phase_ of the velocity field used to advect floats is the primary source of uncertainties to predict the final position. We do not account for strain errors at this point.
26+
The reason why we make random perturbations of the float cycle ``C-1`` position is not because the float position is uncertain (with GPS it is fairly accurate most of the time), but because it is a cheap way to account for errors in the velocity field. Indeed, we assume that the _phase_ of the velocity field used to advect floats is the primary source of uncertainties to predict the final position. We do not account for velocity shear/strain errors at this point.
3727

3828
## Installation
39-
- Get this repository:
29+
30+
Our goal is to distribute VFrecovery as a standalone pipy package. In the meantime, one need to work with this repo only.
31+
32+
- Download his repository:
4033
```bash
4134
git clone [email protected]:euroargodev/VirtualFleet_recovery.git
4235
```
4336
- Add the ``cli`` folder to your path, eg:
4437
```bash
4538
export PATH="/Users/gmaze/git/github/euroargodev/VirtualFleet_recovery/cli:$PATH"
4639
```
47-
- Make sure to get the appropriate Python 3.8 environment ([using this conda file](environment.yml)):
40+
- Make sure to get the appropriate Python 3.9 environment ([using this conda file](environment.yml)):
4841
```bash
49-
conda env create -f environment.yml
42+
mamba env create -f environment.yml
5043
```
51-
- Install the experimental VirtualFleet "GulfStream" branch:
44+
- Install the last VirtualFleet version:
5245
```bash
53-
git clone --branch gulf-stream [email protected]:euroargodev/VirtualFleet.git
46+
git clone [email protected]:euroargodev/VirtualFleet.git
5447
```
5548

5649
## Command line instructions
@@ -72,7 +65,9 @@ recovery_prediction.py 6902919 99
7265

7366
A few options are available:
7467
```
75-
usage: recovery_prediction.py [-h] [--nfloats NFLOATS] [--output OUTPUT] [--velocity VELOCITY] [--save_figure SAVE_FIGURE] [--save_sim SAVE_SIM] [--vf VF] [--json] [--cfg_parking_depth CFG_PARKING_DEPTH] [--cfg_cycle_duration CFG_CYCLE_DURATION] wmo cyc
68+
usage: recovery_prediction.py [-h] [--nfloats NFLOATS] [--output OUTPUT] [--velocity VELOCITY] [--save_figure SAVE_FIGURE] [--save_sim SAVE_SIM] [--vf VF] [--json]
69+
[--cfg_parking_depth CFG_PARKING_DEPTH] [--cfg_cycle_duration CFG_CYCLE_DURATION]
70+
wmo cyc
7671
7772
VirtualFleet recovery predictor
7873
@@ -96,14 +91,15 @@ optional arguments:
9691
Virtual floats cycle duration in [hours], default: use previous cycle value
9792
9893
This script can be used to make prediction of a specific float cycle position.
99-
This script is for testing the prediction system.
100-
Note that in order to download online velocity field from 'https://nrt.cmems-du.eu', you need to set the environment variables: MOTU_USERNAME and MOTU_PASSWORD.
94+
This script can be used on past or unknown float cycles.
95+
Note that in order to download online velocity fields from the Copernicus Marine Data Store, you need to have the
96+
appropriate credentials file setup.
10197
102-
(c) Argo-France/Ifremer/LOPS, 2022
98+
(c) Argo-France/Ifremer/LOPS, 2022-2024
10399
```
104100

105101
So, don't forget to:
106-
- set the environment variables ``MOTU_USERNAME`` and ``MOTU_PASSWORD`` to be able to download the velocity field
102+
- set up your environment to be able to download velocity fields from the Copernicus Marine Toolbox
107103
- use the option ``vf`` to specify where the VirtualFleet software has been cloned (this is temporary and will change once VirtualFleet will be available on Pypi).
108104

109105
### Example
@@ -115,63 +111,3 @@ Below is an example of this prediction for the 99th cycle of the 6902919 float.
115111
The really observed 99th cycle is shown at the tip of the arrow (red point) starting from the previous 98th cycle.
116112
The VirtualFleet Recovery prediction is in the probabilistic red shading: the most probable position predicted is in the redder region.
117113
![Figure](docs/img/vfrecov_predictions_recap_VELARMOR3D_NF2000_CYCDUR240_PDPTH1000.png)
118-
119-
## web API (highly experimental)
120-
121-
In order to easily use prediction results with other (web)applications, we set-up a small web API based on [Flask](https://flask.palletsprojects.com/).
122-
123-
### Server set up
124-
125-
If you used the environment provided with this repo you already have Flask installed.
126-
In order to set up the (dev) server to access the VirtualFleet Recovery web API, open a terminal, navigate to the ``webapi`` folder and type:
127-
```bash
128-
export FLASK_DEBUG=True
129-
export FLASK_APP=myapp
130-
flask -A myapp run
131-
```
132-
133-
### Usage
134-
135-
You should know the <IP> address of the server where the Flask app is running.
136-
137-
#### Make a prediction
138-
139-
**Method 1**
140-
141-
Simply visit the prediction triggering form at:
142-
143-
```
144-
http://<IP>:5000/trigger
145-
```
146-
147-
**Method 2**
148-
149-
Go the full webAPI way: to make a prediction for the position of the ``CYC`` cycle from float ``WMO``, send a POST, request to:
150-
```
151-
http://<IP>:5000/predict/<WMO>/<CYC>
152-
```
153-
This will return a json file with the prediction results. Predictions are saved in cache, so that if you send a request for a prediction already computed, you will have the json results right away.
154-
155-
Options available :
156-
- ``velocity``: to select the velocity field to use, it can be ``ARMOR3D`` (default) or ``GLORYS``
157-
- ``nfloats``: to set the number of virtual floats to use in the probabilistic prediction. The default value is 2000.
158-
- ``cfg_parking_depth``: to set the parking depth, in db, of virtual floats. The default values is that of the previous cycle.
159-
- ``cfg_cycle_duration``: to set the cycle duration, in hours, of virtual floats. The default values is that of the previous cycle.
160-
161-
Options can be used, or combined:
162-
```
163-
http://<IP>:5000/predict/<WMO>/<CYC>?nfloats=1000
164-
http://<IP>:5000/predict/<WMO>/<CYC>?velocity=ARMOR3D
165-
http://<IP>:5000/predict/<WMO>/<CYC>?nfloats=1000&velocity=GLORYS
166-
```
167-
168-
#### Visualise prediction results
169-
170-
We made a small webpage with figures and prediction data results. It is accessible at:
171-
```
172-
http://<IP>:5000/<WMO>/<CYC>
173-
# or
174-
http://<IP>:5000/results/<WMO>/<CYC>
175-
```
176-
Here is a screenshot:
177-
![Screenshot 2022-12-06 at 16 14 17](https://user-images.githubusercontent.com/1956032/205950317-935b815f-c6fd-4e67-8bc3-71ab27d305d0.png)

README_v0.1.md

+177
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
|<img src="https://raw.githubusercontent.com/euroargodev/VirtualFleet_recovery/master/docs/img/logo-virtual-fleet-recovery.png" alt="VirtualFleet-Recovery logo" width="400"><br>``Virtual Fleet - Recovery`` is a CLI and webAPI to make predictions of Argo float positions|
2+
|:---------:|
3+
|[![DOI](https://zenodo.org/badge/543618989.svg)](https://zenodo.org/badge/latestdoi/543618989)|
4+
5+
The goal of this repository is to provide a library to make Argo floats trajectory predictions easy, in order to facilitate recovery.
6+
The library produces a prediction _patch_ or _cone_ that could be displayed on a map like here: https://floatrecovery.euro-argo.eu
7+
More about Argo floats recovery in here: https://github.com/euroargodev/recovery/issues
8+
9+
- [POC #1](#poc--1)
10+
* [Working design of the procedure](#working-design-of-the-procedure)
11+
* [Installation](#installation)
12+
* [Command line instructions](#command-line-instructions)
13+
+ [Usage](#usage)
14+
+ [Example](#example)
15+
* [web API](#web-api)
16+
+ [Server set-up](#server-set-up)
17+
+ [Usage](#usage-1)
18+
- [Make a prediction](#make-a-prediction)
19+
- [Visualise prediction results](#visualise-prediction-results)
20+
21+
# POC #1
22+
23+
## Working design of the procedure
24+
1. Given a specific float cycle to predict ``C``, we extract:
25+
- space/time position of the previous cycle ``C-1``,
26+
- configuration parameters of the previous cycle ``C-1``, such as parking depth, profiling depth and cycling period using the EA API (but these can be overwritten if necessary).
27+
28+
2. We download the hourly CMEMS velocity fields for a large region around the previous cycle ``C-1`` coordinates
29+
30+
3. We run a VirtualFleet simulation:
31+
- where we use a large number of virtual floats located with a random perturbations around the float cycle ``C-1`` position in space/time
32+
- for the cycle ``C-1`` duration
33+
34+
4. We compute the most probable position of the float cycle ``C`` and prediction metrics and figures.
35+
36+
The reason why we make random perturbations of the float cycle ``C-1`` position is not because the float position is uncertain (with GPS it is fairly accurate most of the time), but because it is a cheap way to account for errors in the velocity field. Indeed, we assume that the _phase_ of the velocity field used to advect floats is the primary source of uncertainties to predict the final position. We do not account for strain errors at this point.
37+
38+
## Installation
39+
- Get this repository:
40+
```bash
41+
git clone [email protected]:euroargodev/VirtualFleet_recovery.git
42+
```
43+
- Add the ``cli`` folder to your path, eg:
44+
```bash
45+
export PATH="/Users/gmaze/git/github/euroargodev/VirtualFleet_recovery/cli:$PATH"
46+
```
47+
- Make sure to get the appropriate Python 3.9 environment ([using this conda file](environment.yml)):
48+
```bash
49+
mamba env create -f environment.yml
50+
```
51+
- Install the experimental VirtualFleet "GulfStream" branch:
52+
```bash
53+
git clone --branch gulf-stream [email protected]:euroargodev/VirtualFleet.git
54+
```
55+
56+
## Command line instructions
57+
58+
### Usage
59+
The ``recovery_prediction.py`` script allows making predictions, i.e. at this point to produce:
60+
- a json data files with predictions information for machine/machine applications,
61+
- and a few figures to indicate where the float will make surface contact and how the probability patch was created.
62+
63+
For a simple help, you can type:
64+
```
65+
recovery_prediction.py -h
66+
```
67+
68+
To make prediction of where the 99th cycle of the 6902919 float will be, just type:
69+
```
70+
recovery_prediction.py 6902919 99
71+
```
72+
73+
A few options are available:
74+
```
75+
usage: recovery_prediction.py [-h] [--nfloats NFLOATS] [--output OUTPUT] [--velocity VELOCITY] [--save_figure SAVE_FIGURE] [--save_sim SAVE_SIM] [--vf VF] [--json] [--cfg_parking_depth CFG_PARKING_DEPTH] [--cfg_cycle_duration CFG_CYCLE_DURATION] wmo cyc
76+
77+
VirtualFleet recovery predictor
78+
79+
positional arguments:
80+
wmo Float WMO number
81+
cyc Cycle number to predict
82+
83+
optional arguments:
84+
-h, --help show this help message and exit
85+
--nfloats NFLOATS Number of virtual floats used to make the prediction, default: 2000
86+
--output OUTPUT Output folder, default: webAPI internal folder
87+
--velocity VELOCITY Velocity field to use. Possible values are: 'ARMOR3D' (default), 'GLORYS'
88+
--save_figure SAVE_FIGURE
89+
Should we save figure on file or not ? Default: True
90+
--save_sim SAVE_SIM Should we save the simulation on file or not ? Default: False
91+
--vf VF Parent folder to the VirtualFleet repository clone
92+
--json Use to only return a json file and stay quiet
93+
--cfg_parking_depth CFG_PARKING_DEPTH
94+
Virtual floats parking depth in [db], default: use previous cycle value
95+
--cfg_cycle_duration CFG_CYCLE_DURATION
96+
Virtual floats cycle duration in [hours], default: use previous cycle value
97+
98+
This script can be used to make prediction of a specific float cycle position.
99+
This script is for testing the prediction system.
100+
Note that in order to download online velocity fields from 'https://nrt.cmems-du.eu', you need to set the environment variables: MOTU_USERNAME and MOTU_PASSWORD.
101+
102+
(c) Argo-France/Ifremer/LOPS, 2022
103+
```
104+
105+
So, don't forget to:
106+
- set the environment variables ``MOTU_USERNAME`` and ``MOTU_PASSWORD`` to be able to download the velocity field
107+
- use the option ``vf`` to specify where the VirtualFleet software has been cloned (this is temporary and will change once VirtualFleet will be available on Pypi).
108+
109+
### Example
110+
111+
```
112+
recovery_prediction.py 6902915 116
113+
```
114+
Below is an example of this prediction for the 99th cycle of the 6902919 float.
115+
The really observed 99th cycle is shown at the tip of the arrow (red point) starting from the previous 98th cycle.
116+
The VirtualFleet Recovery prediction is in the probabilistic red shading: the most probable position predicted is in the redder region.
117+
![Figure](docs/img/vfrecov_predictions_recap_VELARMOR3D_NF2000_CYCDUR240_PDPTH1000.png)
118+
119+
## web API (highly experimental)
120+
121+
In order to easily use prediction results with other (web)applications, we set-up a small web API based on [Flask](https://flask.palletsprojects.com/).
122+
123+
### Server set up
124+
125+
If you used the environment provided with this repo you already have Flask installed.
126+
In order to set up the (dev) server to access the VirtualFleet Recovery web API, open a terminal, navigate to the ``webapi`` folder and type:
127+
```bash
128+
export FLASK_DEBUG=True
129+
export FLASK_APP=myapp
130+
flask -A myapp run
131+
```
132+
133+
### Usage
134+
135+
You should know the <IP> address of the server where the Flask app is running.
136+
137+
#### Make a prediction
138+
139+
**Method 1**
140+
141+
Simply visit the prediction triggering form at:
142+
143+
```
144+
http://<IP>:5000/trigger
145+
```
146+
147+
**Method 2**
148+
149+
Go the full webAPI way: to make a prediction for the position of the ``CYC`` cycle from float ``WMO``, send a POST, request to:
150+
```
151+
http://<IP>:5000/predict/<WMO>/<CYC>
152+
```
153+
This will return a json file with the prediction results. Predictions are saved in cache, so that if you send a request for a prediction already computed, you will have the json results right away.
154+
155+
Options available :
156+
- ``velocity``: to select the velocity field to use, it can be ``ARMOR3D`` (default) or ``GLORYS``
157+
- ``nfloats``: to set the number of virtual floats to use in the probabilistic prediction. The default value is 2000.
158+
- ``cfg_parking_depth``: to set the parking depth, in db, of virtual floats. The default values is that of the previous cycle.
159+
- ``cfg_cycle_duration``: to set the cycle duration, in hours, of virtual floats. The default values is that of the previous cycle.
160+
161+
Options can be used, or combined:
162+
```
163+
http://<IP>:5000/predict/<WMO>/<CYC>?nfloats=1000
164+
http://<IP>:5000/predict/<WMO>/<CYC>?velocity=ARMOR3D
165+
http://<IP>:5000/predict/<WMO>/<CYC>?nfloats=1000&velocity=GLORYS
166+
```
167+
168+
#### Visualise prediction results
169+
170+
We made a small webpage with figures and prediction data results. It is accessible at:
171+
```
172+
http://<IP>:5000/<WMO>/<CYC>
173+
# or
174+
http://<IP>:5000/results/<WMO>/<CYC>
175+
```
176+
Here is a screenshot:
177+
![Screenshot 2022-12-06 at 16 14 17](https://user-images.githubusercontent.com/1956032/205950317-935b815f-c6fd-4e67-8bc3-71ab27d305d0.png)

0 commit comments

Comments
 (0)