Releases: COMPASS-DOE/sensor-data-pipeline
v2-0
This release is a major update to the Level 1 COMPASS-FME environmental sensor data, and also introduces a new Level 2.
Level 1
This is version 2-0 (codename “Liam”) of the Level 1 synoptic and TEMPEST sensor data. Level 1 data are close to raw but are units-transformed, have flags added, and include extensive metadata.
What’s New:
✅ Data files are now annual and single-variable, rather than monthly and multi-variable
✅ New F_MAD column identifies likely outliers
✅ New code examples, documentation improvements, and more
What’s Changed:
❗️ 478M data points from late 2019 through June 2025 for TEMPEST and all synoptic sites
❗️ Data plots now include out-of-bounds indicators, color, and informative axis labels
❗️ Many corrections to older data; see release notes. Special thanks to @Wilson, Stephanie here
Level 2
This is the first release (but v2-0 to stay in sync with L1; codename “Emilio”) of the Level 2 environmental sensor data. Compared to the L1 data, these data are more consistent; better QA/QC’d; more complete, with gap-filled time series available, and feature additional derived variables. L2 data intended to be rapidly and easily usable in analyses and simulations.
What’s New:
✅ 172M data points from late 2019 through June 2025 for TEMPEST and all synoptic sites
✅ Data are every 15 minutes, from 1 January 00:00 to 31 December 23:45
✅ Outliers, out-of-bounds, and out-of-service data are dropped
✅ Gap-filling via linear interpolation (small gaps) and mean annual cycle (longer gaps) is available
✅ New derived variables soil salinity and groundwater level; special thanks to @Regier, Peter @Machado da silva, Fausto @Sulman, Benjamin here
✅ Data are in Parquet, an open source, high performance file format that can easily be read by R, Python, etc.
v1-2
Version 1-2 released 2025-02-21
- Covers late 2019 through December 2024 for TEMPEST and all synoptic sites
- All sonde (EXO) data now appear in their own "OW" (open water) plot
- The TEMPEST (TMP) folder README files now include detailed information on flood timings, volumes, etc.
What's Changed
- Create flmd-generator.R by @bpbond in #207
- Simplify code and add comments by @bpbond in #206
- Heatmap of observations by @bpbond in #202
- Create single doc reference for all folders and files by @bpbond in #212
- Add design table info and test data for GWI checktable #204 by @bpbond in #222
- Trim full-length test files down to ~10 rows like others by @bpbond in #223
- Remove SWH and GWI EXO mappings; README tweaks by @bpbond in #224
- Add detailed information on plot treatment floods #219 by @bpbond in #221
- Remove nonexistent TEROS 3 and 4 at PTR-W by @bpbond in #225
- Add preflight folder and update making-a-new-release.md by @bpbond in #230
- Don't use all cores - bad things happen by @bpbond in #234
- Change ExoTable to new OW plot by @stephpenn1 in #229
- v1-2 changes by @bpbond in #235
- Update making-a-new-release.md on pushing via rsync by @bpbond in #243
Full Changelog: v1-1...v1-2
