Herbie 2024.5.0

This release of Herbie has some new features and changes I've been developing and experimenting with, so I hope to hear from you if something isn't working as expected or if you have other ideas.

💣 Deprecation / Changes

I renamed the argument searchString to search. This change only affects you if you are using the keyword argument when calling the inventory or xarray methods. Herbie will now give a warning if you use searchString, so please update your scripts.

 from herbie import Herbie
 H = Herbie('2024-01-01')

- H.inventory(searchString="TMP:2 m")
- H.xarray(searchString="TMP:2 m")

+ H.inventory(r"TMP:2 m")
+ H.xarray(r"TMP:2 m")

The reason for the name change is 1) simplicity and 2) the search string isn't an exact match search but is a regex search. (I would have named it filter, but that is a python built-in and couldn't use that). I should also mention that it is recommended to use the r or raw string representation (i.e. search=r":TMP:\d+ mb") since the string is used as a regular expression.

✨ New Feature: ds.herbie.pick_points xarray accessor

I'm very excited to share this new feature. I created a new xarray accessor to get nearest neighbor point data from a model grid. I've had this on my mind for a long time and am happy with what I have made so far. This uses the Ball Tree algorithm, and thus adds scikit-learn as a new required dependency.

Here is a very simple example getting two values from the HRRR grid:

from herbie import Herbie
import pandas as pd

# Get HRRR 2-m temperature analysis
ds = Herbie('2024-1-1', model='hrrr').xarray("TMP:2 m")


# DataFrame of points to get from model grid
points = pd.DataFrame(
    {
        "longitude": [-100.25, -99.4],
        "latitude": [44.25, 45.4],
    }
)

# Pick points from HRRR grid
ds_points = ds.herbie.pick_points(points)

See more in-depth description and examples in the docs. There are options to get values from $k$ nearest neighbors from the requested point and compute the distance-weighted average of nearby points.

If you give this new feature a try, please let me know if you see any unexpected results.

📄 Documentation

I overhauled the organization of the documentation pages. Hopefully you find them organized a little better. Feel free to contribute to the docs if you see some details missing/misleading/confusing 😄

What's Changed

• Added a __bool__ method to Herbie by @blaylockbk in #298
• Adding Ability to Download NBM QMD Data by @fleegs79 in #301
• Add model templates for GDPS and RDPS. Update HRDPS model template. by @blaylockbk and @bryanguarente in #304
• Use BallTree for getting values at nearest neighbor points by @blaylockbk in #296
• Reorganize docs: spring cleaning 🌷🧹 by @blaylockbk in #308
• Rename searchString argument to search by @blaylockbk in #305

New Contributors

• @fleegs79 made their first contribution in #301
• @bryanguarente made their first contribution in #304

Full Changelog: 2024.3.1...2024.5.0

Future Changes

I am planning to rename the argument fxx for the forecast lead time to step to match the terminology used by cfgrib.

- Herbie('2023-01-01', model='hrrr', fxx=6)
+ Herbie('2023-01-01', model='hrrr', step=6)

Again, this isn't implemented yet, but I plan to implement it eventually and wanted to let you know. Provide any comments if you have any in #160

Herbie 2024.3.1

Just two minor changes; wanted to get these out there for the few it may impact...

What's Changed

• Add Cyrilex to the 'Tip' section in the user guide searchString by @cyrilbois in #295
• change common_features into EasyMap to fix the accessors.plot() by @Davidxswang in #270

Note: the ds.herbie.plot() accessor is not implemented well. It could use a lot of work to make some standard, quick-look plots, which would be really cool, if anyone has some big ideas of what that should look like. I'm open to suggestions.

New Contributors

• @cyrilbois made their first contribution in #295
• @Davidxswang made their first contribution in #270

Full Changelog: 2024.3.0...2024.3.1

Herbie 2024.3.0

ECMWF shared this exciting news a few days ago about their open data products for the Integrated Forecast System (IFS)

ECMWF now provides a much larger open dataset to the public, representing weather forecasts at a higher resolution and a reduction in some release times.
- ECMWF: Media Centre

This Herbie release updates the ECMWF model templates to access the new IFS and AIFS 0.25 degree datasets. Historical access to the 0.4 degree datasets is still available for dates before February 1, 2024. Thanks @alcoat!

Deprecation

Now that ECMWF provides open access for two different models, the argument model='ecmwf' is deprecated. Instead, you should access the Integrated Forecast System (IFS) with model='ifs' or model='aifs for the Artificial Intelligence IFS.

- Herbie('2024-3-1', model='ecmwf')  # Deprecated

+ Herbie('2024-3-1', model='ifs')

  Herbie('2024-3-1', model='aifs')  # New Model

New Feature

Herbie has two new experimental functions that may be helpful to some people.

• HerbieLatest Find the most recent available model run.
• HerbieWait Wait for a particular model run to become available.

You can see examples in the docs: Herbie Latest Data.
If these are helpful to you but you want them to be improved, I'm looking for people to help make these better, (along with FastHerbie which could use some work too).

Herbie at the American Meteorological Society

I gave a talk on Herbie at the American Meteorology Society in January. You can see my slides here. Soon, you should be able to go to the conference website and watch the recorded presentation.

It was a lot of fun talking to some of you at AMS this year. Its always a surprise to me when I'm wandering around the poster hall and someone says, "Hey Brian, Herbie was used to make this poster!" Thanks everyone for your kindness.

Please feel free to open issues when you see problems, share how you use Herbie in the discussions, and feel free to contribute your ideas and PRs.

What's Changed

• Update GEFS model template by @blaylockbk in #268
• New helper functions to find the latest model data, HerbieLatest and HerbieWait by @blaylockbk in #274
• update RRFS template by @blaylockbk in #269
• Check that curl is in system path; warn if it is not by @blaylockbk in #276
• ECMWF IFS change by @alcoat in #283

New Contributors

• @alcoat made their first contribution in #283

Full Changelog: 2023.12.4...2024.3.0

Herbie 2023.12.4

What's Changed

• Hide curl messages unless verbose by @alexander0042 in #258
• Bug Fix: Skip downloading invalid cURL range, such as in the RAP model wind messages. by @blaylockbk in #260

New Contributors

• @alexander0042 made their first contribution in #258

Full Changelog: 2023.12.3...2023.12.4

Herbie 2023.12.3

What's Changed

• BUG FIX: Fix curl range for last grib message by @blaylockbk in #257

Full Changelog: 2023.12.2...2023.12.3

Herbie 2023.12.2

What's Changed

Enhancements

• Refine config file creation by @blaylockbk in #253
• Add option to use environment variables for some settings by @blaylockbk in #254

Documentation

• DOCS: update and fix Sphinx config by @blaylockbk in #255

Full Changelog: 2023.12.1...2023.12.2

Herbie 2023.12.1

It's been a while since the last release, and I wanted to get out all the changes that have built up.

Please open an issue if you see problems. I'm trying a new "auto release on new tag" workflow action, so I may have some micro releases coming.

What's Changed

New model templates

• NEW MODEL: Add template for URMA and more RTMA by @blaylockbk in #190
• NEW MODEL: Add Canada's HRDPS model by @blaylockbk in #208
• NEW MODEL: Add HAFS model as a Herbie template by @blaylockbk in #227
• NEW MODEL: Update gefs.py to add Azure and Google as source options by @williamhobbs in #248

Enhancements

• ENHANCEMENT: Added fallback to ecmwf to also look in short cutoff by @blaylockbk in #224
• ENHANCEMENT: Only print xarray note if self.verbose=True by @joshuaeh in #228
• ENHANCEMENT: Workflow to automatically upload releases to PyPI by @blaylockbk in #211

Bug fixes

• BUG FIX: Added correct URL for 0.25 deg GEFS datasets by @swnesbitt in #176
• BUG FIX: Add quotes around filepath in curl statement by @blaylockbk in #194
• BUG FIX: Add missing requirements for hrrr zarr datasets by @haim0n in #209
• BUG FIX: Fix GEFS member check by @davidlandry93 in #221
• BUG FIX: Fix the UnicodeDecodeError: 'utf-8' error that occurred when Windows usernames contained UTF-8 characters. by @IncubatorShokuhou in #230
• BUG FIX: Fix #232 cURL off-by-one error, add tests, etc. by @GabrielKS in #233

Documentation

• DOCS: Converted searchString docs page to Markdown by @blaylockbk in #238
• DOCS: Add version switcher by @blaylockbk in #191

New Contributors

• @swnesbitt made their first contribution in #176
• @davidlandry93 made their first contribution in #221
• @joshuaeh made their first contribution in #228
• @IncubatorShokuhou made their first contribution in #230
• @GabrielKS made their first contribution in #233
• @williamhobbs made their first contribution in #248

Full Changelog: 2023.3.0...2023.12.0

Herbie 2023.3.0

Thanks again for using Herbie and your kind words. More than a few of you found me at the AMS annual meeting in Denver to thank me for publishing and maintaining Herbie. It's always encouraging to hear how I've helped someone. I'm still in shock how putting some code I wrote in grad school on the internet was discovered by so many. Hebie is still a project I work on in my free time, and I don't claim it is the most efficient or cleanly coded package, but the lessons I learn with Herbie have helped me in many ways in other work I do. So, I thank you for your involvement and use of Herbie and for helping me develop my skills.

Now I want to give a big "thank you" to these two people

1. Ryan May (@dopplershift), thanks for instructing me on how to publish Herbie on conda-forge. This really simplifies installing Herbie given its dependency on cgfrib and cartopy. Herbie can now be installed like this:

conda install herbie-data

Note: The name of this package on pypi and conda is herbie-data because herbie is taken on PyPI.

2. Andreas Motl (@amotl), thanks for your help with migrating the documentation to readthedocs and adding GitHub Actions to run the suite of tests with each pull request. Automating the doc builds and tests really helps.

These changes and other updates help make Herbie feel a little more "grown up" in this release.

Breaking Changes

• I renamed the file herbie/archive.py to herbie/core.py because the name "archive" was starting to lose its meaning. Please pay attention to how you import Herbie...

# DON'T do this anymore
from herbie.archive import Herbie

# DO this
from herbie import Herbie

• I renamed the Herbie method H.read_idx() to H.inventory() because the single word is easier for me to remember. If you use read_idx() you will get a warning for now; it will be removed later. Note that the inventory method is also available for FastHerbie. In fact, I'm migrating away from the term "index" to "inventory" because it I think it's a more accurate description of what a ".idx" file is--in inventory of the file contents. And besides, "index" means other things in python if you are a heavy Pandas user.

## DON'T do this anymore
H = Herbie(date, model)
H.read_idx()

# DO this
H = Herbie(date, model)
H.inventory()

New/Updated Model Templates

• Added rtma model template for RTMA (CONUS and Alaska). Template could be extended to include more products (Hawaii, guam, URMA products, etc.).
• Added gdas to model template by @blaylockbk in #107 Thanks, @alcoat!
• Updated gefs model template (not reforecast GEFS) by @blaylockbk in #133
• Updated ecmwf to include AWS as source for ECMWF forecasts by @blaylockbk in #159

Note: There will be additional changes to the GEFS and GFS model templates to make the more consistent. I also plan to add a template for the URMA model since it exists.

What's Changed

• Install: Herbie can be installed with conda.
• Bug Fix: Essentially turned off 'content-length' check in affdbda because files from NOMADS report a content-length of 20 bytes (something changed on NOMADS in the last year)
• Enhancement: Fix handling idx source location and generate missing idx with wgrib2 for local files. by @blaylockbk in #127
• Documentation: Made the Herbie logo a little more "shiny".
• Documentation: Overhaul of the Herbie documentation (thanks to the flu and being confined to bed for a few days).
• Documentation: Add configuration for "Read the Docs" by @amotl in #142
  • Fix RTD configuration by @amotl in #145
• CI: Run tests on GHA by @amotl in #139
• General updates, improvement, and clean up by @blaylockbk in #158
• Improvements to nearest_points required input. by @blaylockbk in #165
• Add a wrapper for wgrib2 in Herbie. by @blaylockbk in #161

New Contributors

• @amotl made their first contribution in #142

Full Changelog: 2022.09.0...2022.x.x

Herbie 2022.9.0

Huge shoutout to Ryan May for instructing me on how to publish Herbie on conda-forge. You can now install Herbie 2022.9.0.post1 with Conda

conda install -c conda-forge herbie-data

What's Changed

• Changed to CalVer versioning. Release versions will follow the YYYY.mm.micro format as I felt this more clearly highlights when the package was last updated.
• Changed default branch from master to main
• Add missing pygrib dependency by @haim0n in #74
• Remove unused import by @haim0n in #78
• Changes subset hash label to include date and fxx by @blaylockbk in #96
• Blaylockbk/issue98 change setup.py to setup.cfg and pyproject.toml by @blaylockbk in #99
• Added pygrib and cartopy as a dependency.
• Added NAM model #93
• Documentation: added dark mode documentation via PyData's Sphinx theme
• Let Herbie find the most recent GRIB2 file. by @blaylockbk in #103

New Contributors

• @haim0n made their first contribution in #74

Full Changelog: 0.0.10...2022.09.0

Herbie 0.0.10

⭐ Over 110 GitHub stars!

Wow, I had no idea so many people would find Herbie useful. Thanks everyone.

What's Changed

• Now you can import the Herbie class easier with from herbie import Herbie instead of from herbie.archive import Herbie
• Add template for GEFS by @blaylockbk in #56
• Changed how extending Herbie with custom templates works by @blaylockbk in #55
• Herbie helper tool to make an index file for all grib2 files in a directory. by @blaylockbk in #59
• Smarter cURL ranges for faster subset downloads by @blaylockbk in #62
• Moved the grib and index search in the Herbie class to their own functions (find_grib and find_idx)
• Cache the full index file Dataframe first time it is read.
• Added some ASCII art to __init__.py and some Easter eggs (because every project needs cool ASCII art and Easter eggs 😎)
• Add some multithreading tools to Herbie by @blaylockbk in #69 -- have not done extensive tests; watch your memory
• Fixed subset filename bug (de70cb7) that affected GFS files.
• Herbie can make an index file if one does not exist on the remote archive if the full grib2 file is downloaded locally and if wgrib2 is installed. 8653573
• Concat HRRR subh fields, if possible 8a9a862

Full Changelog: 0.0.9...0.0.10