ECMWF realtime data are not available from Microsoft's Azure: "https://ai4edataeuwest.blob.core.windows.net/ecmwf"

[Cong-Lee]

see below for access notes：
Microsoft Azure has recently changed the anonymous access function to ECMWF open data. Users can still access the data anonymously but they must request a short-lived SAS token.

please update your herbie, thanks!

[blaylockbk]

Can you provide more details about this change? I haven't heard about it. I'm not sure what a SAS token is.

[blaylockbk]

Found this

https://confluence.ecmwf.int/plugins/servlet/mobile?contentId=272310539#content/view/272310539

Which points to this...
https://planetarycomputer.microsoft.com/docs/concepts/sas/

Not sure yet how to implement this in Herbie right now. Sounds like we will need to depend on the planetary-computer python package. This is annoying.

Anyone out there have suggestions?

[blaylockbk]

Actually, maybe we just need to make a call to the token endpoint to the file we are trying to download.

Maybe if azure is in Herbie source priority list, get a token even if we don't need it. Or, only get a token when a file is requested from azure. Not sure what to do yet.

[blaylockbk]

I looks to me that this policy change is only affecting ECMWF forecasts right now. Let me know if anyone sees any others.

[blaylockbk]

Created a pull request with a quick and hacky fix. Needs some testing to find any issues and edge cases.

[Cong-Lee]

Each time you download data or an index, you can obtain the complete access URL via https://planetarycomputer.microsoft.com/api/sas/v1/sign?href={url}.
For example, in the following example:

import requests

download_url = "https://planetarycomputer.microsoft.com/api/sas/v1/sign?href=https://ai4edataeuwest.blob.core.windows.net/ecmwf/20250202/12z/ifs/0p25/oper/20250202120000-0h-oper-fc.grib2"
response = requests.get(download_url)
download_url_with_sas_token = response.json()["href"]
print(download_url_with_sas_token)

This will give you the complete download URL with the SAS token appended.

Frequent token retrieval may be subject to rate limits, so it is advisable to further implement a token caching mechanism.

There is another significant issue here. At certain points in time, when downloading data from Azure, an error occurs. The error is as follows:

BlobNotFound
The specified blob does not exist. RequestId:e07361b2-701e-0012-2d25-79623d000000 Time:2025-02-07T05:58:01.1142026Z