fix: earthdata token refresh when not redirected

[chuckwondo]

What I am changing

Fix broken token refresh mechanism for NASA Earthdata Login credentials providers

How I did it

Added status check for response from request to s3 credentials endpoint. We were assuming that the response would always be a redirect (307 with a "location" header), which is not the case.

How you can test it

In the absence of unit/integration tests for this provider, it can be tested manually as follows, with either an appropriate ~/.netrc entry for urs.earthdata.nasa.gov, or EARTHDATA_USERNAME and EARTHDATA_PASSWORD set appropriately.

Sync:

import os
from obstore.store import S3Store
from obstore.auth.earthdata import NasaEarthdataCredentialProvider

if "EARTHDATA_TOKEN" in os.environ:
    del os.environ["EARTHDATA_TOKEN"]

credentials_url = "https://data.ornldaac.earthdata.nasa.gov/s3credentials"
data_url = (
    "s3://ornl-cumulus-prod-protected/gedi/GEDI_L4A_AGB_Density_V2_1/data/"
    "GEDI04_A_2024332225741_O33764_03_T01289_02_004_01_V002.h5"
)
cp = NasaEarthdataCredentialProvider(credentials_url)
print(cp())  # {'access_key_id': ..., 'secret_access_key': ..., 'token': ..., 'expires_at': ...}
print(cp._refresh_with_basic_auth())  # {'accessKeyId': ..., 'secretAccessKey': ..., 'sessionToken': ..., 'expiration': ...}

The output from _refresh_with_basic_auth will have different values than the initial output (keys are also different, but that's because we are calling the "private" method directly).

Previously, the call to _refresh_with_basic_auth raised a KeyError.

Async:

import asyncio
import os
from obstore.store import S3Store
from obstore.auth.earthdata import NasaEarthdataAsyncCredentialProvider

if "EARTHDATA_TOKEN" in os.environ:
    del os.environ["EARTHDATA_TOKEN"]

credentials_url = "https://data.ornldaac.earthdata.nasa.gov/s3credentials"
data_url = (
    "s3://ornl-cumulus-prod-protected/gedi/GEDI_L4A_AGB_Density_V2_1/data/"
    "GEDI04_A_2024332225741_O33764_03_T01289_02_004_01_V002.h5"
)

async def main():
    cp = NasaEarthdataAsyncCredentialProvider(credentials_url)
    print(await cp())  # {'access_key_id': ..., 'secret_access_key': ..., 'token': ..., 'expires_at': ...}
    print(await cp._refresh_with_basic_auth())  # {'accessKeyId': ..., 'secretAccessKey': ..., 'sessionToken': ..., 'expiration': ...}
    await cp.close()

asyncio.run(main())

Related Issues

Fixes #538

[kylebarron]

The := operator here always confuses me because it makes me think that location is scoped just to this if block, while we're using location later 🙃